TBDub puts new words in an AI host's mouth on a DGX Spark. It can't stop her grinning

Rated G

We gave Taobao's mouth-only re-dub model two lines our AI host never said, over the same 6-second clip: her lips followed the new words, short pauses closed, and her source take's grin came through every long one.

Same picture: the original words vs a new line

  • Clip 1: The original take with its own words: AI host Daisy Lane (AI voice) on LTX 2.5 image+audio-to-video. This picture, muted, is what TBDub re-voiced in both clips below. Note the grin from about 4 s on.
    Prompt and settings

    Prompt: Spoken line (its own audio): "Okay, story time. Last summer, four of us drove to the coast to catch the sunrise."

    Settings: LTX 2.5 22B distilled int8, image+audio-to-video, 1280x704, 24 fps, seed 42, Daisy's on-set still; voice ElevenLabs eleven_v3. First 6 s of a 15.7 s clip rendered Nov 2 2026.

    Reference images (1)
  • Clip 2: Clip 1: the same picture with Daisy's next line in the same voice. Watch the lips round for "four" at 1.1 s and stay shut after "gave up" at 3 s, where the original take was still talking. In the long pause after "No signal" (3.9-4.8 s), the grin is back.
    Prompt and settings

    Prompt: New line (driving audio): "Around four, the map just gave up. No signal. So we followed the..."

    Settings: TBDub Student V1.1 (Apache-2.0), student mode, 2 steps, sigma shift 1.0, motion from latents, seed 42, MediaPipe 0.10.18 short-range face detector (one-line patch); source picture 1280x704 at 24 fps, output 1280x704 at 25 fps. Rendered Nov 14 2026 on a DGX Spark. 168 s wall for 6 s, model load included.

    Reference images (2)

Same picture, two new lines in two voices

  • Clip 3: Clip 2: the same picture with a different ElevenLabs voice and a different script. The mouth keeps up with fast speech (20 words in 6 s) and closes on the b of "about" at 3.5 s. The pause after "Video Lab" (1.6-2.5 s) turns into a grin.
    Prompt and settings

    Prompt: New line (driving audio, a stock ElevenLabs voice, not Daisy's): "Hey, welcome to the Video Lab. I'm going to talk for about forty five seconds so we can see what..."

    Settings: TBDub Student V1.1 (Apache-2.0), student mode, 2 steps, sigma shift 1.0, motion from latents, seed 42, MediaPipe 0.10.18 short-range face detector (one-line patch); source picture 1280x704 at 24 fps, output 1280x704 at 25 fps. Rendered Nov 14 2026 on a DGX Spark. 158 s wall for 6 s, model load included.

    Reference images (2)

Verdict

Use TBDub on a Spark to change what an AI host says without re-rendering the shot. Over one 6-second clip of our host, two new lines came out with lips that follow the new words, closing on the m in "map" and the b in "about" where the original take was saying something else. It doesn't fix a host who grins: in every pause longer than about half a second, the source take's open-teeth smile came through, so cut away there as before. Short clips cost about 27 s of compute per second, and a 15-second clip about 16.

Credits and settings

ModelTBDub Student V1.1 (re-dubbing new dialogue onto existing video)
MakerAlibaba Taobao & Tmall (TaoLiveAIGC)
VersionTBDub Student V1.1 (tbdub_student.safetensors + Wan2.2 VAE), 2-step student mode
What it doeslip-sync
LicenseApache-2.0
HardwareDGX Spark: NVIDIA GB10, ARM (aarch64), 128 GB unified memory. TBDub in its own Python venv (torch 2.11, CUDA 13), not ComfyUI; MediaPipe 0.10.18.
Speed27 s of compute per second of video. 1280x704, 2 student steps, 6 s clips. Wall time for one inference.py call, start to file written, model load included (168 s and 158 s).
Peak memory24 GB. Drop in system available memory (/proc/meminfo, sampled every 2 s): 94 to 70 GB and 95 to 71 GB, nothing else rendering.
Tested onNov 14, 2026
AI hosts

Daisy Lane (AI host; ElevenLabs (AI voice), AI voice)

Narration
Not supplied
Music source
Not supplied

What it does well

  • The mouth follows the new line, not the old one: at 1.1-1.5 s the original is closed while the re-voiced clips round for "four" and open on "Lab"
  • Lips close on m and b sounds ("map" at 1.7 s, "about" at 3.5 s)
  • Short pauses close even where the source was talking (clip 1, 2.9-3.3 s)
  • Works with a voice that isn't the host's own (clip 2)
  • Apache-2.0, and light on a Spark: free memory fell 24 GB

Where it breaks

  • Long pauses keep the source take's grin: open teeth at 0.97 s and 0.88 s pauses, and in the half second before clip 1's first word
  • Short clips are expensive: 158-168 s for 6 s of video, model load included
  • Output is 25 fps from a 24 fps source (144 frames in, 150 out)
  • On aarch64 it needs three fixes: an older MediaPipe, HuBERT converted to safetensors, and a patch for the close-range face detector

How we tested

TBDub keeps the video you already have and redraws only a 512x512 crop around the mouth, driven by an audio track. On Nov 2 we pointed it at the wrong job: we fed Daisy's clips their own audio, hoping it would close her mouth at pauses. It didn't. This time we gave it the job it's built for, new dialogue.

One picture, three soundtracks. The picture is the first 6 seconds of an LTX 2.5 image+audio clip of our AI host Daisy Lane saying "Okay, story time. Last summer, four of us drove to the coast to catch the sunrise." We muted it and gave TBDub two lines she never said on that take:

  • Clip 1, same voice, next line: "Around four, the map just gave up. No signal. So we followed the..."
  • Clip 2, a different ElevenLabs voice, a different script: "Hey, welcome to the Video Lab. I'm going to talk for about forty five seconds so we can see what..."

Then we pulled a mouth close-up every 0.2 seconds from the original and from both outputs, each tile marked with the word being spoken at that moment. The three strips are in the "Reference images" fold under the first comparison.

What the frames show

The mouth follows the new words, not the old ones. Three moments make the case:

  • 1.1 to 1.5 s. In the original, Daisy is finishing "story time" and her lips are together. In clip 1 they round for "four", and in clip 2 they open wide on "Lab".
  • 2.9 to 3.3 s. The original is mid-sentence ("four of us"), mouth moving. Clip 1 has its own pause there, after "gave up", and the lips stay shut for all three tiles.
  • Lip closures land on the right sounds. Clip 1 closes on the m of "map" at 1.7 s. Clip 2 closes on the b of "about" at 3.5 s.

The grin is the catch. Daisy's source take smiles with her teeth showing for most of its last two seconds, and TBDub keeps it. Pauses under half a second closed in both clips (0.46 s in clip 1, 0.36 s in clip 2). The long ones, 0.97 s and 0.88 s, came out as an open-teeth smile, and so did the half second of silence before clip 1's first word. It redraws the mouth on top of the face it's given, and that face is grinning.

What I'd do with it

  • Changing a line after the fact: use it. Fix a fluffed word, swap a product name, re-voice a host clip into a new script, and keep the head and eye motion you already liked.
  • Pick a calm source take. Start from a take where the host isn't smiling through the stretch you're re-voicing. A grin in the source comes through in the output.
  • Keep the cutaway for long pauses. Anything over about half a second still gets a cutaway or a b-roll insert, same as our LTX host clips.
  • Re-dub whole lines, not snippets. Longer clips cost less per second: about 27 s of compute per second of video for our 6-second clips here, about 16 for our 14- and 16-second clips on Nov 2.

Setup and gotchas

The same three fixes from our first TBDub run still apply on the Spark's aarch64:

  • MediaPipe 0.10.18, not the 0.10.21 TBDub asks for. 0.10.21 has no ARM wheel.
  • HuBERT as safetensors. The audio model ships as a PyTorch pickle; we loaded it once with the restricted loader and saved it as safetensors.
  • The short-range face detector. TBDub hard-codes MediaPipe's full-range detector, which found Daisy in 0 of 377 frames on our first run. Our one-line patch makes it a setting, and TBDUB_MP_MODEL_SELECTION=0 picks short-range:
- FaceDetection(model_selection=1, min_detection_confidence=0.5)
+ FaceDetection(
+     model_selection=int(os.environ.get("TBDUB_MP_MODEL_SELECTION", "1")),
+     min_detection_confidence=0.5)

With it, the detector found her face in all 144 frames of both runs here.

Two things to know about the output. It comes out at 25 fps even from a 24 fps source (144 frames in, 150 out), so set your edit to 25 or conform it. And it writes the new audio into the file, so you don't have to mux it yourself.

Settings

Student mode, 2 steps, sigma shift 1.0, motion from latents, seed 42. The 6-second cut of the source went in muted, at 1280x704. The script we ran is linked at the bottom of the page.

Speed and memory

168 s for clip 1 and 158 s for clip 2, each for 6 s of video, start to finish with model load. That's 26 to 28 s of compute per second of video. Free memory dropped 24 GB during each run.

The script we ran (TBDub inference.py, student mode) ↗