How we tested
TBDub keeps the video you already have and redraws only a 512x512 crop around the mouth, driven by an audio track. On Nov 2 we pointed it at the wrong job: we fed Daisy's clips their own audio, hoping it would close her mouth at pauses. It didn't. This time we gave it the job it's built for, new dialogue.
One picture, three soundtracks. The picture is the first 6 seconds of an LTX 2.5 image+audio clip of our AI host Daisy Lane saying "Okay, story time. Last summer, four of us drove to the coast to catch the sunrise." We muted it and gave TBDub two lines she never said on that take:
- Clip 1, same voice, next line: "Around four, the map just gave up. No signal. So we followed the..."
- Clip 2, a different ElevenLabs voice, a different script: "Hey, welcome to the Video Lab. I'm going to talk for about forty five seconds so we can see what..."
Then we pulled a mouth close-up every 0.2 seconds from the original and from both outputs, each tile marked with the word being spoken at that moment. The three strips are in the "Reference images" fold under the first comparison.
What the frames show
The mouth follows the new words, not the old ones. Three moments make the case:
- 1.1 to 1.5 s. In the original, Daisy is finishing "story time" and her lips are together. In clip 1 they round for "four", and in clip 2 they open wide on "Lab".
- 2.9 to 3.3 s. The original is mid-sentence ("four of us"), mouth moving. Clip 1 has its own pause there, after "gave up", and the lips stay shut for all three tiles.
- Lip closures land on the right sounds. Clip 1 closes on the m of "map" at 1.7 s. Clip 2 closes on the b of "about" at 3.5 s.
The grin is the catch. Daisy's source take smiles with her teeth showing for most of its last two seconds, and TBDub keeps it. Pauses under half a second closed in both clips (0.46 s in clip 1, 0.36 s in clip 2). The long ones, 0.97 s and 0.88 s, came out as an open-teeth smile, and so did the half second of silence before clip 1's first word. It redraws the mouth on top of the face it's given, and that face is grinning.
What I'd do with it
- Changing a line after the fact: use it. Fix a fluffed word, swap a product name, re-voice a host clip into a new script, and keep the head and eye motion you already liked.
- Pick a calm source take. Start from a take where the host isn't smiling through the stretch you're re-voicing. A grin in the source comes through in the output.
- Keep the cutaway for long pauses. Anything over about half a second still gets a cutaway or a b-roll insert, same as our LTX host clips.
- Re-dub whole lines, not snippets. Longer clips cost less per second: about 27 s of compute per second of video for our 6-second clips here, about 16 for our 14- and 16-second clips on Nov 2.
Setup and gotchas
The same three fixes from our first TBDub run still apply on the Spark's aarch64:
- MediaPipe 0.10.18, not the 0.10.21 TBDub asks for. 0.10.21 has no ARM wheel.
- HuBERT as safetensors. The audio model ships as a PyTorch pickle; we loaded it once with the restricted loader and saved it as safetensors.
- The short-range face detector. TBDub hard-codes MediaPipe's full-range detector, which found Daisy in 0 of 377 frames on our first run. Our one-line patch makes it a setting, and
TBDUB_MP_MODEL_SELECTION=0picks short-range:
- FaceDetection(model_selection=1, min_detection_confidence=0.5)
+ FaceDetection(
+ model_selection=int(os.environ.get("TBDUB_MP_MODEL_SELECTION", "1")),
+ min_detection_confidence=0.5)With it, the detector found her face in all 144 frames of both runs here.
Two things to know about the output. It comes out at 25 fps even from a 24 fps source (144 frames in, 150 out), so set your edit to 25 or conform it. And it writes the new audio into the file, so you don't have to mux it yourself.
Settings
Student mode, 2 steps, sigma shift 1.0, motion from latents, seed 42. The 6-second cut of the source went in muted, at 1280x704. The script we ran is linked at the bottom of the page.
Speed and memory
168 s for clip 1 and 158 s for clip 2, each for 6 s of video, start to finish with model load. That's 26 to 28 s of compute per second of video. Free memory dropped 24 GB during each run.