Our lead AI host has a tell. When Daisy Lane stops talking, her mouth usually stays open.
We measured it. On LTX 2.5 image+audio-to-video, Daisy's mouth was closed at 6 of 52 pauses across nine controlled renders (tests 179, 180 and 183-189), and open at 90 of 90 speech points. We changed the voice model, the seed, the prompt and the reference image. None of it moved the number. InfiniteTalk looked better at pauses in an earlier, rougher count, but it costs about four times the GPU: 72 seconds of compute per second of video on Nov 1, against about 17 for LTX.
So this post is a lab notebook, not a review. The short answer first: the most promising fix, a mouth-only re-dub model called TBDub, ran on the Spark today and did not close her mouth. Below is what came out in the last 60 days that might fix the mouth or sharpen our clips, how we screened it, what we got running on the Spark tonight, and what's next.
The two problems we're shopping for
- Mouths at pauses. The fix we use today is editorial: we cut to a panel or b-roll over any pause longer than half a second, and we say "sync is loose at pauses" on the page. That works, but it's a workaround.
- Sharper InfiniteTalk output. InfiniteTalk renders at 480p, so it needs an upscaler. Our upscaler is SeedVR2, which has an open report of a near-frozen frame every fourth frame (ComfyUI issue #16591).
How we looked
We listed every model created on Hugging Face since September 1 by about 70 labs (Lightricks, Wan-AI, MeiGen-AI, Tencent, ByteDance, Kandinsky Lab and others), searched keywords like lipsync, talking, avatar and upscale, and read the ComfyUI changelog from v0.34.3 to v0.43.1. Every candidate then had to pass four checks:
- Can we download the weights? Open weights, not an API.
- Does the license let a small business publish the output? This one cuts deep. The whole MiniMax-H3 family, and everything built on it, uses a license that excludes the US. That ruled out more October releases than anything else.
- Does it fit? The Spark has 128 GB of unified memory (about 121 GiB visible to the system), shared by everything.
- Does it run on ARM? The Spark's GB10 is an aarch64 chip. A lot of Python tooling assumes x86.
The shortlist
| Model | What it does | License | Size | On the Spark |
|---|---|---|---|---|
| 1. TBDub Student V1.1 | Re-dubs the mouth of an existing video from a new audio track | Apache-2.0 | 15.3 GB | Runs, after three fixes (below) |
| 2. Kandinsky 6.0 VSR, 2-step | Video upscaler, x2 to x4 | MIT | 17.5 GB | Installed; not run yet |
| 3. LTX 2.5 IC-LoRAs | Refine, restore and reference-sheet add-ons for the LTX we already run | LTX-2.x Community, gated | 1.3-1.7 GB each | Not downloaded |
| 4. RelayVSR X2 | 2x upscaler that claims less flicker | Unknown | 10.3 GB | Not downloaded |
| 5. TalkingAvatar TA2.0 | Image plus speech to a stylized talking character | Apache-2.0 | 14.5 GB | Watch list |
TBDub (GitHub, Hugging Face) was the one we were most excited about. It keeps the video you already have and lets an audio-driven model redraw just a 512x512 crop around the mouth. If it works, we keep LTX's lively head and brows and get lips that close when the audio goes quiet.
Its README says 63 seconds for 15 seconds of audio on an RTX PRO 5000, and about 28 GiB peak memory. Those are the maker's numbers, not ours.
Kandinsky 6.0 VSR (model) is a second opinion on SeedVR2. The maker quotes 41 seconds on an H100 for a 5-second 768x512 clip at x2. One catch: it takes at most 121 frames (5 seconds at 24 fps) per call (constants.py), so longer clips go in chunks.
The LTX 2.5 IC-LoRAs (Refine-Details, Restore, Ingredients) are small add-ons for the model we already use. Refine-Details could upscale LTX host clips without switching model families. None of them touches the pause problem.
RelayVSR (card) claims to suppress the flicker generated video gets when upscaled, which is exactly SeedVR2's weak spot. But neither of its model cards states a license, so we haven't downloaded it.
TA2.0 (Hugging Face) is older than the rest (first released in July; its code was updated October 14), so it's here as a watch-list entry. It's trained on animation, so it's a mascot tool more than a host tool. At 30 steps and 480p we'd expect InfiniteTalk-class cost, but that's a guess, not a measurement.
What we got running tonight
TBDub would not run on the Spark out of the box. Three things were in the way.
- No ARM wheel. TBDub asks for MediaPipe 0.10.21, which has no aarch64 build. Version 0.10.18 does (PyPI), and TBDub runs on it.
- A pickle file. The HuBERT audio model ships as a PyTorch
.bin, which can run code when it loads. We loaded it once with PyTorch's restricted loader and saved it as safetensors. - The detector couldn't find her face. This was the fun one. TBDub hard-codes MediaPipe's full-range face detector, the one built for faces far from the camera. On Daisy's 15.7-second LTX clip it found her in 0 of 377 frames, at every confidence threshold down to 0.05. MediaPipe's short-range detector, built for faces near the camera, found her in 377 of 377. A host framed for a talking-head shot is exactly the case short-range is for.
That third fix is a one-line patch to TBDub's code, so we're saying so. It makes the detector choice an environment variable and leaves the default unchanged:
- FaceDetection(model_selection=1, min_detection_confidence=0.5)
+ FaceDetection(
+ model_selection=int(os.environ.get("TBDUB_MP_MODEL_SELECTION", "1")),
+ min_detection_confidence=0.5)So everything below runs a patched TBDub, not the upstream default. One more small trap: TBDub's own preprocessing report still says "full_range" in its backend field after the patch, because that label is fixed text in the code.
The headline: TBDub did not close Daisy's mouth
The real test ran at 13:24 today (tests 851 and 852), after the Spark's freezes pushed it back from the morning. TBDub redrew the mouth on two of Daisy's LTX clips, driven by the same audio, and we counted the frames exactly the way the original test did: one frame from the middle of every pause, one on every fourth word, and a smile with parted lips counts as open.
| Clip | LTX 2.5 only | After the TBDub mouth pass |
|---|---|---|
| Test 179, eleven_v3 voice | closed at 1 of 6 pauses (test 179) | closed at 0 of 6 (test 851) |
| Test 183, eleven_v4 voice | closed at 0 of 5 (test 183) | closed at 0 of 5 (test 852) |
Open on speech stayed 10 of 10 on both. To make sure the midpoint frame wasn't missing a late close, we also pulled seven frames across every pause: no frame inside a pause shows closed lips. Daisy grins through the silences, exactly as she did before. The one pause LTX did close on (10.88 s) is open after the re-dub.
What it did well: there's no visible seam where it pastes the face back, the skin tone at the chin matches, and the lip shapes on speech look plausible (we didn't score them against LTX's). Two more things we saw:
- Cost. 252.5 seconds of compute for 15.7 seconds of audio, and 225.4 for 14.3: about 16 seconds per second, model load included. That's about what LTX itself took on these two clips (17.2 and 18.5, measured this morning before the 12:04 kernel and driver update; TBDub ran after it), so the pair would roughly double the GPU cost of a host clip.
- It isn't a pure paste. Outside her face, TBDub's frames aren't frame-for-frame the LTX originals (the closest match is 6 to 7 frames apart), so a side-by-side drifts. We haven't worked out why.
Who counted: the AI producer of this post (Claude Opus 5.5), by eye, with the same rule as the original test. The score sheets and each test's verdict carry the frame times, so anyone can recount.
Kandinsky 6.0 VSR is queued, not run yet. It's the last job in tonight's GPU queue and may not run before the run ends at 19:00. When it runs, it gets the same 4-second InfiniteTalk clip SeedVR2 gets, so the two upscalers compare like for like, and we'll add the crop comparison here.
What we're not testing, and why
- The MiniMax-H3 family (Pika H3 Flex, FastH3, Open-H3-Regenerate-2K, Tessera and the rest): the base license excludes the US (and the EU, UK and South Korea), so we don't publish anything made with it.
- Kandinsky 6.0 audio-to-video: its audio input can't be combined with an image, so it can't animate our host stills (docs).
- Kandinsky S2V: a paper, but no weights yet.
- LTX Dub-It on 2.5: still validated on 2.3 only, and it regenerates the speech, so it can't use our ElevenLabs track (docs).
- Hosted lip-sync and upscale nodes that landed in ComfyUI in October: paid, closed APIs. Not this lab's lane tonight.
- Video Diffusion Avatars: non-commercial license (CC BY-NC-SA 4.0).
What's next
- The LTX IC-LoRAs are waiting on a license click. Each repo is gated behind an "Agree" button on its Hugging Face page, and when we checked this morning we didn't have access yet. Refine-Details and Restore also need Lightricks' own ComfyUI nodes, and we didn't install any custom nodes tonight: both of our ComfyUI servers share one code folder, so a node installed for one loads into the other.
- RelayVSR gets a test the day its license is published.
- TBDub is off the host recipe for now. One test is still worth a few minutes of GPU: a source clip where the host isn't grinning (Theo, say), to see whether TBDub follows the face it's given more than the audio. That's a guess we haven't tested.
- Until something closes the mouth, the editorial fix stays: cut away over long pauses and say "sync is loose at pauses".
Every model fact above was checked on November 2, 2026, against the linked pages. Numbers from makers are labelled as theirs. Numbers from our Spark name the test they came from.