Here's the problem with judging video models by the clips people post: you only ever see the best take. Somebody ran a prompt twenty times, kept the one that worked, and now that clip is "what the model does." You can't compare that to anything, and you can't repeat it.
So the Video Lab runs a method instead. Same prompts, same seeds, same sizes for every model. Every run writes the same measurements to a file. Every clip gets cut into frames and scored on the same seven criteria, and lip-sync gets counted on specific frames instead of eyeballed. Then the weak spots get written down too, because there are real ones.
This post is the whole method, with the files and the numbers from the Nov 2 run on our DGX Spark, so you can copy it. Everything below has a test id you can trace. Where something wasn't measured, it says so.
How we tested
The short version, in order:
- Pick a prompt from a fixed suite. Ten prompts, each built to stress one thing.
- Run it at a fixed seed and a fixed size for that model family, through one queue, so only one heavy job runs at a time.
- Log the measurements: wall time (including model load, and we say when a load happened), seconds of compute per second of video, the lowest free memory during the run, and the exact graph that ran.
- Cut the clip into frames: a 12-frame contact sheet, an 8-frame identity strip, and for talking clips a lip-sync score sheet built from the narration's word timestamps.
- Score it 1 to 5 on seven criteria, with a one-line note per score saying what the frames show.
- Compare only like with like: same prompt id, same seed. A model "wins" on a criterion only when both seeds agree.
The method doc is framework/METHODOLOGY.md v1.0, the prompts are framework/prompts.json v1.0, and the scoring tools are four small scripts: run-test, score-sheet, identity-strip and review-test (plus words-approx for old clips with no timestamps). Tests land in tests/<id>/, each with settings.json, timing.json, the outputs, a contact sheet (a thumbnail for stills) and verdict.md.
Why fixed prompts
A model that nails a coffee mug can fall apart on a kickflip. If you test each model on whatever prompt you felt like that day, you're measuring your prompts, not the models. So the suite is ten prompts, and each one is aimed at a specific failure:
| id | what it stresses |
|---|---|
| P01 talking host | face, mouth, identity, idle motion |
| P02 product | object fidelity, slow push-in, steam |
| P03 landscape | camera orbit, parallax, a stable horizon |
| P04 fast action | a kickflip: limbs, blur, physics |
| P05 hands | fingers kneading dough |
| P06 text on screen | a neon sign that must read VIDEO LAB |
| P07 stylized cartoon | holding a look over time |
| P08 robot mascot | mascot identity, a tiny mouth |
| P09 night street | many moving things, reflections |
| P10 animal fur | fur, water, fast secondary motion |
Each prompt has two texts: a motion prompt for text-to-video and image-to-video, and a still prompt for the source image. The source still for every image-to-video test is the Z-Image Turbo render of that still prompt at seed 42, so every image-to-video model starts from the same pixels. The prompt text is frozen once tests run. A new idea gets a new id (P11 and up exist now for another post), it never edits P06.
The clips are 5 seconds: 121 frames at 24 fps for LTX 2.5, 81 frames at 16 fps for Wan 2.2. That's the shot length our edits actually use, both models handle it natively without windowing, and it keeps a full sweep affordable on one GPU. Long-clip drift is its own test with its own lines.
Why fixed seeds
Same prompt, different seed, different clip. Sometimes a very different clip. If model A got a lucky seed and model B got an unlucky one, a one-run comparison tells you nothing.
So every prompt runs at seeds 42 and 1234 on every model, and the comparison rule is strict: before we say A beats B on a criterion, A's two seeds both have to beat B's two seeds. If the seeds disagree, the post says "mixed" and quotes both.
Honest status as of this writing: the seed-42 sweep is scored for stills, LTX 2.5 text-to-video and LTX 2.5 image-to-video (30 tests). The seed-1234 repeats did not run tonight: they were parked at 05:02 to make room for host renders, and come back only if the queue runs dry. So everything in this post comparing families is a one-seed lead, labelled that way.
What gets measured
run-test wraps one job and writes timing.json and settings.json into the test folder. Every number has a named source:
| measurement | where it comes from |
|---|---|
| wall time | job submit to ComfyUI reporting done. It includes the model load when the model family changed on that server; cold_load_hint says so |
| seconds of compute per second of video (s/s) | wall time divided by output length |
| min MemAvailable | /proc/meminfo sampled every 2 s for the life of the job |
| resolution, fps, frames, audio | ffprobe of the output |
| models, quant, steps, sampler, cfg, seed | read out of the exact API graph that was sent |
| ComfyUI and torch versions | /system_stats at start |
The load matters more than people think. Test 011 (LTX text-to-video, P01) was the first LTX job after the stills: 108.1 s wall, 21.4 s/s, with the hint "family switch on this port." Test 012, the next prompt on the same warm model, took 75.1 s, 14.9 s/s. Same model, same settings. If you quote 21.4 as "LTX speed" you're off by more than 40 percent (21.4 against 14.9 is 44 percent high). The scorecard reports both an all-runs mean and a warm-only mean for that reason (LTX text-to-video: 15.25 all, 14.57 warm).
Min MemAvailable matters because this box has one pool of unified memory and no clean out-of-memory error: when the pool runs out, it freezes. During test 027 free memory went from 24 GB at the start to a 15 GB low. That number is in the file, not in somebody's memory of watching a dashboard. (Memory figures in this post are GiB, the unit /proc/meminfo uses, written GB.)
Wall times are for this box with whatever else was resident at the time: a voice model, the other ComfyUI server, other agents. They compare fairly with each other inside one run, not with a clean machine.
The rubric: seven criteria, 1 to 5, with anchors
A score without an anchor is a mood. So each criterion has a written 1, 3 and 5 that two reviewers should agree on from the same frames:
| criterion | 1 | 3 | 5 |
|---|---|---|---|
| Prompt adherence | subject or action missing or wrong | subject right, one named beat or camera move missing | every element, beat and camera move there, in order |
| Motion quality | frozen, jittery, slideshow, limbs teleport | plausible but soft or floaty, one stutter | looks filmed: weight, timing, blur match the action |
| Temporal consistency / identity | subject changes identity, colour or shape; background morphs | identity holds, visible drift by the end | first and last frame could be the same photo session |
| Artifacts (hands, faces, text) | extra fingers, melted face, garbled text in most frames | one or two frames with a clear artifact | nothing wrong in any sampled frame |
| Lip-sync accuracy | mouth unrelated to audio | closed at about 2/3 of pauses, open at about 2/3 of speech points | closed at every sampled pause, open at every speech point |
| Audio quality | missing, clipped, out of sync, added voices | intact and in sync, flat or slightly noisy bed | clean, in sync, ambience fits |
| Overall usable for publication | wouldn't ship in any cut | b-roll insert or after a trim | ship it as a hero shot |
Whole numbers only. If the frames can't tell you, the note says so instead of defaulting to a 3. Lip-sync and audio are scored only on lip-sync tests; everywhere else they're N/A. Text counts under artifacts only when the prompt asks for text. Uninvited text (a logo nobody asked for) is an artifact and a prompt miss.
Here's what that looks like on one real clip. P06 asks for a neon sign reading VIDEO LAB.
- Test 016, LTX 2.5 text-to-video: the sign reads VIDDEO LAB, extra D, in all 12 sampled frames. Prompt adherence 2, artifacts 2, overall 1. "A misspelled title sign cannot ship in any cut."
- Test 026, LTX 2.5 image-to-video from the Z-Image still (test 006, spelled right): VIDEO LAB in every sampled frame. Prompt adherence 4, artifacts 5, overall 4, with the note that the camera pushes in when the prompt said static.
Same prompt, same seed, same model family. The only change was starting from a correct still. That's the kind of finding the method is for: for words on screen, make the still first (or put the text in an overlay), then animate.
The flicker in that prompt is a good example of measuring instead of guessing. The prompt says the sign flickers twice. Instead of squinting at frames, the reviewer ran ffmpeg's signalstats for the mean brightness of all 121 frames. Test 016: one fade-up from 27 to 46 by frame 24, then steady. No flicker. Test 026: two dips (61 to 54 around frames 21-26, 62 to 58 around frames 39-42), then steady. Two flickers. Soft ones, which is why motion got a 4 and not a 5.
The contact sheet
sheet.jpg is 12 frames spread evenly over the clip at 480 px, one about every 0.45 s on a 5 s clip. The reviewer scores from that, and grabs extra frames whenever something needs a closer look, naming the time in the note.
Identity drift strips
identity-strip takes 8 evenly spaced frames, first and last included, crops the centre 60 percent, and lays them in one row. You compare the first and last cells, then look for the frame where a change starts. The call is none, slight or clear, plus what changed. "Clear" caps temporal consistency at 2.
That strip is a nice lesson in what drift means. Drift: none. The sign is perfectly stable. It's just stably wrong. Consistency and correctness are different columns for a reason.
A real "slight" from the same run: test 024 (LTX image-to-video, the kickflip) holds the rider's face and build, but a red waistband appears at 2.72 s and a chest logo from 4.08 s that the source still never had. Drift: slight. Temporal consistency 4, and the uninvited logo also cost it on artifacts.
Counting lip-sync on frames
This is the part people usually skip, and it's the part where vibes lie the most. A talking head that moves its mouth looks synced at normal speed. The question is whether the mouth closes when the speech stops and opens when there's speech. So we count.
score-sheet does it from the narration's word timestamps (ElevenLabs returns per-character timing; our TTS wrapper saves it next to the mp3):
- A word's speech ends at its last letter. (Gotcha below.)
- A pause is any gap longer than 0.35 s between one word's end and the next word's start, plus the lead-in and the tail when they're longer than 0.35 s. Pull the frame at the midpoint of each pause. The mouth should be closed.
- A speech point is the midpoint of every 4th word. The mouth should be open or visibly moving.
- Lay the pause frames on the top row (red labels) and speech frames on the bottom row (green labels), face-cropped, and write every sampled time to a JSON file so the count can be checked.
- Report it as a fraction: "closed at 5/6 pauses, open at 7/8 speech points." The 1-5 score follows: both fractions at or above 0.9 is a 5, both at or above 0.6 with one miss is a 3, below that is a 1 or 2. The tail is sampled and reported separately, not in the headline.
Here's a real one: AI host take DAISY-1 from job 058 (LTX 2.5 image+audio-to-video, 1280x704, seed 42, voice on ElevenLabs eleven_v4):
Eight pauses, at 1.80, 2.58, 3.52, 7.84, 8.84, 9.76, 10.92 and 14.36 s, every one between 0.36 and 0.56 s long. Nine speech points. The mouth is open or moving at most speech points. The pauses are the problem: at 2.58, 3.52 and 7.84 s the lips are parted mid-smile, at 10.92 s it's a toothy grin.
The producer who ran that take counted closed at about 4-5 of 8. A second count of the same sheet for this post came out at 3, maybe 4 (8.84 and 14.36 clearly closed, 1.80 closed in a wink, 9.76 a judgment call). Same frames, two counts, a point or two apart. Hold that thought for the weak-spots section.
Across job 058's four takes, the count was 6-7 of 19 pauses closed. That's under our gate of 4 of 6. The obvious suspect was the voice: eleven_v4 caps sentence pauses at roughly 0.55 s, so every "pause" is a breath.
So this post's four host lines were written for eleven_v3 with "..." at sentence ends. On one test script that made 14 pauses longer than 0.6 s, against 1 on eleven_v4 at the cast settings (an interim run for the long-host post). Then the controlled test ran (framework/TTS.md): one Daisy window, nine renders, one thing changed at a time. Longer pauses did not close her mouth. It came out 6 of 51 pauses closed: eleven_v3 2 of 12, eleven_v4 0 of 10, multilingual v2 with a break tag 1 of 6. A closed-lips prompt, a different reference image and a second seed didn't help either. The current advice is eleven_v4 at the cast settings for ordinary host lines, and "..." on v3 only where you need a real silence to cut on. The fix that's left is in the edit: write host lines with few long pauses, and cut to a panel or b-roll over any pause longer than about half a second instead of holding on the face.
This post's own four host takes say the same thing, and they're a second example of two reviewers disagreeing. The pauses ran 0.36 to 1.35 s (timings corrected with ffmpeg silencedetect, see the gotcha below). The producer's count was 7 to 9 of 21 pauses closed. The independent editor (see "The same idea, applied to the posts") recounted the same four sheets and got 6, or 7 on a generous reading. We agree on the speech points: 34 of 35 open. Either way it's under the gate, and the instructional's caption says sync is loose at pauses. Here are two of the four sheets, the worst and one of the two best:
The gotcha that would have wrecked the counts. Our TTS wrapper's words.json originally set each word's end to the end of its last character, and ElevenLabs hangs the silence after a sentence on the punctuation character. So "desk." could "end" most of the way through the pause, the gap shrinks below 0.35 s, and pauses vanish from the sheet. You'd under-count pauses and the model would look better than it is. score-sheet now reads the raw .alignment.json and ends each word at its last letter or digit, and the fix went into the TTS wrapper too.
Calibrating on old clips. Before any new test ran, the tools were checked on the Nov 1 clips, which have no real timestamps. words-approx spreads the words over the audio with pauses after sentence punctuation, and a re-count of the Nov 1 host clips on those approximate times came out InfiniteTalk 3/4, LTX 2/4, S2V 0/4 pauses closed. Those are labelled approximate and are never used in a comparison. They were a tool check, not a result.
The verdict file
Every test ends in a verdict.md: the header (family, prompt, seed, output size), the seven scores each with a one-line note of what was seen, the lip-sync count, the drift call, a "What the frames show" list with times, the measured numbers copied from timing.json, a one-line call (keep, alternate take, reject) and a licence line. A blank or half-filled verdict isn't a scored test.
The reviewer scores one test at a time and never picks a winner. Comparisons happen afterwards, from the verdicts, under the comparison rules.
What the method says so far
From the 30 scored seed-42 tests (framework/SCORECARD.md, rebuilt from the verdicts):
| family | tests | prompt | motion | identity | artifacts | overall | s/s warm |
|---|---|---|---|---|---|---|---|
| Z-Image Turbo stills | 10 | 4.5 | - | - | 4.9 | 4.4 | n/a (a still takes about 9 s warm, test 002) |
| LTX 2.5 text-to-video | 10 | 3.5 | 3.6 | 4.3 | 3.7 | 3.1 | 14.57 |
| LTX 2.5 image-to-video | 10 | 4.0 | 4.0 | 4.6 | 4.5 | 3.9 | 16.17 |
Prompt by prompt on "overall," image-to-video scored higher on 7 of 10, tied on 2 (P02, P03) and lost on 1 (P04, the kickflip, where the image-to-video rider hangs in the air for about 1.8 s). Image-to-video costs about 1.6 s more compute per second of video, warm. The best clip of the night so far is test 027, the cartoon fox, 5 on every scored criterion.
That's a lead, not a result. One seed. The rule says wait for seed 1234 before calling it, and seed 1234 hasn't run.
The interactive scorecard above has all 20 video rows with the evidence line and a frame from each test. Click a column to sort.
Where the method is weak
Every one of these is real, and every post that quotes a score should carry it.
- The scores are a convention. 1 to 5 with anchors is a way to make two reviewers agree, not a measurement. The lip-sync thresholds (0.9 for a 5, 0.6 for a 3) are a choice we made. Treat a one-point gap between models as noise unless both seeds agree.
- One reviewer. Each test is scored once, by an AI reviewer (Claude Opus 5.5) looking at jpgs. The DAISY-1 sheet above got two counts a point or two apart from the same frames. Nobody's measured inter-rater agreement across the suite yet. That's the next thing to add: a second blind scorer on a sample of tests.
- Small n. Two seeds per prompt is the floor, and tonight only one is in. Ten prompts can't cover every shot type.
- Frames, not motion. 12 frames over 5 s is one every 0.45 s. A three-frame flash or a hand that melts between samples gets missed. When a score matters for a post, we grab a denser sheet first.
- Lip-sync is sampled. About 6 pauses and 6-12 speech points per line. A model can be right at every sampled frame and wrong between them. ElevenLabs timestamps are close, not frame-exact, which is why we sample midpoints.
- No audio scoring on text-to-video. LTX 2.5 generates its own ambient audio. We don't score it; the rubric marks audio N/A unless it's a lip-sync test, and audio on lip-sync tests is scored from ffprobe facts and sync evidence, not by listening.
- Language models judge some things worse than others. "Is the mouth open," "are there six fingers," "did the hair change colour": good. Naturalness of motion and how audio sounds: weaker.
- The face crop is fixed per clip. A host who leans out of the crop produces an unfair "mouth not visible" frame. You re-run with a different crop and say so.
- Wall times are this box, tonight. Whatever else was resident is in the number.
None of that makes vibes better. It makes the numbers honest about what they are.
The same idea, applied to the posts
Clips aren't the only thing that gets checked by someone who didn't make them. Every Video Lab post now gets an independent editor pass before it goes live: a fresh AI editor that didn't write the post, with a written brief (framework/EDITOR-BRIEF.md). It looks for leftover placeholders, checks every number against its source file, checks that the film, the instructional, the narration and the article say the same thing, frame-grabs every video, and reports must-fix, should-fix and nice-to-have items. The author fixes every must-fix, then re-runs the automated checks.
The reason is the first post of the run. An editor pass on it, after it went live, found an unfilled template variable (an as-of time) printed on the page, a unit error (ComfyUI's "MB Staged" is really MiB; the post had divided by 1,000 and compared the result with free memory in GiB, so a 37.6 GiB server read as 38.5), claims made stale by later findings, and video captions asserting a cause the article itself called unproven. Every one of those had passed both automated checks, the schema check and the media QA. A script checks that the parts are there. It takes a reader to check that they're right.
On this post the editor found five must-fixes, all after both automated checks had passed. Two were claims that went stale while the post waited for its host renders: that seed 1234 was "queued" (it had been parked) and that the voice-model test was "in the queue" (it had run, with the answer above). One was the host count, where the editor got 6 and the draft said 7 to 9. Two were small numbers: on-camera time quoted as speech time, and a runtime one second off.
Setup and gotchas
- One queue, one lock. Every render goes through a single runner that takes a file lock, so two heavy jobs never overlap and wall times aren't polluted by a neighbour. Before each job the other ComfyUI server is told to free its models.
run-testsamples/proc/meminfoitself. On the GB10,nvidia-smireports memory as N/A, so free system memory is the only real gauge.- This box's ffmpeg has no
drawtextfilter. All sheet labels are drawn with Python's PIL. - The word-end gotcha above: if your TTS gives you word timings, check where it puts the silence after a full stop.
- A second timing gotcha, found by the captions post the same night: on eleven_v3, the timestamp of the word after a pause starts a median 346 ms before the voice actually comes back (40 pauses over 5 clips). That shortens every measured pause and pulls its midpoint early. We snap those word starts to the real end of the silence (ffmpeg
silencedetect) before captioning or counting a v3 line; eleven_v4 was within 37 ms. - Never edit a prompt after tests have run against it. Add a new id.
- Keep tool checks out of
tests/. The Nov 1 calibration lives inframework/checks/nov1/so the test counts stay honest.
Settings
- Prompt suite:
prompts.jsonv1.0, P01-P10, seeds 42 and 1234. - Stills: Z-Image Turbo bf16, 1344x768, 8 steps, res_multistep, shift 3.
- LTX 2.5 text-to-video and image-to-video: 22B distilled int8, 1280x704, 121 frames at 24 fps, 2-stage 8+3 steps, CFG 1.0, image strength 1.0 for image-to-video.
- Wan 2.2 image-to-video (queued): A14B high+low fp8, lightx2v 4-step LoRA, shift 5, 1280x704, 81 frames at 16 fps.
- Lip-sync families: InfiniteTalk 832x480 25 fps; LTX 2.5 image+audio 1280x704 24 fps; Wan 2.2 S2V 832x480 16 fps.
- Score sheet: pause > 0.35 s, sampled at the midpoint; every 4th word a speech point; tail reported separately.
- Identity strip: 8 frames, centre 60 percent, 300 px wide.
- Contact sheet: 12 frames at 480 px.
Speed and memory
The method itself is cheap. On this box the sheets are seconds of CPU per clip. The expensive part is the sweep: at the warm LTX 2.5 rates above, ten 5 s clips are about 12-14 minutes of GPU per seed per family, and the Nov 1 baseline for Wan 2.2 image-to-video was about 66 s of compute per second of video, so the same sweep on Wan is closer to an hour. Lowest free memory during the 30 scored tests: 13 GB (LTX image-to-video block, the per-test minimum in SCORECARD.md), with other agents' CPU renders sharing the pool at the time.
If you want to copy this, start with the prompt suite and the score sheet. Those two give you most of the value: a fair comparison and a count you can argue about with actual frames.