We planted seven faults in our own AI film. Our QA gate stopped three.

Rated PG-13crime scene

An automated gate is the only review a Video Lab post gets. Here's its two-week record, the faults we hid in a short film to test it, the three that walked right past it, and what we fixed the next day.

Instructional (about 3 minutes): Sloane Mercer, an AI host with an AI voice (ElevenLabs), on the gate's record, the planted-fault test and what got through. Music: ElevenLabs Music, AI-generated.
Late Checkout, a short film (37 s): a detective walks an empty motel room the morning after. Marcus Bell is an AI host. Picture and music are AI-generated.
Prompt and settings

Prompt: s01: He steps two paces into the room and stops, looking around. The door stays completely still. Static camera. s02: He looks slowly across the room and takes one slow breath. Small natural movement only. Steady camera. s03: The warm flashlight beam slides slowly down the edge of the frame and stops on the pry mark. Nothing else in the frame moves. Static camera. s04: Slow push-in toward the phone. Nothing moves on the nightstand. s05: He stays crouched and moves the flashlight beam slowly under the bed. Steady camera. s06: The flashlight beam settles on the key card. The card stays still. Static camera. s07: He turns his head slowly to look back over his shoulder toward the door. Steady camera. s08b: Very slow push-in toward the bed. Nothing in the room moves. Faint dust drifts in the window light.

Settings: Stills: FLUX.2 klein 4B from reference sheets, 1280x704. Takes: LTX 2.5 22B distilled int8, image-to-video from each locked still, 1280x704, 24 fps, CFG 1. Model audio muted; one ElevenLabs Music bed.

Missed on Thursday's gate: a different AI host in a different room, planted for 3 s. No flag then; since Friday's fix, the face check flags it.
Prompt and settings

Settings: A planted copy of Late Checkout, made with ffmpeg from the finished film. Same length and frame rate as the clean film.

Blocked: a planted sign misspelled VACANCV. The vision model read the word.
Prompt and settings

Settings: A planted copy of Late Checkout, made with ffmpeg from the finished film. Same length and frame rate as the clean film.

Missed on Thursday's gate: the picture freezes for 1.5 s under a moving caption. Since Friday's fix, it's flagged.
Prompt and settings

Settings: A planted copy of Late Checkout, made with ffmpeg from the finished film. Same length and frame rate as the clean film.

Verdict

An automated gate is worth running, but only if you know what it can't see. On Thursday ours blocked three of seven planted faults: a jump in the music, a misspelled sign and an audio dropout. It warned on half a second of black. It missed a 1.5 second freeze, a click, and a different person on screen for three seconds. On Friday we fixed the freeze, black, click and face checks, and all seven copies now get flagged. White flashes of 1 to 4 frames still get through. Trust the measured checks, treat the vision model as a lead, and plant faults in your own pipeline before you rely on it.

Credits and settings

WorkflowAn automated QA gate for AI video
MakerLimited Edition Jonathan Video Lab
VersionVideo Lab gate as of Nov 20 2026
What it doesquality control
LicenseThe film uses LTX 2.5 under the LTX-2.x Community License. The gate's scripts are not released.
HardwareDGX Spark: NVIDIA GB10, ARM (aarch64), 128 GB unified memory. ComfyUI 0.43.0, ffmpeg, Chrome headless, faster-whisper large-v3 on CPU.
Tested onNov 18, 2026
AI hosts

Sloane Mercer (AI host; AI voice (ElevenLabs), AI voice)

Marcus Bell (AI host; AI voice)

Narration
Not supplied
Music source
ElevenLabs Music, AI-generated

What it does well

  • Measured audio faults are caught reliably: a +9 dB music jump and a 1.2 s dropout both blocked the post.
  • The vision model read a planted misspelled sign word for word.
  • On the film's first clean cut, the soundtrack check found a real 4.1 LU jump in the music.
  • The vision model flagged a real continuity error in four runs out of four.

Where it breaks

  • On Thursday's gate, a different host in a different room for 3 s raised no flag. Friday's face check flags it.
  • On Thursday's gate, freezes of 1.5 s and under weren't reported and a 3 s freeze only warned. Since Friday, a freeze from 0.5 s inside a shot is flagged.
  • White flashes of 1, 2 and 4 frames went unnoticed.
  • A 2 ms click went unnoticed on Thursday. Friday's click check catches it in music, but not under speech.
  • The vision model's flags change from run to run: five runs on the same clean film gave three different answers, and it kept flagging the phone after we fixed it.
  • Whisper large-v3 invented 119 words on 50 seconds of music with no voice in it.

Nobody watches a Video Lab post before it goes up. Since Nov 2, an automated gate is the only review a post gets. So we tested the gate the only fair way: we made a short film, broke copies of it on purpose, and ran every copy through.

It stopped three of the seven faults we planted. It warned on one. Three got through, including a different person standing in a different room for three seconds. The next day we fixed the checks behind those four, and the same broken copies now all get flagged.

What the gate is

The gate is a set of scripts that runs on the same DGX Spark that makes the videos. It has six parts:

  • Page: opens the post in Chrome at phone and desktop width, plays every clip and watches for crashes.
  • Copy: reads the record against our rules: ratings, privacy, AI disclosure, galleries, mistakes shown.
  • Picture and loudness: ffmpeg measures freezes, black frames, audio drift, loudness and clipping.
  • Soundtrack: looks for level jumps, holes and dropouts in the music.
  • Speech: Whisper transcribes every voice line and compares it with the script.
  • Vision and faces: Claude Haiku 5.5 reads contact sheets of every shot, and a face-recognition model compares faces with the film's references.

A failed check stops the post. A vision flag needs a ruling from our reviewing agent, Claude Opus 5.5, before the post can pass.

Two weeks of its record

From the fault logs for Nov 2 to Nov 15:

Log entries
Faults the gate caught8
Faults the gate missed23
False alarms32
Notes17

Don't read that as a catch rate. Faults fixed in the middle of a run rarely got a line in the log, so it leans toward trouble. That's why we planted faults: with a planted fault, you know the right answer before the test starts.

The misses in that log were real. One film had a woman holding two lit cigarettes in one hand. Another had a locked still with an older man's face where our reference was a woman. Our face check seemed to catch that one with a low score, and we logged it as a win. Three days later we found the score came from a patch of harbor water. On his real face the check passed.

The planted-fault test

Late Checkout is a 37-second film about a detective walking an empty motel room. We made one copy per fault and ran the gate's picture, soundtrack, text, face and vision checks on each. The clean film went through too, as the control.

Planted faultWhat the gate did on ThursdayHow
Music 9 dB louder from one cut onwardBlockedLevel step of 8.8 LU at 22 s
A misspelled VACANCV sign for 3 sBlockedThe vision model read the word
Audio drops out for 1.2 sBlockedSilence from 30.02 to 31.21 s
0.5 s of blackWarningMeasured, but a warning doesn't stop a post
1.5 s freeze under a moving captionMissedNothing reported
A 2 ms click in the musicMissedNo check listens for clicks
A different host in a different room for 3 sMissedNo vision flag, and the face warnings looked like the clean film's

The swapped host is the one that should worry anyone building one of these. He was on the contact sheet the vision model read. It reported nothing. The face check did warn on that shot, but it also warns on the clean film whenever the detective turns his head, so the warning told us nothing.

How short a glitch gets through

We planted freezes and white flashes of different lengths in one shot.

GlitchThursdayAfter Friday's fix
Freeze of 0.25 sMissedWarning
Freeze of 0.5, 1 or 1.5 sMissedFlagged
Freeze of 3 sWarningFlagged
Freeze of 6.5 sBlockedBlocked
White flash of 1, 2 or 4 framesMissedMissed

What we changed the next day

Friday is when we fix the gate with whatever the week's tests showed. This Friday we went after the three misses and the warning.

  • Freezes: the old check didn't report a held frame shorter than 2 seconds. The new one ignores the caption band at the bottom of the frame and flags a held frame from half a second inside a shot.
  • Black: half a second of black inside a shot is now a flag, not a warning.
  • Clicks: a new check listens for a sudden spike in the audio that's gone within 5 milliseconds.
  • Faces: the face check now skips profiles and faces too small to judge, so it stopped warning on the clean film, and a face that doesn't match on a film is a flag.

We ran the same seven copies again, and all seven raised a flag. A post with a flag can't pass until our reviewing agent rules on it. The white flashes still get through. So does a click hidden under speech: in a voice track it looks just like a hard consonant.

A fix that blocks good posts is its own failure, so we ran it the other way too. Across 184 films and audio files from the last three weeks, the new click check found one borderline spike, below the line that raises a flag. Two posts we published last week still pass.

Why you can't trust one pass of a vision model

We showed Haiku the same clean film five times. Three runs raised nothing. Two flagged the phone on the nightstand, and it really had moved: front and center in the close-up, tucked by the lamp in the wide shots. So we moved it to the lamp in every shot and ran Haiku five more times. Two runs raised nothing. One flagged the phone again. Two flagged the wall color in the doorway shot, which really is off.

Read the wording and you can see what happened. Every phone flag, before and after the fix, said the phone was missing from another shot. None said it had moved. Haiku was right about the shot and wrong about the reason, and fixing the real fault didn't make the flag go away. A single clean pass from a vision model tells you less than it feels like it does.

Speech checks have their own trap. We ran Whisper large-v3 on our music bed, which has no voice in it, and it invented 119 words in 50 seconds, mostly "If you have any questions or other problems, please post them in the comments." On 10 seconds of pure silence it heard "Thank you." Check your transcription tool on audio with no speech before you trust its miss rate.

What it caught in our own film

The gate paid for itself before we planted anything. On the first clean cut, the soundtrack check measured a 4.1 LU jump in the music at 26 seconds, so we leveled the bed. The vision model then flagged an evidence marker on the nightstand that no other shot shows. It flagged that in four runs out of four, so we remade the shot.

It also missed two. A second look by a different model, Gemini, found the door shut in the crouching shot and latched in the close-up of the pried lock, though the opening shot shows it standing open and our one rule for the room was that it never moves. Our still check and take check had passed both, because they compare each shot with its own prompt, not with the other shots. We cropped the door out of the crouching shot and remade the close-up. That took two stills: with the door itself in frame, LTX swung it shut in both takes, so the close-up now shows only the open frame.

It also cried wolf while we built the film. The first-pass take review failed 27 of 32 reviews, and several of those were smooth camera moves it read as jumps. We measured the frame-to-frame change before overruling it. If you use a vision model as a gate, budget for ruling on its flags.

What to do with this

  1. Plant faults in your own pipeline. A check that has never caught a planted fault hasn't earned your trust.
  2. Run a clean control every time, so you know what your checks complain about when nothing is wrong.
  3. Measure what you can. Loudness, dropouts and black frames were the checks we could count on.
  4. Treat a vision model as a lead, and rule on what it flags.

Next on our list: the white flashes, and a check that compares every shot in a one-room film against the others. That's the one that would have caught the door.

How we tested

Each planted copy was made with ffmpeg from the finished film, with one fault per copy. Each went through the same scripts the live gate uses. The page, copy and speech checks need a published post, so they didn't run on these film copies. Each copy took 72 to 146 seconds on the Spark.

Setup and gotchas

  • Vision flags change from run to run on the same picture, so compare against a clean control before you call something caught.
  • ffmpeg's freeze detection can be fooled by anything moving on screen, like a sliding caption.
  • A face score means little until you look at what the detector actually boxed.

Settings

Film: FLUX.2 klein 4B stills from reference sheets, LTX 2.5 22B distilled int8 image-to-video at 1280x704, 24 fps, seeds picked per shot, model audio muted, one ElevenLabs Music bed. Gate vision pass: Claude Haiku 5.5. Speech: faster-whisper large-v3 on CPU. Faces: YuNet and SFace.

Speed and memory

A gate pass on one film copy took 72 to 146 seconds. We didn't measure the gate's peak memory.

ComfyUI API graph for Late Checkout, shot 1 ↗