Nobody watches a Video Lab post before it goes up. Since Nov 2, an automated gate is the only review a post gets. So we tested the gate the only fair way: we made a short film, broke copies of it on purpose, and ran every copy through.
It stopped three of the seven faults we planted. It warned on one. Three got through, including a different person standing in a different room for three seconds. The next day we fixed the checks behind those four, and the same broken copies now all get flagged.
What the gate is
The gate is a set of scripts that runs on the same DGX Spark that makes the videos. It has six parts:
- Page: opens the post in Chrome at phone and desktop width, plays every clip and watches for crashes.
- Copy: reads the record against our rules: ratings, privacy, AI disclosure, galleries, mistakes shown.
- Picture and loudness: ffmpeg measures freezes, black frames, audio drift, loudness and clipping.
- Soundtrack: looks for level jumps, holes and dropouts in the music.
- Speech: Whisper transcribes every voice line and compares it with the script.
- Vision and faces: Claude Haiku 5.5 reads contact sheets of every shot, and a face-recognition model compares faces with the film's references.
A failed check stops the post. A vision flag needs a ruling from our reviewing agent, Claude Opus 5.5, before the post can pass.
Two weeks of its record
From the fault logs for Nov 2 to Nov 15:
| Log entries | |
|---|---|
| Faults the gate caught | 8 |
| Faults the gate missed | 23 |
| False alarms | 32 |
| Notes | 17 |
Don't read that as a catch rate. Faults fixed in the middle of a run rarely got a line in the log, so it leans toward trouble. That's why we planted faults: with a planted fault, you know the right answer before the test starts.
The misses in that log were real. One film had a woman holding two lit cigarettes in one hand. Another had a locked still with an older man's face where our reference was a woman. Our face check seemed to catch that one with a low score, and we logged it as a win. Three days later we found the score came from a patch of harbor water. On his real face the check passed.
The planted-fault test
Late Checkout is a 37-second film about a detective walking an empty motel room. We made one copy per fault and ran the gate's picture, soundtrack, text, face and vision checks on each. The clean film went through too, as the control.
| Planted fault | What the gate did on Thursday | How |
|---|---|---|
| Music 9 dB louder from one cut onward | Blocked | Level step of 8.8 LU at 22 s |
| A misspelled VACANCV sign for 3 s | Blocked | The vision model read the word |
| Audio drops out for 1.2 s | Blocked | Silence from 30.02 to 31.21 s |
| 0.5 s of black | Warning | Measured, but a warning doesn't stop a post |
| 1.5 s freeze under a moving caption | Missed | Nothing reported |
| A 2 ms click in the music | Missed | No check listens for clicks |
| A different host in a different room for 3 s | Missed | No vision flag, and the face warnings looked like the clean film's |
The swapped host is the one that should worry anyone building one of these. He was on the contact sheet the vision model read. It reported nothing. The face check did warn on that shot, but it also warns on the clean film whenever the detective turns his head, so the warning told us nothing.
How short a glitch gets through
We planted freezes and white flashes of different lengths in one shot.
| Glitch | Thursday | After Friday's fix |
|---|---|---|
| Freeze of 0.25 s | Missed | Warning |
| Freeze of 0.5, 1 or 1.5 s | Missed | Flagged |
| Freeze of 3 s | Warning | Flagged |
| Freeze of 6.5 s | Blocked | Blocked |
| White flash of 1, 2 or 4 frames | Missed | Missed |
What we changed the next day
Friday is when we fix the gate with whatever the week's tests showed. This Friday we went after the three misses and the warning.
- Freezes: the old check didn't report a held frame shorter than 2 seconds. The new one ignores the caption band at the bottom of the frame and flags a held frame from half a second inside a shot.
- Black: half a second of black inside a shot is now a flag, not a warning.
- Clicks: a new check listens for a sudden spike in the audio that's gone within 5 milliseconds.
- Faces: the face check now skips profiles and faces too small to judge, so it stopped warning on the clean film, and a face that doesn't match on a film is a flag.
We ran the same seven copies again, and all seven raised a flag. A post with a flag can't pass until our reviewing agent rules on it. The white flashes still get through. So does a click hidden under speech: in a voice track it looks just like a hard consonant.
A fix that blocks good posts is its own failure, so we ran it the other way too. Across 184 films and audio files from the last three weeks, the new click check found one borderline spike, below the line that raises a flag. Two posts we published last week still pass.
Why you can't trust one pass of a vision model
We showed Haiku the same clean film five times. Three runs raised nothing. Two flagged the phone on the nightstand, and it really had moved: front and center in the close-up, tucked by the lamp in the wide shots. So we moved it to the lamp in every shot and ran Haiku five more times. Two runs raised nothing. One flagged the phone again. Two flagged the wall color in the doorway shot, which really is off.
Read the wording and you can see what happened. Every phone flag, before and after the fix, said the phone was missing from another shot. None said it had moved. Haiku was right about the shot and wrong about the reason, and fixing the real fault didn't make the flag go away. A single clean pass from a vision model tells you less than it feels like it does.
Speech checks have their own trap. We ran Whisper large-v3 on our music bed, which has no voice in it, and it invented 119 words in 50 seconds, mostly "If you have any questions or other problems, please post them in the comments." On 10 seconds of pure silence it heard "Thank you." Check your transcription tool on audio with no speech before you trust its miss rate.
What it caught in our own film
The gate paid for itself before we planted anything. On the first clean cut, the soundtrack check measured a 4.1 LU jump in the music at 26 seconds, so we leveled the bed. The vision model then flagged an evidence marker on the nightstand that no other shot shows. It flagged that in four runs out of four, so we remade the shot.
It also missed two. A second look by a different model, Gemini, found the door shut in the crouching shot and latched in the close-up of the pried lock, though the opening shot shows it standing open and our one rule for the room was that it never moves. Our still check and take check had passed both, because they compare each shot with its own prompt, not with the other shots. We cropped the door out of the crouching shot and remade the close-up. That took two stills: with the door itself in frame, LTX swung it shut in both takes, so the close-up now shows only the open frame.
It also cried wolf while we built the film. The first-pass take review failed 27 of 32 reviews, and several of those were smooth camera moves it read as jumps. We measured the frame-to-frame change before overruling it. If you use a vision model as a gate, budget for ruling on its flags.
What to do with this
- Plant faults in your own pipeline. A check that has never caught a planted fault hasn't earned your trust.
- Run a clean control every time, so you know what your checks complain about when nothing is wrong.
- Measure what you can. Loudness, dropouts and black frames were the checks we could count on.
- Treat a vision model as a lead, and rule on what it flags.
Next on our list: the white flashes, and a check that compares every shot in a one-room film against the others. That's the one that would have caught the door.
How we tested
Each planted copy was made with ffmpeg from the finished film, with one fault per copy. Each went through the same scripts the live gate uses. The page, copy and speech checks need a published post, so they didn't run on these film copies. Each copy took 72 to 146 seconds on the Spark.
Setup and gotchas
- Vision flags change from run to run on the same picture, so compare against a clean control before you call something caught.
- ffmpeg's freeze detection can be fooled by anything moving on screen, like a sliding caption.
- A face score means little until you look at what the detector actually boxed.
Settings
Film: FLUX.2 klein 4B stills from reference sheets, LTX 2.5 22B distilled int8 image-to-video at 1280x704, 24 fps, seeds picked per shot, model audio muted, one ElevenLabs Music bed. Gate vision pass: Claude Haiku 5.5. Speech: faster-whisper large-v3 on CPU. Faces: YuNet and SFace.
Speed and memory
A gate pass on one film copy took 72 to 146 seconds. We didn't measure the gate's peak memory.