Captions that land on the word: ElevenLabs character alignment into Remotion captions

The short version

When ElevenLabs reads a script it can hand back, next to the audio, a start and end time for every character. Group those characters into words and you have word-timed captions for free. We wanted to know how good "free" is. So we took real ElevenLabs output from this run (13 clips, 218 seconds, 593 words, two models, three cast voices) and captioned it four ways: the ElevenLabs timestamps, words spread evenly across the clip, a length-weighted guess, and Whisper large-v3 running locally. Then we checked all of them against the one thing nobody can argue with: where the audio is actually silent.

Here's what came out:

• eleven_v4 timestamps are good enough to ship as-is. The word after a pause starts a median 37 ms after the audio gets loud (48 pauses, 8 clips), and none of them more than 100 ms early. A word is lit up during 1% of the measured silence.
• eleven_v3 timestamps start the next word early after a pause. A median 346 ms before the audio comes back (40 pauses, 5 clips), and 70% of those starts are more than 100 ms early. On Daisy's intro, "Today" starts 667 ms before she says it. One ffmpeg pass fixes it (below).
• Even spacing drifts with every pause. Median 338 ms off on v4 clips and 600 ms on v3, and on a 48-second line the median word was 2.4 seconds off.
• Whisper is the wrong rescue when you already have the timestamps. large-v3 heard almost every word but started them early: about 200 ms early against the v4 timestamps, and about 415 ms early right after a pause. It took 2.3 seconds of wall-clock time per second of audio (int8, 8 CPU threads).

The house recipe is four steps. Take the ElevenLabs timestamps, end each word on its last letter, snap any word that starts inside a silence forward to where the sound comes back, and render with the Captions template, paged by phrase with the spoken word in the accent color. The last shot of the short film and the captions in the instructional use that recipe, and the two players let you flip between timing sources on a real clip.

Daisy Lane and Theo Lindqvist are AI hosts: AI-generated faces lip-synced with LTX 2.5 on this machine, voiced by ElevenLabs.

How we tested

The audio. Thirteen clips, all made with bin/tts.py (the with-timestamps endpoint, mp3_44100_128) during this run:

• 5 on eleven_v3 (102.5 s, 261 words): three test lines in the Jessica voice at stability 0.5 / style 0 (8.1, 20.4 and 48.3 s), Daisy's intro for this post (13.4 s) and Theo's first line on v3 (12.3 s), both at the cast settings.
• 8 on eleven_v4 (115.8 s, 332 words), all on the cast recipe: Daisy's intro and Theo's first line again, Theo's second line and Daisy's outro, two smoke-test lines (Daisy and Rafael) and the 17.3 and 39.0 s test lines.

So two lines exist on both models, word for word: Daisy's intro (13.36 s on v3, 12.16 s on v4) and Theo's "Picture it like this" line (12.32 s on v3, 12.80 s on v4). That's the cleanest v3 vs v4 comparison we have.

The four timing sources.

1. ElevenLabs alignment. bin/tts.py walks the alignment arrays and groups characters into whitespace-separated words. A word's start is its first letter and its end is its last letter or digit, not its trailing punctuation (more on why in "Setup and gotchas").
2. Even spacing. N words spread evenly from 0 to the end of the file. This is what you get from a caption tool that only knows the text and the duration.
3. Length-weighted guess. bin/words-approx, the run's fallback for audio with no timestamps: word lengths proportional to their letters, with fixed pauses after sentence ends and commas, scaled to fit the file.
4. Whisper large-v3. faster-whisper 1.2.1 with the Systran large-v3 weights, int8 on the CPU (8 threads, beam 5, word_timestamps=True). It transcribed the audio without being given the script, which is how most people use it. Its words were matched to the script with a sequence match on normalized tokens. It heard 591 words, 567 of the 593 script words matched, and the misses were things like "Hiii" (heard "Hi"), "queue" (heard "cue") and "ElevenLabs" (heard "Eleven Labs").

Drift against the alignment. For each source and each script word: start error in ms against the alignment, as median, 95th percentile and worst. Plus "right word lit": the share of 10 ms ticks inside a word during which that source highlights the same word.

The check that trusts nobody's timestamps. ffmpeg silencedetect (-35 dB, at least 0.2 s, touching pieces merged) finds the pauses inside the speech. At each pause we compare where each source says the next word starts with the moment the audio gets loud again, and where the previous word ends with the moment it went quiet. We also measure the share of silence during which a source has some word lit. This only tests word edges next to a pause (88 of them). Boundaries in the middle of a phrase have no silence to anchor to, so they are not checked against the audio here.

The per-clip numbers, the audio check and the snap counts are in the summary JSON linked at the bottom of this page. The scripts (measure.py, analyze.py, snap_words.py) and the every-word drift.json stay in our run folder.

Drift against the ElevenLabs timestamps

Median of the per-clip medians, in milliseconds. "p95" is the median of the per-clip 95th percentiles; "worst" is the single worst word.

Per clip (start error vs the alignment, median / p95 / worst in ms):

Read that whisper column carefully. Against v3 it looks better than against v4, and the signed column says why: Whisper always starts words early, and so does v3 after a pause, so the two agree with each other. That's why you check against the audio.

Checked against the audio

The same two lines on both models, next word after a pause vs the audio (median): Daisy's intro -456 ms on v3 and +4 ms on v4; Theo's line -579 ms on v3 and -9 ms on v4. Same voice, same words, same settings, but v3 also held the written pauses longer (0.8 to 0.9 s in Daisy's line, against about 0.5 s on v4, which caps pauses near 0.55 s). So we also split by pause length. On pauses of 0.2 to 0.6 s, the only lengths both models make, v3 is a median 111 ms early (19 pauses) and v4 37 ms late (48). On v3's longer pauses it is 584 ms early (21). The model matters, and on v3 the error grows with the pause. We could not test v4 on long pauses because it doesn't make them.

One example you can see in the film: in Daisy's v3 intro the audio sits at -50 to -60 dB from 8.58 s to 9.50 s, and "Today" becomes audible at about 9.50 s. The v3 timestamps say "Today" runs from 8.84 to 9.84 s: one second for a two-syllable word, with the pause folded into its front.

Two more things the timestamps show:

• eleven_v4 times word starts in 40 ms steps; v3 does not. 99.9% of v4 word starts sit on a 40 ms grid (within 2 ms); v3's land on it 37% of the time. On both models, inside a word, most letters get an equal share of the word's time, so treat the word as the unit, not the letter.
• Punctuation eats the pause. ElevenLabs hangs the silence after a sentence (and all of a "...") on the punctuation characters. If you end a word on its last character including punctuation, a word is lit during 62% of the silence on v4 and 82% on v3. Ending on the last letter takes v4 down to 1%.

The fix for v3: snap starts out of silences

snap_words.py (in tests/captions-drift/): run ffmpeg silencedetect on the mp3 (-35 dB, at least 0.2 s, touching pieces merged). For each silence, the first word whose last letter ends after the silence gets start = max(start, silence_end - 0.03 s). Nothing else moves. The 30 ms lead lets the word light a hair before the sound, which reads as on time.

On the five v3 clips it moved 36 word starts, by a median of 351 ms and at most 883 ms, and took the lit-in-silence share from 53% to 5%. On the eight v4 clips it moved one word, by 10 ms. So it's safe to run on everything, and it only does real work on v3.

Be clear about what that number is: the snap uses the same silence detector we measure with, so "5% lit in silence" shows it did its job. It doesn't independently prove the new starts are right. What does: the waveform example above, and the fact that v4's own timestamps land where the snap puts v3's.

Page by phrase vs word highlight

The house Captions template (remotion/src/templates/Captions.tsx) pages the words into short lines. A page holds at most 7 words. A new page starts at a gap over 0.7 s, after 3.5 s, after a word ending in ., ? or !, or when the text would run past two lines. Each page holds up to 0.6 s past its last word, or until the next page starts, so it doesn't flicker. With "word highlight" the word being spoken is bold and pink; with "page by phrase" the page just sits there.

In our read (not measured), paging forgives small errors: only the page flips need to be right, and a page that turns 300 ms early during a pause reads fine. A 667 ms miss on a flip does not. In the film's third shot the raw v3 page with "Today" comes up 0.66 s before she says it. Word highlight shows every error: the even-spaced highlight runs visibly ahead and behind, and Whisper lights each word a beat before the voice gets there. The page-flip error (first word of each page) tells the same story as the word numbers: median 97 ms for Whisper on v3, 345 ms on v4; even spacing 507 ms on v3 and 265 ms on v4.

Our call: word highlight with the snapped ElevenLabs timestamps for host segments, because that's what makes captions feel attached to the face. Page by phrase for anything where you only have guessed timings.

Setup and gotchas

• Use alignment, not normalized_alignment, for captions that match your script. normalized_alignment times the spoken expansion ("2026" becomes "twenty twenty-six"), which won't match the text you show.
• The with-timestamps endpoint worked for both models on this account. No fallback fired in any call this run (tts.py falls back v4, then v3, then multilingual_v2 on a 400/404/422).
• faster-whisper's audio decoder broke in our venv (av.open() got an unexpected keyword argument 'metadata_errors': a newer PyAV than faster-whisper 1.2.1 expects). Decoding with ffmpeg to 16 kHz mono float and passing the array works.
• Whisper was not given the script. We didn't test forced alignment (ElevenLabs has a /v1/forced-alignment endpoint, billed like speech-to-text; WhisperX isn't installed here), and we didn't test Whisper with the script as an initial prompt. Those are the next tests if you have audio without timestamps.
• Whisper also invents words on longer audio. We used it to check this page's 290-second narration against the script. With faster-whisper's default condition_on_previous_text=True it inserted three phrases nobody said ("the one thing that doesn't work", "hangs out with us to use", and a whole repeated sentence in place of "about 1% of the silence"). A second pass with that option off heard the script as written. As a caption source, that's text on screen that was never spoken.
• The YouTube captions come from the same words. bin/youtube-package builds captions.en.srt from each block's words.json with the same paging rules as the template, so the site and YouTube match.
• Loudness normalization doesn't move the timestamps. tts.py --loudnorm re-encodes through ffmpeg loudnorm; durations stayed identical in earlier measurements this run.

Settings

• ElevenLabs: POST /v1/text-to-speech/{voice_id}/with-timestamps?output_format=mp3_44100_128. Cast settings from cast.json: Daisy (Jessica voice) stability 0.45, similarity 0.75, style 0.3; Theo stability 0.55, similarity 0.75, style 0.15; speaker boost on; then ffmpeg loudnorm I=-16. Test lines in the Jessica voice: stability 0.5, similarity 0.75, style 0.
• Words: bin/tts.py words_from_alignment: start = first letter or digit, end = last letter or digit; end_punct kept separately.
• Snap: snap_words.py, silencedetect n=-35dB:d=0.2, merge pieces closer than 50 ms, lead 30 ms.
• Whisper: faster-whisper 1.2.1, ctranslate2 4.8.4, Systran/faster-whisper-large-v3, int8, CPU, 8 threads, beam 5, language en, word_timestamps=True, no VAD filter, no initial prompt.
• Remotion 4.0.539: Captions template, style bar, 46 px Inter, max 7 words per page. The comparison renders and the player use a new composition, CaptionLab (remotion/src/players/CaptionLab.tsx), with the same paging.
• Host clips: LTX 2.5 audio-to-video from each host's on-set still, 1280x704, 24 fps, seed 42, through bin/host-segments; the clean ElevenLabs mp3 is muxed over the result.

Speed and memory

• ElevenLabs latency was not logged for this post (framework/TTS.md measured 4.8 s for 17 s of v4 audio and 8.4 s for 39 s). 2,939 characters were sent for this post's host, VO and comparison lines, plus the narration.
• Grouping characters into words and snapping: milliseconds. silencedetect on a 48 s file: well under a second.
• Whisper large-v3 int8 on 8 CPU threads: 506 s of wall-clock time for 218 s of audio (2.3 s per second overall; 0.85 to 4.6 per clip, slower on short clips), plus 3.8 s to load the model.
• Remotion: the 58.8 s film took 87.7 s of wall time plus 10.3 s to encode, and the 175.5 s instructional 273.6 s plus 24.9 s, all on the CPU at concurrency 6. The AI host clips took 856 s of GPU for 49.4 s of video (17.3 s per second, LTX 2.5 audio-to-video).
• Memory: not measured for these CPU jobs.