Video Lab · memory management
121 GB, one pool, no OOM error: how we run 22B and 14B video models back to back on a DGX Spark without freezing it
Rated G
The GB10 has no clean out-of-memory error, the box just stops. Here is what a night of back-to-back LTX 2.5, Wan 2.2, InfiniteTalk and SeedVR2 jobs looks like next to an 18.7 GB voice model and a second client's ComfyUI, with every model's measured weight and the lowest free memory each job reached.
Listen
Narrated by Rafael Duarte (AI voice, ElevenLabs)
Transcript
Hey, it's Rafa. This one's about memory. How we ran a twenty-two billion parameter video model, two fourteen-billion experts, a lip-sync stack and an upscaler, back to back, all night, on one box... without freezing it. Here's the thing about the DGX Spark. It has one pool of memory. One hundred twenty-one point seven gigabytes. The CPU uses it, the GPU uses it, the desktop uses it. ComfyUI's startup line even says it twice: total VRAM, one two four six one zero megabytes... total RAM, same number. Same pool. And when that pool runs out, there is no out-of-memory error. The box freezes. We've never caught a freeze in a log, because when it happens, nothing gets written. What we have is one near miss, and the rules we built around it. Okay, numbers. At three thirteen in the morning, before the first job, we read the meters. Thirty-three point eight gigs available. Eighty-seven used. Where did it go? The voice model, eighteen point seven gigs, resident all night. The LTX server, thirty-seven and a half, with the whole LTX two point five stack staged. And the video-lab server, which looked idle... three hundred sixty-seven megabytes on the GPU... but fourteen point seven gigs of system RAM. We figured that was parked weights from its last Wan job. Turns out, not quite. Hold that thought. Swap? Sixteen gigs, five megabytes free. Thirty-seven days of uptime filled it, and it never drained. That swap is already spent. Now the models. ComfyUI prints a line every time it stages weights, and those lines are the budget. LTX two point five, thirty-five point nine gigs. Wan two point two image-to-video, two experts plus the text encoder, thirty-three point one. InfiniteTalk, twenty-seven point nine. S2V, twenty-two point three. And SeedVR2, the upscaler... three point seven. You see it. Thirty-three point one gigs of Wan into thirty-three point eight gigs of room. Seven hundred megs to spare, before it computes a single frame. It doesn't fit next to a resident LTX. So here's what we do, in the order a job meets it. One. One lock. Every render goes through a script called gpu-run. It takes a file lock, runs the job, and while the job runs it reads free memory every two seconds and keeps the lowest number. That minimum goes on the end line of the log. Most of tonight's numbers come from that log. Two. Free the other server. Every job starts with one curl: post, slash free, unload models true. Tonight, the very first one of those calls came with nineteen gigs back. And that fourteen point seven gigs of RAM? Still there afterwards. So the call releases something real, but not that RSS. Later we tried a restart: fourteen gigs down to one, and free memory went from eighteen to thirty-one. Now the queue restarts that server whenever it's sitting idle and fat. And the same call the other way, to the LTX server, is what's supposed to make room for Wan. We'll know when the first Wan job runs. Three. Batch by family. Reloading the Wan pair costs about two and a half minutes of "model initializing"... we timed it in the server log. Ten Wan jobs in a row pay that once. Four. Chunk SeedVR2. Always. This is the near miss. On November first we fed it a three-hundred-and-two frame clip in one pass. Three point two gigs of weights. Free memory went to zero. Something hit interrupt... the guard or a person, the logs don't say which... and the box lived. The weights were never the cost; the activations were. Now the graph cuts the clip into twenty-one frame chunks, and a four-second clip upscales in three hundred six seconds. Five. Lip-sync windows under twenty seconds. InfiniteTalk costs about seventy-two seconds of compute per second of output. A forty-five second take is an hour you can't interrupt without losing it. Six. The tripwire. A twelve-line script called memguard runs beside the long jobs. Every two seconds it reads free memory. Under five gigs, it posts interrupt to the video-lab server and logs it. As of tonight's start, that log file doesn't exist... which means it has never had to fire. And one flag: the video-lab server starts with reserve-vram eight. ComfyUI says that holds eight gigs back for the OS. Does it change anything? Both servers print the same headroom line at startup, with or without it. Honestly... not measured. Tonight's results, so far. The first three jobs under the lock made ten stills and twenty LTX clips. During the text-to-video clips, the lowest free memory was twenty-three gigs, on the skateboard and the wet-dog prompts. During the image-to-video clips, same model, same size, it dropped to thirteen. Memguard hasn't fired... but to be fair, it only runs beside the video-lab server, and that server hadn't had a job yet. And here's the one that surprised me. The rule for CPU work says check free memory before a render, but only during the big Wan and SeedVR2 jobs. Tonight, during an ordinary LTX job, other workers' Remotion renders were holding eight gigs, fourteen agent sessions almost four more, and a transcription job two and a half. Nothing went through the lock, and free memory still dropped. So the rule's getting wider: every heavy job, not just the scary ones. The heavier families, Wan, InfiniteTalk and SeedVR2, run later tonight. Their numbers go into the table on the page as they land. Next: try the vram-headroom flag and see if the minimum moves. Log memory pressure next to it, because stall time rises before free space hits zero. And test the seven-billion SeedVR2, chunked, with the guard running, before it goes near a queue. That's the post. Every number has a file name next to it. Runs local. No per-clip bill for the video. I'm happy.
Prompt and settings
Settings: Remotion ShortFilm, 1920x1080, 30 fps. Clips: LTX 2.5 22B distilled at 1280x704, 121 frames, seed 42, tests 012, 013, 014, 019, 020 (text-to-video) and 023, 025 (image-to-video from Z-Image Turbo stills), their own audio muted
Prompt and settings
Settings: Four on-camera segments (63.2 s of speech) made with LTX 2.5 image+audio-to-video at 1280x704 from each host's on-set still, seed 42, voices by ElevenLabs (eleven_v3 for Daisy's three lines with pauses, eleven_v4 for Rafa); Daisy's closing line was re-recorded at 05:00 without a pet name; voice-over panels in Remotion ExplainerScene; 1920x1080, 30 fps
Prompt and settings
Prompt: A skateboarder in a grey hoodie rides toward the camera on a concrete plaza and pops a kickflip, the board spinning under his feet before he lands and rolls past. Tracking camera, bright overcast daylight, photorealistic, sharp.
Settings: LTX 2.5 22B distilled int8, 1280x704, 121 frames at 24 fps, seed 42, :8192 freed before the job
Verdict
On a unified-memory box, treat free memory as the only gauge and run one heavy job at a time: a flock around every render, a POST /free to the other ComfyUI server before each job, SeedVR2 chunked at 21 frames, lip-sync windows under 20 s, and a 2-second poll that interrupts the job under 5 GB MemAvailable. With LTX 2.5 resident (37.6 GB) and the voice model up (18.7 GB), the box had 33.8 GB to work with, and the Wan 2.2 I2V pair alone stages 33.1 GB before a single activation, so the /free between families is the step that makes the night possible. An idle server's leftover host RAM is a different matter: /free did not return the 14 GB the idle :8192 held, and a restart did (MemAvailable 17.9 to 31.2 GB), so the queue now restarts it automatically. The one near miss we have on record was SeedVR2, the smallest set of weights on the box, at 0 GB free on an unchunked 302-frame pass. As of 04:49 ET, mid-run, the lowest free memory of the night was 8 GB, during the AI-host takes for this very post, while CPU renders and agents outside the lock held about 18 GB. So the rule we are adding: CPU work waits for 20 GB during any GPU job, not just Wan and SeedVR2.
| Workflow | Memory discipline for local video generation on a unified-memory box |
|---|---|
| Maker | LEJ Video Lab (house workflow); models by Lightricks, Alibaba Wan team, MeiGen-AI, ByteDance Seed, Tongyi-MAI, hzwer |
| Version | ComfyUI 0.43.0, comfy-aimdo 0.6.0 (DynamicVRAM), two servers (:8191 LTX; :8192 video lab, --reserve-vram 8); house scripts gpu-run, queue runner, memguard.sh, as of 2026-11-02 |
| What it does | memory management |
| License | House scripts, no license. Models in the run: LTX-2.x Community License (LTX 2.5), Apache-2.0 (Wan 2.1 and 2.2, InfiniteTalk, SeedVR2, Z-Image Turbo, umT5), MIT (RIFE v4.26, CLIP ViT-H); chinese-wav2vec2-base (license not checked) |
| Hardware | DGX Spark: NVIDIA GB10, ARM (aarch64), 128 GB unified memory (121.7 GiB MemTotal), 16 GB swap file full at the start of the night, driver 580.173.02, CUDA 13.0. ComfyUI 0.43.0, PyTorch 2.11.0+cu130, int8 LTX 2.5 transformer and text encoder, fp8 Wan 2.1/2.2, int8 SeedVR2, bf16/fp16 VAEs. Resident alongside the tests: an 18.7 GB voice model and a second client's ComfyUI. |
| Peak memory | 113.7 GB. Whole-box figure: 121.7 GB MemTotal minus the lowest MemAvailable that bin/gpu-run sampled (every 2 s) during the job. Includes the 18.7 GB resident voice model and whatever the other ComfyUI server kept. GPU-side counters are N/A on the GB10. |
| Tested on | Nov 2, 2026 |
What it does well
- One flock around every render means two agents can never start two 30 GB jobs at once, no matter how many are writing to the queue
- gpu-run writes MemAvailable at start and the minimum during every job, so each family's real headroom is a log line, not a guess
- A POST /free with unload_models before each job is meant to empty the resident LTX server (37.6 GB) for the 33.1 GB Wan pair; not measured yet tonight, because no :8192 job has run (job 200 will show it)
- SeedVR2 chunked at 21 frames finished a 4 s 2x upscale in 306 s where the unchunked pass went to 0 GB free and had to be interrupted
- memguard.sh is a 12-line script with no dependencies: poll /proc/meminfo every 2 s, POST /interrupt under 5 GB
- Restarting an idle :8192 that holds more than 6 GB of host RAM (bin/reclaim-8192, run by gpu-run before :8191 jobs) took it from 14.2 GB to 1.0 GB and MemAvailable from 17.9 to 31.2 GB, where /free had moved RSS by nothing
- Batching the queue by model family pays the Wan 2.2 cold load (about 2.5 min of Model Initializing) once per block instead of once per job
Where it breaks
- The lock only sees GPU jobs: on Nov 2 other workers' Remotion renders (8.0 GB across 46 processes), 14 agent sessions and a speech-to-text job shared the pool with LTX jobs, and the night's low so far, 8 GB free, came during ordinary LTX 2.5 host takes, not a Wan or SeedVR2 job
- There is no OOM error to catch: when the pool is gone the Spark freezes, and nothing gets written to any log
- Staged weights are not the whole cost: SeedVR2's 3.2 GB of weights pulled the box to 0 GB free on a 302-frame unchunked pass, because the activations scale with frames times resolution
- Restarting a ComfyUI server from inside a locked queue job let the new server inherit the lock and stalled the queue, and five jobs whose header curl hit the server while it was down died at once (rc=7); fixed with 9>&- 8>&- and || true
- A ComfyUI server that looks idle in nvidia-smi can still hold 14 GB of host RAM (the :8192 server after its last Wan job), and POST /free does not give it back: RSS was 14.2 GB before and after /free at 04:16. Only a restart did (RSS 1.0 GB, MemAvailable 17.9 to 31.2 GB). bin/reclaim-8192 now restarts an idle :8192 above 6 GB RSS before each :8191 job
- Swap is 16 GB and was full after 37 days of uptime, so it provides no cushion; emptying it fully needs a reboot, and restarting the idle :8192 gave back about 2.3 GB
- nvidia-smi reports memory.total, memory.used and memory.free as N/A on the GB10; only /proc/meminfo tells the truth
- memguard only watches :8192; an LTX job on :8191 that overran would not be interrupted
- A 45 s InfiniteTalk take is close to an hour of compute you cannot interrupt without losing the whole clip
- --reserve-vram 8 on :8192 has no visible effect in the startup log (both servers print the same 7788 MB headroom line); its real effect is not measured
The problem in one paragraph
The DGX Spark has one pool of memory. /proc/meminfo says MemTotal: 127600752 kB, which is 121.7 GiB, and free -g rounds it to 121. ComfyUI's startup line says the same thing twice: Total VRAM 124610 MB, total RAM 124610 MB. The CPU, the GPU, the desktop, an 18.7 GB voice model, two of our ComfyUI servers and a second client's ComfyUI all draw from that one pool. When it runs out there is no CUDA out-of-memory error to catch. The lab notes from Nov 1 (WORKFLOWS.md) put it plainly: "Running out of unified memory freezes the whole Spark; there's no clean crash." We have never captured a freeze in a log, because when it happens nothing gets written. What we have is the one near miss that was caught in time, and the rules we built around it. Tonight those rules have to carry a 22B model, two 14B experts, an InfiniteTalk stack and SeedVR2 through dozens of back-to-back jobs without anyone touching the box.
A note on units: every GB in this post is a GiB (1024³ bytes), the unit /proc/meminfo, free -g and ComfyUI's own "MB" (which is MiB) all use. ComfyUI's 124610 MB is 124,610 MiB, which matches MemTotal to the kilobyte.
Status: published mid-run at 04:22 ET on Nov 2, corrected and republished at about 05:00 with numbers through 04:49. By then the queue had run the Z-Image stills, the whole first LTX 2.5 block (20 clips at 1280x704), several batches of AI-host takes, and a restart of the idle :8192 server. Wan 2.2, InfiniteTalk, S2V, SeedVR2 and RIFE were still queued, so their rows in the table below say "not measured yet". We republish this page as they land. The 05:00 version fixes a units error in the first one (it compared MiB with GiB; details in the weights section) and replaces our early reading of /free with what a restart showed.
How we tested
This is a workflow review, so the "test" is the night itself: the Nov 2 burn run, where every render goes through one queue and one lock, and the lock logs the lowest MemAvailable (the kernel's estimate of the memory a new program can get without swapping) it saw during each job. The numbers come from five places, and each one is named next to its number:
/proc/meminfo,free -g,/proc/swapsandnvidia-smi, read at 03:13 ET on Nov 2 before the queue started. That is the "resident" baseline.- The two ComfyUI server logs (
comfy-8191.logfor the LTX server,comfy-8192.logfor the video-lab server). Every time ComfyUI prepares a model it printsModel <name> prepared for dynamic VRAM loading. <N>MB Staged."Staged" is ComfyUI 0.43's dynamic-VRAM term for weights prepared for on-demand loading, so those lines are the per-model weight footprint, not the peak while it computes. The Nov 1comfy-8192.logwas overwritten when we restarted that server at 04:17 tonight; its lines survive as quoted tool output in our run transcript (see the note on sources at the end). - The Nov 1 batch logs (
svr_run.log,wan_run.log,s2v_run.log,it_robot_run.log,post_run.log), which hold wall time per job. logs/gpu-queue.log, written bybin/gpu-runtonight: oneSTARTline withMemAvailableat launch and oneENDline withminMemAvailablefor every job.logs/memguard.log, which only gets a line when the guard has to interrupt something.- Each test's
tests/<id>/timing.json, written bybin/run-test:mem_available_gb_before,mem_available_gb_after,mem_available_gb_min(sampled during the job) andfree_other_port, which records whether the/freeagainst the other server answered.
What we did not measure: GPU-side peak allocation per job (the GB10 reports [N/A] for memory.used in nvidia-smi --query-gpu, so there is no device-level counter to read), and the cost of a freeze, since we did not have one.
The budget, measured
At 03:13 on Nov 2, before the first job, free -g read: 121 total, 87 used, 28 free, 7 buff/cache, 33 available. MemAvailable in /proc/meminfo was 35,395,512 kB, which is 33.8 GB (35,395,512 / 1,048,576). Swap was gone: /proc/swaps shows a 16 GB /swap.img (16,777,212 kB) with 16,771,944 kB in use, and SwapFree: 5268 kB. The box has been up 37 days (uptime) and vm.swappiness is 60, so that swap filled slowly with cold desktop pages and never drained. That swap is already spent.
Where the 87 GB sits, from nvidia-smi and ps:
| Resident process | GPU allocation (nvidia-smi) | Host RSS (ps) | Notes |
|---|---|---|---|
| ComfyUI :8191 (LTX 2.5 + image models) | 38,538 MiB (37.6 GB) | 6.7 GB | LTX 2.5 22B int8 transformer, Gemma text encoder, VAEs, upscaler, all staged |
| resident voice model | 19,101 MiB (18.7 GB) | 0.1 GB | Resident all night, not ours to unload |
| ComfyUI :8192 (video lab: Wan, InfiniteTalk, SeedVR2) | 367 MiB | 14.7 GB | Idle on the GPU, but holding 14.7 GB of host RAM; see below |
| ComfyUI :8196 (another client) | 323 MiB | 0.1 GB | Leave alone |
| Desktop (Xorg, gnome-shell, remote desktop, Firefox, portals) | about 885 MiB combined | Eight small G entries |
Add the GPU column and you get 59,214 MiB, about 58 GB. The other 29 GB of "used" is host-side: the :8192 server's 14.7 GB of RSS (resident memory, the RAM a process holds right now), the :8191 server's 6.7 GB, and a few GB of Electron apps and agent processes.
The 14.7 GB on :8192 is the interesting one. Its last job on Nov 1 was Wan 2.2 I2V, its GPU allocation is 367 MiB, and ComfyUI's --highvram help text says the default is that "models will be unloaded to CPU memory after being used." That reads like parked Wan weights waiting in system RAM. The first job of the night tested the idea, and it did not hold up. bin/run-test POSTs /free to the other server before every test, and tests/001-still_zimage-P01-s42/timing.json records free_other_port: 8192: freed, mem_available_gb_before: 33, mem_available_gb_after: 52. So 19 GB came back during a 21 s still, while the :8191 server was growing at the same time (38,538 to 42,252 MiB in nvidia-smi with Z-Image added). /free is asynchronous in ComfyUI (the call sets a flag and the worker acts on it after the call returns), so we are inferring the release from the before and after, not watching it.
But at 03:24 ps still showed the :8192 process at 14.6 GB RSS, with the same 367 MiB on the GPU. So whatever /free released, it was not that RSS. At 04:16 we tested it directly: a second /free on the idle :8192 moved MemAvailable from 13.9 to 17.9 GB and left RSS at 14.2 GB. Stopping the process freed the rest (MemAvailable 32.4 GB), and the restarted server sat at 1.0 GB RSS with 31.2 GB available (queue/logs/052_restart_8192_reclaim_rss.log). We still can't say what the 14 GB was. We can say /free won't return it and a restart will, so the queue now restarts an idle :8192 when its RSS passes 6 GB (bin/reclaim-8192, which gpu-run calls before any job that frees :8192, logged to logs/reclaim.log). The first /free of the night coincided with 19 GB coming back; the second was worth 4 GB.
So the working number is this: with LTX resident on :8191 and the voice model up, there was 33.8 GB available for whatever the queue wanted to run next.
What each model family actually weighs
These are the MB Staged lines from the two server logs, in MiB as ComfyUI prints them, with the total in GB (GiB). Staged weights are the footprint, not the peak with activations.
| Family (server) | Models staged | Weights staged | Source |
|---|---|---|---|
| LTX 2.5 22B distilled (:8191) | Gemma 4 12B text encoder 14,612 MB; LTXAV transformer 20,484 MB; latent upscaler 949 MB; audio VAE 693 MB (a loaded completely line, not a Staged one) | 35.9 GB | comfy-8191.log; the process shows 38,538 MiB in nvidia-smi, so about 1.8 GB (1,800 MiB) on top for the CUDA context, the video VAE and working buffers |
| Z-Image Turbo stills (:8192 on Nov 1, :8191 tonight) | Qwen 3 4B text encoder 7,671 MB; Lumina2 transformer 11,738 MB; VAE 159 MB | 19.1 GB | comfy-8191.log (tonight) |
| Wan 2.2 I2V A14B (:8192) | high-noise expert 13,630 MB; low-noise expert 13,630 MB; umT5 text encoder 6,419 MB; Wan VAE 241 MB | 33.1 GB | comfy-8192.log. Both experts stay staged in one job: the log prints 0 models unloaded between them |
| InfiniteTalk (:8192) | Wan 2.1 I2V 480p 15,638 MB; MultiTalk patch 4,887 MB; umT5 6,419 MB; CLIP vision 1,205 MB; wav2vec2 170 MB; VAE 241 MB | 27.9 GB | comfy-8192.log |
| Wan 2.2 S2V (:8192) | S2V 14B 15,632 MB; umT5 6,419 MB; wav2vec2 large 585 MB; VAE 241 MB | 22.3 GB | comfy-8192.log |
| SeedVR2 3B (:8192) | transformer 3,295 MB; VAE about 0.5 GB on disk (INVENTORY.md; its Staged line did not survive) | 3.7 GB | comfy-8192.log |
| RIFE v4.26 (:8192) | IFNet 10 MB | 0.01 GB | comfy-8192.log |
Two things fall out of that table. First, the Wan 2.2 I2V family stages 33.1 GB, and 33.8 GB was available at 03:13. That leaves 0.7 GB for activations, latents and decode, which is nothing, so it does not fit in practice next to a resident LTX unless something gives, and what gives is supposed to be the :8191 /free at the top of every :8192 job (the queue scripts do that before anything else). No :8192 job had run by 04:49, so that release is not measured yet; job 200 will show it. Second, SeedVR2 is the smallest set of weights on the list and it is the one that nearly took the box down, which is the whole lesson about activations below.
Setup and gotchas
Here is what keeps the box alive, in the order a job meets it.
1. One lock, one queue. bin/gpu-run <label> -- <cmd> takes flock (a file lock: the second job waits until the first lets go) on gpu.lock and holds it for the life of the command. The queue runner in tmux picks job scripts from queue/pending/ in name order and runs each one through the same lock. So there is never a second heavy render, no matter how many agents are writing jobs. While it holds the lock, gpu-run samples MemAvailable every 2 seconds and writes the minimum to logs/gpu-queue.log on the END line. That log is the measurement instrument for this whole post.
2. Free the other server before you start. Each job script opens with curl -s -X POST 127.0.0.1:<other port>/free -d '{"unload_models":true,"free_memory":true}'. Running LTX on :8191 after a Wan job on :8192 without that call means two 30-plus GB families competing for 33 GB. With it, the :8191 server should drop to its context and give the :8192 family the room; we'll have the number when job 200 runs. In the other direction, /free on :8192 does not give back that server's idle host RAM; only a restart does (see the budget section), and the queue now does that automatically.
3. Batch by family. Job numbers group the queue: 100s are LTX on :8191, 200s Wan I2V on :8192, 300s lip-sync, 400s SeedVR2 and interpolation, 500s the Remotion renders. Reloading the Wan 2.2 pair costs real time: the Nov 1 wan_run.log shows 336.3 s wall for a 5 s clip, and the server log for that job shows "Model Initializing" taking 1:16 for the high-noise expert and 1:13 for the low-noise one, so about 2.5 of those 5.6 minutes were loading. Ten Wan jobs in a row pay that once.
4. Chunk SeedVR2 in time, always. On Nov 1 the first SeedVR2 run tried the whole 302-frame InfiniteTalk clip at 1664x960 in one pass. post_run.log has it: up_seedvr2_3b_it_host, wall 377.7 s, status error. The server log shows what happened: SeedVR2 prepared ... 3295MB Staged, then the sampler sat at "Model Initializing" for 1:42, then Global interrupt (no prompt_id specified) and Processing interrupted. WORKFLOWS.md records that the run "drove the box to 0 GB free". That is 3.2 GB of weights (3,295 MiB) pulling the box to the edge, because the activations (the working memory a model uses while it computes, which grows with frames × resolution) for 302 frames at that resolution are the cost, not the weights. The fix is in tools/vlab.py lines 292-294: SeedVR2TemporalChunk in manual mode, frames_per_chunk 21, temporal_overlap 1, plus tiled VAE encode and decode (tile 512, overlap 128, temporal 64/8). The chunked rerun of a 4 s clip finished in 306.2 s (svr_run.log) and the sheet was clearly sharper. The comment in vlab.py says "ALWAYS chunk in time" and it means it.
5. Keep lip-sync windows short. tools/lipsync_batch.py says to keep each shot at or under about 20 s. InfiniteTalk works in 81-frame windows with 9 motion frames carried between them; the 12.1 s host clip on Nov 1 ran 4 windows in 865 s (Prompt executed in 00:14:25). Longer audio means more windows, not more memory per window. InfiniteTalk costs about 72 s of compute per second of output, so a 45 s take is an hour you cannot interrupt without losing it.
6. The tripwire. tools/memguard.sh runs beside long :8192 jobs. Every 2 seconds it reads MemAvailable; under 5 GB it POSTs /interrupt to :8192, logs the event to logs/memguard.log and sleeps 20 s. It only guards :8192, because that is where the big activations live. As of 03:15 on Nov 2 there is no memguard.log on disk, which means it has never had to fire. Whether the Nov 1 SeedVR2 interrupt came from the guard or from a hand on the keyboard, the logs do not say, though memguard.sh was last edited 6 minutes after that run (file times 15:44:58 against post_run.log 15:38:24), so it was probably a person. At 04:49 there is still no memguard.log. Be clear about what that means tonight, though: the guard starts beside :8192 jobs, and no :8192 job had run yet, so tonight it has not been tested at all.
7. The pre-flight rule for 14B. From WORKFLOWS.md and the brief: more than 15 GB MemAvailable before a Wan 14B job starts. gpu-run writes the number on the START line so you can check it after the fact.
8. CPU work waits too. ffmpeg, Remotion and node do not need the lock, but they take memory from the same pool. The rule in the worker brief: check awk '/MemAvailable/{print int($2/1048576)}' /proc/meminfo and wait if it is under 20 GB while a SeedVR2 or Wan 14B job is running. Tonight showed that is too narrow (below).
9. --reserve-vram 8 on :8192. start_comfy_8192.sh starts the video-lab server with --reserve-vram 8. The :8191 server in start_comfy.sh has no reserve flag. ComfyUI's help describes the flag as "the amount of vram in GB you want to reserve for use by your OS/other software." On this box the OS and the GPU share one pool, so that is 8 GB the server will not plan to use, which is the same idea as the memguard threshold applied before the fact instead of after. Both server logs print the same comfy-aimdo integrated Linux GPU RAM headroom: 7788 MB line at startup, with or without the flag, so the flag's effect is not visible in that line and we have not isolated it. Not measured.
Gotcha: swap is full, and only killing a process drains it. 16 GB of swap with 5 MB free means any new memory pressure goes straight to reclaiming page cache and then to the freeze. Memory pressure stall (PSI, /proc/pressure/memory: the time every task was stuck waiting for memory) at 03:15 read full avg10=0.36, total 218,658,076 µs, which is about 218 seconds of full memory stall over the 37-day uptime. That total is the running tally to watch tonight. At 04:14 the total read 227,605,088 µs, so the night so far added about 8.9 s of full stall. At 04:44, after the :8192 restart, SwapFree was 2.4 GB: killing the old process gave about 2.3 GB of swap back.
Gotcha: restarting a server from inside a locked job. The first time we restarted :8192 from a queue job (04:17), the new ComfyUI inherited the lock's file descriptor, so the lock never released and the queue stalled. And while the server was down for about two seconds, five queued jobs died at once with rc=7, because their header curl .../free ran under set -e (gpu-queue.log, 04:18). The fixes: start the server with 9>&- 8>&- so it can't inherit the lock, and || true on the header curl.
Gotcha: nvidia-smi cannot tell you the total. memory.total, memory.used and memory.free all return [N/A] on the GB10. The per-process list works, and /proc/meminfo is the truth. Every script here reads MemAvailable, nothing else.
Settings
The exact knobs, all from files you can open:
bin/gpu-run:flockon$B/gpu.lock;MemAvailablesampled every 2 s;WAIT,START(withMemAvailable),END(withrc,wall,minMemAvailable) lines inlogs/gpu-queue.log.- Queue job header on :8192 jobs:
curl -s -X POST 127.0.0.1:8191/free -d '{"unload_models":true,"free_memory":true}'; on :8191 jobs, the same against :8192. bin/reclaim-8192: before any job that frees :8192, if the idle :8192 holds more than 6 GB RSS, stop it and start it again (start script launched with9>&- 8>&-).tools/memguard.sh: threshold 5 GBMemAvailable, poll 2 s, actionPOST 127.0.0.1:8192/interrupt, back-off 20 s.- Pre-flight: more than 15 GB
MemAvailablebefore Wan 14B; more than 20 GB before CPU-side renders during SeedVR2 or Wan. - SeedVR2 (
tools/vlab.py):SeedVR2TemporalChunkmanual,frames_per_chunk21,temporal_overlap1;VAEEncodeTiledandVAEDecodeTiledtile 512, overlap 128, temporal 64, temporal overlap 8. - LTX 2.5 decode (
tools/vlab.py):VAEDecodeTiledtile 512, overlap 64, temporal 64/16. - Lip-sync: shots at or under 20 s (
tools/lipsync_batch.py); InfiniteTalk 81-frame windows, 9 motion frames. - Server flags: :8192
--use-pytorch-cross-attention --reserve-vram 8; :8191--use-pytorch-cross-attentiononly. Both run ComfyUI 0.43.0 with comfy-aimdo 0.6.0, "DynamicVRAM support detected and enabled", "Using async weight offloading with 2 streams", "Using RAM pressure cache", andModel storage policy: fast_disk=Truefor every model file (all from the server logs).
ComfyUI flags we looked at and did not use, with the help text's own words so you can judge: --vram-headroom ("the amount of vram in GB for DynamicVRAM to maintain as extra headroom above default", a candidate for a stricter :8192), --disable-smart-memory ("aggressively offload to regular ram instead of keeping models in vram", which on a unified box moves bytes within the same pool), --highvram and --gpu-only (keep everything resident; the opposite of what we need), --lowvram ("doesn't do anything if dynamic vram is enabled"), --cache-none ("reduced RAM/VRAM usage at the expense of executing every node for each run"), --cpu-vae (on unified memory, the same pool, slower). None of these were tested tonight, so none are recommended here. The one we would test first is --vram-headroom on :8192.
Speed and memory
Nov 1 wall times, with weights loading included, from the batch logs; tonight's minimum MemAvailable from logs/gpu-queue.log as each family runs:
| Family | Weights staged | Nov 1 wall | Tonight: MemAvailable at START | Tonight: min MemAvailable during |
|---|---|---|---|---|
| Z-Image Turbo still, 1344x768 (10 stills on :8191) | 19.1 GB | 9 s warm, 24 s cold (WORKFLOWS.md); tonight 21.0 s cold then 9.0 s (queue/logs/100_stills_zimage.log) | 33 GB (gpu-queue.log START, 03:15:09) | 33 GB (gpu-queue.log END: wall=104s minMemAvailable=33GB) |
| LTX 2.5 t2v, 1280x704, 5 s | 35.9 GB | 78 s i2v (WORKFLOWS.md); tonight 108.1 s for the first clip, then 72.1 to 75.1 s (14.3 to 21.4 s per second of video, queue/logs/110_ltx_t2v_s42.log) | 50 GB (gpu-queue.log START for 110_ltx_t2v_s42) | t2v (job 110): 23 GB, tests 014 (P04 skateboard) and 020 (P10 wet dog). i2v (job 120, START 30 GB): 13 GB, test 028 (P08), while other workers' CPU renders shared the pool (below) |
| LTX 2.5 image+audio-to-video (AI-host takes), 1280x704, 5.7 to 18.3 s | 35.9 GB (LTX stack) | not run on Nov 1; tonight 1031 s for 59.5 s of speech in four takes, 16.2 to 18.2 s per second (queue/logs/051_hosts_*.log) | 27 GB (job 050, two smoke takes), 21 GB (job 051), 27 GB (job 058) | 9 GB (job 050), 8 GB (job 051), the night's low so far; 25 GB (job 058, after the :8192 restart) |
| LTX 2.5 t2v, 1920x1088, 5 s | 35.9 GB | 183 s (comfy-8191.log; date and resolution not recorded in the log) | not measured yet (job 130 queued at 04:49 ET) | not measured yet |
| Wan 2.2 I2V A14B, 1280x704, 5 s | 33.1 GB | 336.3 s (wan_run.log) | not measured yet (job 200 queued at 04:49 ET) | not measured yet |
| InfiniteTalk, 832x480, about 12 s | 27.9 GB | 865 s (comfy-8192.log, 00:14:25) | not measured yet (job 310 queued at 04:49 ET) | not measured yet |
| Wan 2.2 S2V, 832x480, 12 s | 22.3 GB | 295.4 s (s2v_run.log) | not measured yet (job 330 queued at 04:49 ET) | not measured yet |
| SeedVR2 3B, 2x, chunked 21 frames | 3.7 GB | 306.2 s for 4 s (svr_run.log) | not measured yet (job 401 queued at 04:49 ET) | not measured yet |
| SeedVR2 7B, 2x, chunked 21 frames | not measured (8.3 GB on disk, INVENTORY.md) | not measured | not measured yet (job 402 queued at 04:49 ET) | not measured yet |
| RIFE v4.26 x2, 12 s clip | 0.01 GB | 12.0 s (post_run.log) | not measured yet (job 410 queued at 04:49 ET) | not measured yet |
What the first LTX clips showed, from queue/logs/110_ltx_t2v_s42.log: MemAvailable was 50 GB when the first 1280x704 clip started and 28 GB at the lowest point during it; the second clip started at 32 GB. So about 18 GB stayed with the :8191 server after the first clip (the LTX stack moving from staged to resident), and each later clip dipped a further 3 to 7 GB below its own start (32 to 25, 31 to 26, and 29 to 23 on the fast-action skateboard prompt). That dip is the activation cost of a 121-frame clip at 1280x704 with the weights already resident.
The number that matters in that table is the last column, and specifically how close the SeedVR2 and Wan rows get to 5 GB. Those rows are still empty. So far the lowest minMemAvailable of the night is 8 GB, on job 051, the four AI-host takes for this post's instructional (LTX 2.5 image+audio-to-video on :8191, 03:46 to 04:03), and memguard has not fired (it has not run yet; see rule 6).
The surprise: the floor dropped with nothing new in the lock
The i2v block is where the night got interesting. The LTX server's weights were already resident from the t2v block, and every clip in it is the same size (1280x704, 121 frames). The start-of-clip MemAvailable in tests/<id>/timing.json still went from 30 GB to 24, and the per-clip minimum went 24, 26, 26, 25, 21, 19, 15, 13, 20 and 20 GB, and wall time on the same-size clips rose from 78.1 s to 87.1 s as it squeezed. Nothing new went through the lock. What changed was everything outside it. At 03:40, with job 120 running, ps summed by process family showed:
| Outside the lock, 03:40 ET | Processes | Host RSS |
|---|---|---|
| Remotion renders (headless Chrome) by other workers | 46 | 8.0 GB |
| ComfyUI :8192, idle | 3 | 14.2 GB |
| Agent sessions (Claude Code) | 14 | 3.7 GB |
| A speech-to-text job (faster-whisper) | 3 | 2.4 GB |
| Desktop chat apps | 32 | 3.6 GB |
MemAvailable read 18.5 GB at that moment. None of that is a GPU job, so none of it took the lock, and rule 8 (wait for 20 GB before CPU renders) only names SeedVR2 and Wan. It should name every heavy job: a Remotion render at concurrency 10 is 8 GB on this box, which is a quarter of the headroom we started with. We have not tied each dip to a specific render (we did not log process RSS over time), so treat the table as a snapshot, not a cause.
The cleanest before-and-after we have is the restart. Host batches 050, 051 and 052 (LTX image+audio-to-video takes) bottomed at 9, 8 and 9 GB. Then the idle :8192 was restarted at 04:17, and batch 058, the same kind of work, started at 27 GB and bottomed at 25 GB (gpu-queue.log). Load outside the lock was not the same in each, so this is a snapshot too, but it is the biggest single change of the night.
We are not innocent here either. Job 050, a 5.7 s and 5.6 s pair of AI-host test takes on :8191, started at 27 GB and bottomed at 9 GB (gpu-queue.log, 03:43 to 03:46). Our own short-film render for this post ran in the same three minutes: bin/post-to-video checked for 20 GB before it started, as the rule says, and then its Remotion workers took their share while the GPU job ran. A check at the start of a CPU render is not a guarantee for its whole length. Then job 051, this post's own four host takes, set the night's low so far: 8 GB, while other producers' renders and agents ran beside it.
Peak memory for a job, if you want one figure, is 121.7 GB minus that job's minMemAvailable. It is a whole-box number, including the 18.7 GB voice model, whatever :8191 kept and every agent and render outside the lock, which is the honest way to state it on a unified-memory machine. So far tonight that peak is 113.7 GB (121.7 minus 8), and as the table above shows, a good part of it was not the model.
What this costs in time: the /free plus reload at every family switch is the price of the discipline. For Wan 2.2 it is about 2.5 minutes of "Model Initializing" per cold start (the two 1:16 and 1:13 progress lines in comfy-8192.log). For LTX 2.5 on :8191, tonight's first t2v clip (test 011, family switch) took 108.1 s against 72.1 to 75.1 s for the next nine at the same size, so about 35 s. Batching by family turns those into one payment per block instead of one per job.
What we would change next
- Widen rule 8: every CPU render (Remotion, ffmpeg) waits for 20 GB
MemAvailablewhenever any GPU job holds the lock, and Remotion runs at lower concurrency while one does. - Log the top process families' RSS on the
ENDline, so the next dip has a cause and not just a size. - Put
--vram-headroomon :8192 and measure whetherminMemAvailableon a chunked SeedVR2 job moves. Right now the 5 GB guard is reactive; a headroom flag would make ComfyUI plan around it. - Swap. A 16 GB swap file that is already full is dead weight. Emptying it fully needs a reboot or root, and we have neither tonight; restarting the idle :8192 gave back about 2.3 GB of it.
- Log
/proc/pressure/memoryon theENDline next tominMemAvailable. Stall time is a better early warning than free space, because it rises before the number hits zero. - Test SeedVR2 7B chunked, with the guard running, before trusting it in a queue.
A note on sources
The Nov 1 numbers cited to comfy-8192.log (the Wan, InfiniteTalk, S2V, SeedVR2 and RIFE Staged lines, 0 models unloaded, the 1:16 and 1:13 Wan load times, 00:14:25, and the SeedVR2 Model Initializing 1:42 and Global interrupt) come from that server's log for the Nov 1 session. The log was overwritten when the server restarted at 04:17 on Nov 2 (its start script truncated the file; it appends since 04:50). The lines survive verbatim as tool output in our run's cycle-1 lead transcript, and we cite them as quoted there. The SeedVR2 VAE's Staged line is the one figure that did not survive, so we use its size on disk instead. The 03:13 baseline readings (/proc/meminfo, free -g, /proc/swaps, nvidia-smi, ps) are from the same transcript; MemTotal, swappiness, uptime and the [N/A] counters were re-checked live at 04:44.
memguard.sh: the 5 GB tripwire (poll /proc/meminfo, interrupt :8192) ↗