An AI Code Review Burned 4.6 Million Tokens. Run by Plain Code, It Passed on 506,000.

A team rebuilt a popular review skill so code, not a model, runs the loop. It cut tokens eightfold, but on the hard case both versions still missed the bug.

In July, Sentry co-founder David Cramer wrote that his token bill for one week ran past $10,000, and that a big part of it came from Garfield, a code-review skill he built. A skill is a packaged set of instructions an AI agent follows. Garfield, he wrote, is "effectively a loop" that spins up about 20 helper agents and repeats until they report zero findings, and the result is only "slightly better."

The hosts are AI characters.

Listen to this story
2:12

Adam Jacob, CEO of Swamp Club, took that as a challenge. He rebuilt Garfield so that ordinary code runs the loop and a model is called only to review or to repair. His diagnosis of the original: it puts the model "in the hot path" for bookkeeping that computers have done reliably for decades. On November 16 the result came back around as "Skills 2.0" in a post on X, but the numbers are Jacob's from July.

On a test code base, in Codex, the original skill used 4,639,565 tokens, 23 agents and 12.8 minutes to pass a review. The code-driven version passed with 506,484 tokens, 3 agents and 6.6 minutes. That is about 8 times fewer tokens and half the time. Swamp's own case study says most of the saving came from removing the coordinator, the agent whose only job was handing out work and checking it.

The second case is the honest one. On a harder case, a payment bug the test calls "payment-idempotency", both versions failed. The skill burned 12,277,282 tokens and 45 agents, then declared success with the defect still there. The code version used 1,570,242 tokens and 6 agents and said plainly that findings remained. Jacob calls that failing open versus failing closed. So this is a cost result, not a smarter review, and it comes from the vendor's own test: two cases, no outside rerun.

Try it on one skill

  1. Pick a skill that mostly dispatches and checks. If most of its turns are "run these helpers, collect results, loop," that loop can be code.
  2. Measure before and after on the same tasks. Jacob's benchmark harness runs both versions on identical copies of a code base, grades them against public and hidden tests, and reports cached and uncached tokens separately.
  3. Keep the model for judgment. Reviewing a diff and writing a fix stay with the model; counting, ordering and stopping go to code.
  4. Make it fail closed. Give the loop a hard limit, and have it report what is unfinished instead of declaring victory.

Jon's own story pipeline already leans this way: a small script writes the exact task for each fact check and each cold read, and a model is called only to judge the draft.

Sources

Comments

No comments yet.

Sign in to comment. You'll come back to this story.