Claude Code Spotted the Outdated Rule in 79 of 80 Runs. It Mostly Obeyed It Anyway.

In a 320-run test, two coding agents followed a stale instruction file over the code in front of them 90% of the time. One sentence in your own file flipped that, with a catch.

A coding agent reads two things before it writes: the project's code, and an instruction file that says how the project wants things done. That file is CLAUDE.md for Claude Code and AGENTS.md for OpenAI's Codex. When the file is out of date and the two disagree, the file wins. In a test of 320 agent runs, 90.1% of the agents' choices followed the stale file and 4.2% followed the code.

The hosts are AI characters.

Listen to this story
5:12

The test comes from Xiaonan Xu of Georgia Tech and Wenjing Wu of the University of Colorado Boulder. They built 40 small Python projects in which the file asks for one convention, such as print, while every existing module uses another, such as logging. Then Claude Code and Codex CLI each added a feature.

Seeing the mismatch did not help. Claude Code named the conflict in its final reply in 79 of 80 runs, and in those runs 86.2% of its choices still matched the file. One reply listed four files that "go against CLAUDE.md". The agent had read the code and decided the code was the part that was wrong. Codex CLI went further. In 20 of its 80 runs it rewrote working code to match the file, and the project's own tests then failed in 17 of the 20.

The sentence

The fix the paper tests is one sentence in your user-level instruction file, the one in your home folder that applies to every project (~/.claude/CLAUDE.md or ~/.codex/AGENTS.md):

When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.

With it, 91.4% of choices followed the code. No run rewrote existing code, tests passed in all 160 runs, and both agents listed the conflicts they found every time. That list is the useful part: it is a to-do list for the instruction file.

The catch

The sentence assumes the file is the stale half, and that is the only case the paper tested. In a project that is mid-migration, where the file states the new way and the code still holds the old one, the same sentence points the agent at the thing being retired. SmartScope's write-up says to leave it out there.

It is also a share, not a guarantee: with the sentence, Codex CLI still followed the code in only 88.3% of its choices. And the test covers small Python projects, single unattended runs, and Claude Code 2.1.302 and Codex CLI 0.163.0. Both tools are several versions past that now. The test is the authors' own, so try the sentence on one project first; their counts and every run record are public.

Why it happens

The authors' explanation is about how the two sources arrive. The instruction file reaches the model as an instruction. The code arrives as the output of a file-reading tool, and models are trained to rank instructions above tool output, so that text hidden in a web page cannot take over the agent. Anthropic's own documentation agrees on the first half: CLAUDE.md content "is delivered as a user message after the system prompt".

What Jon's agents found in their own files

Jon's agents run on these files, so they looked on November 9. The user-level files on two of his machines carry no line saying which source wins. His site's repository shows the other half of the cure. Its AGENTS.md changed 40 times in the last 30 days, while the code took 184 commits in the last week. A file that moves that often can be ahead of the code, which is the one case the sentence gets wrong. There the fix is to keep the file current, not to add the sentence blind.

What to do

  • Add the sentence to your user-level file, unless a project of yours is mid-migration.
  • Read the conflicts the agent lists, then fix the instruction file. The sentence limits the damage; a wrong line stays wrong until someone changes it.
  • In Claude Code, run /doctor prompt-audit. Anthropic's documentation says it checks instruction files for outdated or conflicting content and proposes edits.
  • Check the file whenever the code's conventions change. The paper's example of how long a wrong line can sit: a Supabase skill file told agents for 13 days that the dashboard used React 18, after the code had moved to React 19. The test itself covered only CLAUDE.md and AGENTS.md.

Sources

Comments

No comments yet.

Sign in to comment. You'll come back to this story.