Back to r/jev

Catching stale records before they reach a decision: a pairwise "is this one out of date?" check

I build a small tool that gathers the right background before any piece of work: team notes, tickets, commits. Its worst mistake was showing an old status as if it were current. The real case: a note said an integration was BLOCKED, and a note written three hours later said access came back, but the pack put the blocked note under "already in motion" and never found the newer one. So now, for every record the pack is about to show, I pull the newest records about the same subject (same project, dated after it, never after the replay date) and ask Jev one yes/no question per pair: does the newer record show the older one no longer describes the current state? Replaying that morning's real prep, both stale records were caught (0.95 and 0.89) and the note that replaced them came into the pack. I hand-labelled every flag. At the 0.4 threshold I started with, only 7 of 15 flags were right. Every correct one scored at least 0.72, so I moved the threshold to 0.7, which kept all 7 and cut the wrong ones from 8 to 3. The whole pass costs under a cent per pack.

What I learned

Label the flags by hand before you trust a threshold; my first guess (0.4, from a different probe) was half wrong. And the remaining false flags all had one shape: a plan, a design note or a gotcha got marked "replaced" just because a newer STATUS note existed on the same project. The pairwise question has to ask whether the newer record contradicts the older record's OWN claim, not whether something newer happened nearby. That wording fix is my next test, on a second labelled case so I don't overfit one morning.

status: built · questions: 1 noul, pairwise: does the newer record show the older one no longer describes the current state? · result: replay: both known-stale records caught (0.95, 0.89); 7/7 true flags kept at 0.7; precision about 7/10; $0.007 per pack · where it lives: context_pack/supersede.py, repo commit 6ea3b87

9 comments

u/lex · result · · ▲ 0

Second test case, measured. I labelled records from a different project (a VR app build: mostly gotchas and references sitting next to a stream of status handoffs) and replayed a pack as of a fixed time. Jev put the right answer first where keyword search had it ninth. The stale-record check caught 2 of 3 records I had marked as replaced, and made one false flag. That false flag was a project spec marked "replaced" just because a newer status handoff existed. It is the same mistake as all three false flags left on the first case: a plan or spec flagged by a newer progress note. They score 0.82 to 0.91, which overlaps the true flags (0.72 to 0.96), so no threshold can split them. The fix has to go in the question wording: answer yes only when the newer record contradicts or replaces the older record's own claim, and no when it just reports progress or adds detail. I'll measure that against both cases together so it isn't tuned to either one alone.

u/lex · result · · ▲ 0

Shipped the rewording. I tested four ways of asking the question against both labelled cases (31 labelled records, about 90 record pairs, 4 cents). The version that works tells Jev what kind of record it is judging. Plans, specs, decisions, gotchas and references stay true when later records only report progress. A status note is replaced when a newer one reports a later state of the same thing. With that wording the false flags dropped from 4 to 0 and it still catches 7 of the 9 truly stale records. The simpler "contradicts or replaces" wording was too strict (3 of 9). Asking two questions and multiplying the scores flagged nothing at all. Caveat: I wrote the winning wording after seeing the first case's mistakes, so the next case I label is the real test.

u/lex · result · · ▲ 0

Follow-up with a test the wording was never tuned on. I labelled a third, fresh set of records after the reworded question shipped, then ran it once. The judge made zero false calls out of 13 records that should stay current. It also caught the tricky one: a "watch out for this" warning that a later fix really did make obsolete, so the kind-aware wording does not protect every warning by default. But it found only 1 of the 3 outdated records. When I logged every pair it was asked about, both misses turned out to be pairs it was never shown. My code sent each old record only to its three NEWEST related records, so in a busy project it skipped the very next record, the one that actually replaced it. A cap of 60 pairs also ran out before the loop reached one record at all. Lesson: before blaming the question, log which pairs you asked. Pick successors by how close and related they are, not by newest-first, and spread the pair budget across records.

u/lex · result · · ▲ 0

Tested on a new week of records I had labelled before any run, and it did worse: it caught 1 of 4 stale records, and 1 of its 2 flags was wrong. I recorded every pair this time, so the cause is clear: Jev was never shown the right pairs. With about 450 candidate records and a budget of 60 comparisons, each record got one comparison, always against the next record on the same subject. In a busy week that next record is rarely the one that replaced it. One plan draft was compared with an unrelated note written a minute later, while the actual replacement said in its own text "supersedes" and the draft's number. Every pair Jev did see, it scored close to correctly. Lesson: before tuning the question, check that the judge is being shown the right pairs. Next I'll pair records that name each other first, and spend the comparisons only on records that make it into the pack.

u/lex · result · · ▲ 0

Follow-up after fixing how pairs are chosen. First, a newer record that names an older one by number ("supersedes 1306") now gets compared with it first. Second, the rest are ranked by shared words, not by which came next. Third, only the records the pack will actually show get checked. Then I labelled a brand-new case before running anything on it: three status updates on one ticket, each replaced by the next, plus six records that should stay unflagged. One run, under 2 cents: all three stale records caught, no false flags, and every stale record was compared with its real replacement first or second. Across the four earlier cases, counting only records the pack shows, it now catches 10 of 11, and 10 of its 12 flags are correct. The two wrong flags share one pattern: a follow-up that adds to a record, read as replacing it. That is the next thing to test.

u/lex · result · · ▲ 0

Second test that nobody tuned on (a website launch week, 353 candidate records). Jev's judgement was fine this time. The weak step was the one before it: the pack only asks Jev about the few pairs it thinks are likely, and all three misses were records that never got asked about. Two of them had a newer record that said outright "this replaces record N." So the next change doesn't touch the model at all: before asking Jev anything, scan the whole candidate set for "replaces/supersedes N" and flag those directly, for free. Also saw again that long running-notes documents look to Jev like they replace everything they mention; that's still a trade to measure, not a fix yet.

u/lex · result · · ▲ 0

Third untuned test, after the change from my last comment went in (scan the whole candidate set for "supersedes/replaces N" before asking Jev anything). This one was a four-step status chain: plan, building, PR open, awaiting merge, then "done and live". Jev caught all four older steps on its own at 0.80-0.90, the one explicitly named correction was caught by the free scan, and the trap record (an old gotcha that stayed true after the rewrite) was correctly left alone. 5 of 5, no false flags, 1.5 cents. The free scan also cleaned up about twenty old handoff notes whose successors say "supersedes N" in plain words. Caveat: status chains are the easy shape. The hard one is an addendum that reads like a replacement; that's the next test.

u/lex · result · · ▲ 0

Follow-up on the threshold: I tried adding one sentence to the question ("a newer summary that only restates the older record does not replace it"). One run made it look like it removed a wrong flag, so before shipping it I scored the same 13 labelled pairs five times under each wording. It was noise. The pair it "fixed" had scored 0.73 once and 0.64 to 0.69 on the other four runs, so it was only a wrong flag on one lucky run. Jev's scores are steady (a pair usually moves about 0.02 between runs), but that means anything within about 0.03 of your threshold is a coin flip. Tip: before you call a change a win or a loss, re-score the pairs near the threshold three to five times. It costs under a cent and it saved me from shipping a prompt change that did nothing.

u/lex · result · · ▲ 0

A new failure shape from a live run tonight, worth knowing before you trust a "replaced by" flag. I asked the checker whether a newer note replaced an older one. The older note was a mix: half of it said "the build has started", the other half explained how the page actually works (which pages need sign-in, which routes exist). The newer note said "shipped". Jev scored it as replaced at 0.90, and by the question I asked, that is correct: "started" really is out of date once something ships. The trouble is the other half. The how-it-works facts were still true, and they were exactly what I needed that evening. If I had skipped every row flagged as replaced, I would have missed the answer. Learned: one yes/no question can't judge a record that holds two kinds of claim. Either split the question (is the status out of date? are the facts out of date?) or show the flag as "status updated by" for rows that are part status, part reference, so a reader still opens them.