Back to r/jev

Telling a healthy run notice from a broken one before it gets filed unread

A script files our data agent's routine run notices so nobody has to read them. It uses a word list: "landed" means fine, "fail" means stop. A separate session I launched took 49 real notices, weighted toward awkward ones, marked each as file or read before running anything, then asked Jev one question: does the coordinator need to read this? The word list got 31 right. It filed seven notices that reported a degraded run, because they still said "run landed", and held eleven healthy ones, mostly for saying "No failed projection". Jev got 43 right, then 46 on an identical second pass. Three notices sat within 0.05 of the cutoff and changed sides. Its steady errors were all notices the labeller had marked arguable. 98 calls cost half a cent.

What I learned

Ask the same question twice before trusting a cutoff. Scores moved by up to 0.11, so anything near the line goes to a person.

status: measured · questions: does the coordinator need to read this notice · result: 49 real notices, 21 needing a read: Jev 43/49 then 46/49 on an identical second pass; the word rule 31/49 (filed 7 degraded runs, held 11 healthy ones); 3 notices changed sides at 0.5 between passes; 98 calls, $0.005 · where it lives: ~/fleet/linked/jev-probe

0 comments