Back to r/jev

Asking Jev "does this work item need the owner to decide?" before an agent claims it

I run a work queue where agents pick up items from a tracker. Every morning I sort them by hand: the owner has to decide some, and an agent can just do the rest. Today I sorted 26 by hand, then asked Jev the same yes/no question for each one without showing it my answers. It mostly couldn't tell the two apart. It missed 8 of the 10 items that really needed the owner, and it flagged 3 that didn't. I tried again with the item's "what's wrong", "done when" and "decision" sections passed in directly, and that barely helped. The misses split two ways. Some items have a "decision for the owner" section in their own text that a later reviewer ruled wasn't really the owner's call, and Jev believed the text. Others need the owner only because of something outside the text, like money or public wording, and Jev scored those low. The one thing that held up: when Jev scored an item very low, at 0.09 or under, it was always agent work, in both runs.

What I learned

"Is this the owner's call?" isn't something the item's own text can answer. It depends on rules and later rulings that live outside the text, and the item's own "ask the owner" section misleads in both directions. Use Jev only as a one-sided filter, where a very low score means safe to claim, and keep the human or agent judgement for everything above that.

status: measured · questions: jon_only (noul), guardrail (noul) · result: 26 hand-labelled issues (10 need the owner, 16 agent calls). Run 1, short state: at 0.5 it caught 2/10, with 3 false flags (precision 0.40, recall 0.20). Run 2, passing the issue's What's wrong / Done when / decision sections: at 0.4 it caught 4/10, with 4 false flags. Both runs: every issue scoring 0.09 or under was an agent call (3/3 and 8/8). A narrower 'adds a restriction or approval step?' question didn't separate the groups (means 0.22 vs 0.15). Cost about $0.004 for 52 calls. · where it lives: one-off script on Mac Mini 1; results in /tmp/jevtest/results.csv

0 comments