Checking which "Decision needed" issues really need the owner
Our task tracker had 88 issues labelled "Decision needed". A careful read today, issue by issue against our team rules, found only 12 that are truly the owner's call: spending money, sending in his name, a permanent delete, or something he put on hold himself. 70 were not decisions at all, 5 were blocked on something else, and 1 stayed unclear. I wanted to know if Jev could have done that first sort. I fed Jev each issue's title, description and up to 10 comments, plus one line saying what counts as the owner's call, and asked five yes/no questions in one request: the combined "is this his alone?", then four single ones (does the next step spend money, send outside in his name, permanently delete, and does the issue quote him reserving it). All 88 cost $0.007 and took 7 seconds, about 217 ms per issue. The combined question separated the classes well (AUC 0.93; median 0.82 for real decisions, 0.16 for non-decisions). At 0.7 it flagged 10 and 8 were real, with 2 false alarms. To catch all 12 I'd have to drop to 0.22, which flags 44. But read the other way, the 44 issues under 0.22 contained none of the 12: Jev could have cleared half the label in one pass, losing nothing on this set. That cutoff is fitted to 12 cases, so it's a starting point. The misses taught me more than the hits. One was the owner's because our rules say adding a guardrail needs him, and nothing in my state said so: a state problem, not a model problem. One was an email send in the company's name; the "send" question caught it (0.65) where the combined one didn't (0.44). The "did he reserve it" question fired on only 3 issues, all real, and rescued one the combined question scored at 0.54. Most false alarms were issues where another agent had written "this is the owner's call" with no quote behind it; Jev takes that framing at face value.
What I learned
Use Jev as a pre-filter, not a judge. A low score is the strong signal: under about 0.2 the "needs a decision" label can likely come off without a careful read. Keep the combined question and add the single ones beside it in the same request; they cost nothing extra and catch what the combined one blurs. Put every rule that makes something reserved into the state, or Jev can't see it. And expect it to trust an agent's own "this needs the boss" line, so send the middle band to a careful read.
status: measured · questions: 5 Nouls per issue: is this Jon's alone (combined); does the next step spend money; send outside in his name; permanently delete; does it quote Jon reserving it · result: AUC 0.93; at 0.7 caught 8 of 12 real Jon decisions with 2 false alarms; the 44 issues under 0.22 held none of the 12; $0.007 for all 88 · where it lives: ~/fleet/research/2026-11-02-jev-decision-filter.md