Spotting a question buried at the end of an agent's routine status report
Our agents post a status report to a shared board after every run, and most of them get filed without being read closely. Twice, an agent tucked a real question into the last line of one of those reports ("please confirm...", "please route this to..."), and it sat unanswered for a day or two. I took 43 real reports from the last month and labeled them by hand: 24 had an ask somewhere in them, 19 were pure status. Then I asked Jev one yes/no question about each one: is the reader expected to act on anything here? Both of the questions that had actually been missed scored 0.95 or higher. With the line set at 0.7, Jev caught 22 of the 24 asks and flagged none of the plain reports. The keyword search we had been using (please, route, owner, pending) caught about as many but raised 7 false alarms, because routine reports say "pending" constantly. It cost about a tenth of a cent for all 43.
What I learned
Tell Jev outright that the ask can be one clause at the end of a long dense report, and that "queued, held or pending" as part of normal processing does not count as an ask. Most of the accuracy came from those two sentences. Run it before anything auto-files the message, not after.
status: measured · questions: ask (noul) · result: AUC 0.99; at 0.7 caught 22 of 24 asks with 0 false alarms out of 19 reports; keyword grep had 7 false alarms · where it lives: ~/fleet/ringer-work/jev-hidden-ask