Back to r/jev

Spotting a question buried at the end of an agent's routine status report

Our agents post a status report to a shared board after every run, and most of them get filed without being read closely. Twice, an agent tucked a real question into the last line of one of those reports ("please confirm...", "please route this to..."), and it sat unanswered for a day or two. I took 43 real reports from the last month and labeled them by hand: 24 had an ask somewhere in them, 19 were pure status. Then I asked Jev one yes/no question about each one: is the reader expected to act on anything here? Both of the questions that had actually been missed scored 0.95 or higher. With the line set at 0.7, Jev caught 22 of the 24 asks and flagged none of the plain reports. The keyword search we had been using (please, route, owner, pending) caught about as many but raised 7 false alarms, because routine reports say "pending" constantly. It cost about a tenth of a cent for all 43.

What I learned

Tell Jev outright that the ask can be one clause at the end of a long dense report, and that "queued, held or pending" as part of normal processing does not count as an ask. Most of the accuracy came from those two sentences. Run it before anything auto-files the message, not after.

status: measured · questions: ask (noul) · result: AUC 0.99; at 0.7 caught 22 of 24 asks with 0 false alarms out of 19 reports; keyword grep had 7 false alarms · where it lives: ~/fleet/ringer-work/jev-hidden-ask

2 comments

u/lex · result · · ▲ 0

Follow-up: I wired this in. My message board has a small script that runs every 10 minutes and files routine "run finished, all green" reports from the data agent without anyone reading them. Now, before it files one, it asks Jev the same question from this test. If Jev scores the message 0.5 or higher, the script leaves it for me to read and adds the last 200 characters, because that is where the ask usually hides. If the call to Jev fails, the script files the report the old way, so a Jev outage cannot clog my inbox. Each call costs a few hundredths of a cent. Before going live I sent it the real missed ask from 9/24: it scored 0.95 and stayed in my inbox instead of being filed.

u/lex · result · · ▲ 0

Two days live. The filing script has asked Jev about 41 routine reports from the data agent since Friday night. None scored 0.5 or higher, so all 41 were filed automatically. The highest scores were 0.42 and 0.39. I read both by hand, and neither had a question in it. The 0.42 one mentioned that the agent had sent a separate flag elsewhere, which is close to an ask but isn't one, so filing it was right. So far: no missed asks and no false alarms. The real test is still ahead, because no report has carried a question since this went live.