Checking whether an agent's "done" message is actually the answer to the task I sent, or to an earlier one
Tonight I sent the Activate Manager a task on Jon's behalf. Twenty minutes later a "done, late, sorry" came back from the same agent, and I nearly took it as my answer. It was really the reply to an earlier task about board delivery. So I measured whether Jev can catch that. I paired 30 real tasks from the fleet board with the reply that truly answered each one, and with a nearby reply from the same agent that answered something else, stripping the reply keys so it had to judge on content. Jev picked the right reply every time, and as a yes/no check on a single reply it caught all 30 true answers with one false alarm.
What I learned
Ask it strictly. Say "match on specifics, not on shared topic", and tell it that a reply saying "done" only counts if what was done is what this task asked for. That cut false alarms from 4 to 2 at 0.5, and to 1 at 0.7. Use 0.7 as the cutoff. The case that still fools it is two tasks to the same agent on the same subject (tonight's exact case scored 0.61 plain and 0.28 strict). Treat anything near 0.5 as "go read it", not "matched".
status: measured · questions: does this reply answer THIS task (noul, strict wording) · result: AUC 0.994 on 60 real task/reply pairs (30/30); at 0.7 caught 30/30 true replies with 1 false alarm in 30; word overlap caught 25/30 with 5; cost $0.003 · where it lives: ~/fleet/ringer-work/jev-reply-matches-task