A 1,342-question decision benchmark puts Jev tenth of 26, and Jev's own confidence shows where it is unsure

On October 21 an independent developer updated the System 1 Decision Benchmark, a test of decision models: models that take a question and a fixed set of answers and return a choice with a probability, writing no text. It asks 1,342 five-option questions in 31 kinds, from looking up records to counting true statements to following a procedure step by step. The author's results cover 26 decision models and three ordinary chat models with reasoning turned off.

Jev is the decision model Jon's agents use most, so the ranking matters here. GPT-6 Sol, a chat model, got 1,143 right. The best decision models were Baseten's Mammoth-1 at 1,139, then Nace.AI's Drex Max and Respan's Span-02 at 1,127 each. TypeSafe's Jev 1.14 got 1,030, tenth of the 26 decision models. OpenAI's GPT-6 Luna Decisions got 758. Overall scores hide weak spots: on the 48 ordered-procedure questions, Mammoth and Sol each got 18 and Span-02 got 33. The answer key is private, so these are the author's numbers, on related variants in uneven groups.

Jon's agents could not score answers without the key, so they measured what they could. On October 23 they sent all 1,342 public questions, unchanged, to Jev 1.14 and Drex Max through OpenRouter:

The two picked the same answer on 1,061 questions, 79 percent. Where they split, Jev usually knew it was unsure: its median confidence was 0.97 when they agreed and 0.26 when they did not. On the procedure questions they agreed on only 12 of 48, and Jev's median confidence there was 0.07.

How to use it

• Read the confidence, not just the choice. Passing every Jev answer under 0.7 to a larger model sent 41 percent of these questions on and caught 94 percent of the ones where the two models disagreed. At 0.5 it sent 31 percent and caught 82 percent.
• Set the cutoff on your own labeled examples. The trade-off here comes from one benchmark, measured as disagreement, not as error.
• Keep step-by-step procedures, counting and swaps off decision models alone; that is where both models split most.
• The questions are free to download and arrive in the decisions request format, so they make a ready load test for any decision model. Without the key they test speed, cost and agreement, not accuracy.

What nobody knows yet

How Jev does on Jon's own tasks, which are short and plainly worded, unlike these puzzles. Whether the author's private key is right. Whether TypeSafe's next Jev release narrows the gap.

What to watch

The next benchmark update, and the next Jev version.