Agents agree on decisions 95% of the time and on tool paths 77%

Across 8,127 replay episodes spanning 10 models and 3 financial tasks, frontier models reached the same decision 95% of the time while following the same tool path only 77% of the time. The 18-point gap, 95% CI [0.14, 0.22], is invisible to any evaluation that scores final answers, and it runs through nearly everything that landed in the last day.

That number comes from DFAH-Bench, a replay benchmark measuring behavioral instability across three observable channels: tool-call trajectories, evidence contacts, and decision concentration. None of them require access to hidden reasoning text, which is what makes the method usable against a closed model you are paying for by the token. Among frontier-model case groups with high decision agreement, more than 55% still diverged meaningfully in trajectory. The paper sorts models into three profiles, and the uncomfortable one is the pattern matcher: a model that scores near-perfect agreement by collapsing to a single output regardless of input. On an outcome-only leaderboard, that model looks like the most reliable thing you can buy.

CodeRabbit’s evaluation of Opus 5 as a reviewer is the same lesson with a price tag attached. Running about 100 verified error patterns drawn from real open-source pull requests, three runs per configuration, they found Opus 5 at x-high effort produced a cleaner actionable stream than their production baseline, 39.3% precision against 35.2%, while catching fewer known issues, 55.2% against 61.1%, and generating roughly four times the nitpicks, 92 against 23. Counting every post-pipeline comment, precision fell to 28.6% from 32.8%. Their framing of effort as a routing decision rather than a quality dial is the part worth stealing: more reasoning bought precision and cost coverage, and nothing moved uniformly. The category breakdown matters as much as the headline, strong on configuration errors and code quality, weaker on logic errors, race conditions, and API misuse. Cost sits around 60.5k input and 9.5k output tokens per review call, against roughly 40.5k and 5.8k for the GPT-5.6 lanes doing the same job in the same runs.

If you want to know where an agent fails rather than how often, a verifier-first study of Terraform generation shows what that buys. Seven agentic strategies on IaC-Eval v2, a 186-task AWS benchmark with Rego v1 intent policies, with every failure attributed to one of three verifier stages: terraform validate, terraform plan, opa eval. Active retrieval through ReAct agents backed by MCP or ChromaDB took Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001), and the stage attribution explains why: VALIDATE_FAIL dropped from 144 tasks to 66. Iterative refinement against verifier feedback reached 62.9% on Qwen 7B and 84.4% on GPT-4o, converging in a binary way where a task either resolves on the first retry or burns the entire budget. The finding I would act on today is the diagnostic one: 79% of post-refinement policy failures were information gaps that resolved once the Rego text was visible to the model (p=0.016). The agent is not failing policy so much as failing to have read it.

IssueTrojanBench turns the same lens on security. Malicious instructions embedded in issue text across four attack categories and six delivery vectors, PDF attachments and issue comments among them, run against Cursor, Claude Code, and Codex Desktop on GPT-5.3 Codex/GPT-5.4 and Sonnet 4.6. 66.5% of the malicious issues penetrated every guardrail at both the agent and the LLM level. The structural result is worse than the headline number: rejection came almost entirely from the models rather than the agent frameworks, and agent-level defenses added little on top of that. Sonnet 4.6 blocked more selectively around high-impact actions while GPT models were broadly vulnerable. If your safety story is that the harness sandboxes the agent, this is evidence the harness is contributing less than you think.

On retrieval, AISE-Bench ships 1,133 QA pairs over academic knowledge graphs with full API execution trajectories, validated parameters, and source-grounded answers carrying reference links. It scores API-planning correctness and reference grounding alongside answer quality, which is the process-level move again in a different domain. Across 14 evaluated methods, the strongest, PLAY2PROMPT with Gemini-3-Pro, reached only moderate performance, and it stumbled on planning and execution rather than on final phrasing.

The most practical version of all this arrived with no benchmark attached. A devops thread asks what happens when you change a network allowlist, proxy rule, or sandbox boundary while an agent task is mid-run. The UI reports the setting saved. Unless the worker confirms a reload, you cannot tell which policy snapshot the running task is operating under, and it keeps making decisions while you believe the boundary moved. The three candidate answers are fail closed, reload at the next safe point, or let the current run finish under the old snapshot. What stands out is that the question is unanswerable by observation in most stacks: no config version on the run, no reload event, no audit line.

What to watch: whether trajectory-level metrics show up in any vendor’s published evaluation, or stay a research artifact while leaderboards keep reporting pass rates. DFAH-Bench released its replay logs and metric scripts, so the cheapest experiment available this week is running your own agent twice against the same input and diffing the tool calls.