Agent reliability converges on one rule: trust the trace, not the reply

A model swap made an agent stop calling cancel_subscription while still telling users “done.” The replies read fine. The evals still passed. Nobody caught it until someone went looking at what tools actually got called, not what the agent said it did. That gap between an agent’s claim and its trace is the thread running through the last day or two of agent-reliability work, and for once it’s showing up as much in shipped code as in papers.

The developer who found the cancel_subscription bug built whatbroke, a CLI that diffs agent trajectories instead of agent text. Record a JSONL trace before a change and one after; whatbroke diff reports which tools got called with which arguments, in what order, at what cost and latency, and what the final output was, then exits 1 on a breaking change so it slots straight into CI. Because agent behavior is nondeterministic, each scenario gets recorded a few times and findings come back with a flap rate, so a difference that already existed between two baseline runs gets demoted instead of flagged as a regression. It imports Langfuse, LangSmith, and OTel GenAI exports directly, runs fully offline, and the author built it because Opus 5, having just shipped, means a wave of model-string swaps is coming.

Proof-or-Stop makes the same argument at far larger scale and with harder numbers. The method treats an agent’s claim of “reviewed,” “tested,” or “DONE” as exactly that, a claim, and blocks lifecycle transitions until fresh, mechanically verifiable evidence bound to the current source state satisfies a gate. In a 9,240-cell ablation, a naive compute-budgeted loop let 31 of 1,800 injected failures pass silently through to a hidden-fail state; the gated loop cut that to 2 of 1,800, a 1.6-point improvement with a 95 percent confidence interval of [0.8, 2.5]. A near-compute comparison against a loop that added more review but no lifecycle gate landed at 14 of 1,800, evidence the authors read as showing that enforcing review as a gate does more work than simply adding a reviewer.

Grab’s engineering team reached the same conclusion from the opposite direction, at production scale. Their internal framework, LLM-Kit, now backs more than 500 services and 50-plus MCP servers behind one LLM gateway handling billions of tokens a month, but it started as a single GPT-4-32k support bot in 2023 that taught the team, in their words, that “vibe check is not an evaluation strategy.” The other lessons read like a checklist for anyone scaling past a demo: model and provider switching has to be a config change, not a rewrite; a wrong answer needs one trace spanning the agent loop, the tool calls, and the model call, not three separate log greps; and the plumbing around an agent, auth, secrets, tracing, health checks, takes longer to build than the agent itself. Their template now takes a new service from zero to a working, evaluated, gRPC-connected agent in about an hour, down from two weeks, and it ships ROUGE, BLEU, and LLM-as-judge evals before a developer writes a line of their own logic.

CAVA pushes the evidence problem into governance. Agent activity today scatters across local coding hooks, SDK tool calls, browser automation, managed-agent traces, and workflow engines, each keeping its own incompatible record of what happened, which makes a basic question hard to answer: what action was actually approved, and can an independent verifier reproduce that same action’s identity later? CAVA’s answer is a runtime-semantics layer that canonicalizes heterogeneous agent activity into a single action object, tested against a 384-variant benchmark covering wrapper bypass, approval binding, and attestation tamper detection.

The training side of agentic coding is converging on the same insight from a different angle: an agent’s production behavior only means something if it was trained inside the actual harness it runs in. OpenForgeRL trains agents end-to-end inside real inference harnesses, Claude Code, Codex, OpenClaw, by proxying the harness’s own model calls into RL training data and running each rollout in its own Kubernetes-orchestrated container. With only hundreds to a few thousand tasks it beats open baselines of similar size on nearly every benchmark tested and, on GUI tasks, matches models several times its size. The more interesting result is qualitative: some harnesses are measurably harder to learn from than others, and RL improves self-verification and multi-step planning while error recovery stays weak across the board.

NOOA closes the loop by making the agent itself look like ordinary software. NVIDIA’s framework represents an agent as a Python object: methods are the actions the model can take, fields are its state, docstrings are its prompts, type annotations are contracts, and a method whose body is just ... gets completed at runtime by the model while methods with real bodies stay deterministic Python. The point isn’t cleverness, it’s that the same object can be tested, traced, and refactored the way any other code is, which is the property every project above this one is trying to bolt on after the fact.

Opus 5’s rollout is about to generate exactly the kind of silent regression whatbroke was built to catch, at a scale most teams have no way to detect yet. Worth watching which of these approaches, trajectory diffing, evidence-gated lifecycles, or canonical action attestation, ends up wired into a CI pipeline near you first.