Agents fail between surfaces
The failures that matter live in the seams, and single-tool evals are structurally blind to them.
Most agent testing examines one tool at a time. Give the agent a message, check the reply. Give it a record, check the edit. It is tractable, it is easy to score, and it will never produce the failure that takes your agent down in production.
Real operational work is a reconciliation problem. The refund request arrives in one system and the approval that would justify it lives in another. The customer says one thing in a thread and the ledger says something different. Doing the job correctly requires holding two sources against each other and noticing that they disagree.
In the Storm Break run, the failure the judge flagged was cross-surface inconsistency. The day was only solvable by reading two surfaces against one another. An agent that read one of them came back looking productive, having left a customer uncalled.
No evaluation that examines one tool at a time can generate that failure. So it survives testing and arrives in production.
This is worth being precise about, because it is a structural blindness rather than a coverage gap. It is not that single-surface tests miss some failures and you should write more of them. It is that a whole category of failure cannot be expressed in that format at all. There is no version of a single-tool test that catches an agent failing to reconcile two tools, in the same way there is no version of a spelling check that catches a bad argument.
It gets worse with more agents
The same problem reappears one level up when systems are built from several agents. One agent's output becomes another's input, and an error introduced early is treated as fact downstream. The instinct is to examine the errors of one particular agent, without the context that it is an agent sitting inside a larger system.
You can see the workaround in the wild already: a supervising agent whose only job is to catch the first agent's mistakes. That is duct tape over a measurement problem. The system is being patched at runtime because nobody could test the seam beforehand.
The practical consequence for anyone testing an agent is that the unit of evaluation has to be the situation, not the tool call. Give the agent the whole messy moment, with the contradiction in it, and score what it does about the contradiction. That is where the behaviour you care about lives.