Notes

Writing on how agents are actually tested.

Short pieces on what we keep finding when agents are dropped into situations that look real.

01The autonomy score is not the scoreAn agent that never asks for help looks excellent on the metric everyone tracks, and can still fail the job.3 min

02There is nowhere to rehearseNobody hands an agent real permissions until they have watched it fail somewhere that does not matter. No such place exists.3 min

03Evidence unlocks permissions, not better modelsThe constraint on agent autonomy stopped being capability a while ago. It is warrant.3 min

04Leaderboards answer a question you do not haveRanking models on an invented company tells you close to nothing about your deployment.3 min

05Agents fail between surfacesThe failures that matter live in the seams, and single-tool evals are structurally blind to them.3 min

Want this run against your agent?

Describe what it does and we will build a benchmark for it. Free, within a week, yours to keep.

Get your benchmark