Leaderboards answer a question you do not have
Ranking models on an invented company tells you close to nothing about your deployment.
Agent benchmarks have converged on a familiar shape. Build one fictional company, write a fixed set of tasks inside it, run every model through them, publish a table. TheAgentCompany from CMU is the clearest example: one generic firm, 175 fixed tasks, a leaderboard at the end.
As research this is useful. As a purchasing decision for your deployment it is close to noise, and there is a result that shows why.
ClawsBench, published in April 2026, built five mock Google and Slack services and ran 44 tasks across them. The finding worth remembering was not which model won. It was that scaffolding moved the results more than the choice of model did.
If your harness matters more than your model, then a ranking of models is not telling you about your system.
Think about what varies between the leaderboard and your deployment. Different tools. Different prompt. Different retry logic. Different data, with your customers and your edge cases in it. Different definition of done. The model is one component, and by that evidence not the dominant one.
The question you actually have
Nobody deploying an agent is trying to find out which model is best in general. They are trying to find out whether the thing they built will behave when it meets the situations their business produces. Those are different questions and only one of them has a leaderboard.
There is also a quieter problem with fixed public task sets. Once a benchmark is public and popular, it starts to leak into training data and into the design of the systems being measured. Performance on it improves. Whether the underlying behaviour improved is a separate matter, and harder to check.
The alternative is not more tasks. It is tasks that belong to you: your tools, your data, your definition of an acceptable outcome, scored the same way every time so that two runs are comparable. You give up the ability to compare yourself to other companies. You gain the ability to answer the question you actually have.
Which is not whether this model wins. It is whether this agent can be trusted with this job.