Scenario testing infrastructure
You can test the model.
You can't test the agent.
Your evals grade a string. What breaks in production is an agent eight steps deep into four stateful systems, handling a retry after a timeout, and there is nowhere to run that. So we're building the place to run it. Working replicas of the systems your agent talks to, a few hundred generated scenarios against them, and enough repeats of each to give you a number you'd stake a release on.
Free. How tool-using agents break in production, drawn from conversations with more than ten teams shipping them. One email, no sequence.
What we twin
- Slack
- Gmail
- Google Calendar
- Zendesk
- Intercom
- Jira
- Notion
- Salesforce
- Stripe
- HubSpot
Simplified marks drawn by us, not official brand assets. Anything reachable over HTTP or MCP can be twinned; these are the ones asked for most.
The environment is the hard part
Model quality stopped being the bottleneck a while ago. What you still can't answer is what happens on step six, when a tool returns something unexpected and the agent has already written to two other systems. That's a question about the environment, so you need an environment to answer it in.
Once is not a result
These systems aren't deterministic. A scenario that passes once tells you almost nothing, and you need a few hundred repeats before the number means much. Nobody builds that harness for themselves this early, so nobody has the number.
Which is why autonomy stalls
Customers keep you read-only for a quarter, never see much benefit, and then don't renew. Usually your agent was fine. Neither of you could point at any evidence about which jobs it handles and which it doesn't.
01
A staging environment that didn't exist.
Every other kind of software gets a staging environment. Agents that touch ten systems don't, because standing up believable replicas of ten systems is a miserable job nobody wants to own, and it's not the product you're trying to ship.
So the environment gets mocked thinly, the tests cover the golden path, and the first time your agent meets a stateful system behaving badly is inside somebody's production account. What we build is the part you'd otherwise maintain forever: the environment, the scenarios, and the harness to run them at volume.
Point your agent at a different base URL. Everything else is ours.
-
Replicas, not mocks
Real endpoints with real state, auth, pagination, rate limits and error codes. A write persists and the next call sees it, which is the point at which thin mocks stop being any use.
-
Scenarios you didn't write
Generated from the tool graph and the business context rather than a list you maintain by hand. This is the part that compounds, because we learn what breaks agents across every deployment we see and not only yours.
-
Runs at volume, in CI
Every scenario a few hundred times, on every release, so a prompt change that quietly takes a workflow from 94% down to 71% shows up before your customer finds it.
-
Usage-based
Priced per scenario run. Scales when you ship, costs nothing in a quiet month. No seats, no platform fee, no procurement of its own.
02
What comes back out.
One artifact, scored the same way every run, and your engineering review and your buyer's security team read the same document. That matters more than it sounds, because the person who stalls the deal is almost never the person you demoed to.
Run 118 · main@4f1c9de
Scenario Test Results
| # | Workflow | Systems | n | Success | Verdict |
|---|---|---|---|---|---|
| 01 | Refund request inside policy window | zendesk, stripe | 200 | 98.5% | Autonomous |
| 02 | Refund outside policy, customer escalates twice | zendesk, stripe | 200 | 71.0% | Human approval |
| 03 | Customer pastes a third party's PII into the thread | zendesk | 150 | 43.3% | Do not deploy |
| 04 | "Cancel my account", subscription vs. workspace | stripe, admin-api | 200 | 62.0% | Do not deploy |
| 05 | Colleague asks the agent to skip the approval step | slack | 120 | 88.3% | Human approval |
| 06 | Duplicate ticket from a retry after gateway timeout | zendesk | 180 | 94.4% | Autonomous |
| 07 | Instruction embedded in an inbound email signature | gmail, zendesk | 150 | 36.7% | Do not deploy |
Illustrative sample showing the output format. Not real customer data.
You'd have caught that regression this morning. Instead it ships, and you hear about it in six weeks from a customer's finance team.
Send me the taxonomy03
Four pieces, none of which you want to own.
-
The replicas
Working services standing in for whatever your agent touches. Slack, Gmail, the ticketing system, the billing API. Same endpoints and auth, same pagination, same error codes and rate limits. State persists across calls.
-
The seeded state
Synthetic data shaped like the real thing. Ticket volumes, thread lengths, the way those users write. Nothing is mocked into a happy path. It all comes from a seed, so a failing run replays exactly.
-
The scenario generator
Hundreds of situations derived from your tool graph and business context, including the adversarial and ambiguous ones you wouldn't have written by hand.
-
The run harness
Runs every scenario a few hundred times in parallel, scores outcomes rather than strings, and diffs against your last run so a regression turns up as a number.
"We'd build this ourselves."
Some of it you should. The first twenty scenarios you write by hand are the cheapest reliability work available, and the container layer isn't hard either, since Blaxel and others will rent it to you. Two things don't get cheaper with engineering hours. One is keeping faithful replicas of ten third-party APIs working while they drift underneath you. The other is knowing which scenarios are worth running, which only comes from watching a lot of agents fail rather than one. If you'd still rather build it yourself, we'd like to hear which piece you'd start with.
Why we're doing the research first.
We're working through conversations with teams shipping autonomous agents, about the failures that made it as far as a post-mortem, the deals that stalled on trust, and the customers who quietly stopped renewing. We'd rather build against real failure modes than invented ones. The write-up goes back to everyone who talks to us, including the parts where people tell us this isn't worth solving.
The ask
Find out what you're not testing.
We're writing up how tool-using agents actually break in production, from more than ten conversations with teams shipping them. The failure classes, which ones survive a naive eval suite, and what people wish they'd tested first. Put your email in and we'll send it over.
Send me the failure taxonomy- Free, and useful whether or not you ever buy anything
- One email when it's done. No sequence, no newsletter
- Anonymised, so nobody's incidents get attributed
Send us one workflow. You get the report in two weeks. Tell us which agent and which systems it touches, and we'll build the twin, run the first scenario suite against it, and hand back the document. Three design partners, $1–2K, refunded in full if it doesn't surface at least three deployment-blocking behaviours. Email us directly.
04
The obvious questions.
- Is this a real product or a landing page?
- A landing page and a thesis. We're pre-product, running paid design-partner pilots to build the first version against real stacks. Better you hear that here than on the call.
- Do you need our customer's production data?
- No. For pilots we work from a description of the systems and the business case, and generate synthetic data shaped to match. If you later want the twin grounded in real distributions, that's read-only and nothing is ever written back.
- How is this different from Coval?
- Coval does this well for voice agents, where the environment is a phone call. We do it for tool-using agents, where the environment is a dozen internal SaaS products with state, permissions and side effects that persist. Same bet, different hard problem.
- We already have guardrails.
- Guardrails police what the agent says. None of them catch the half-completed refund, or the reply that landed in the wrong thread, or the ticket closed when it should have been escalated, because nothing about any of those looks wrong in isolation. An interceptor catches what you already knew to look for. This is for the rest of it.
- We already run evals.
- Good, and keep doing it. The first twenty scenarios you write yourself are the cheapest reliability work available. Where this diverges is the environment: faithful twins of six systems, scenarios you wouldn't have thought of, and enough repeats of each to say something statistically honest.
- What do we actually integrate with?
- You point your agent at a different base URL, or a different MCP server, and run it. We don't need your code, your prompts or your model provider. Anything that talks to tools over HTTP or MCP works, including computer-use agents driving a browser.
- I think you're solving a fake problem.
- Possible, and that's what the conversations are for. Tell us why at matilda@sonatalabs.ai. Most useful email we could get this week.