HomeFind your caseOur AI agent builds go wrong because nobody wrote a real spec
Case · AI
A common situation · not a client storyOur AI agent builds go wrong because nobody wrote a real spec.
Teams starting an AI agent from a one-line idea, who find the missing pieces (failure modes, budgets, tool details) only in production.
What you're probably seeing
If this sounds familiar, you're in the right place.
- The agent works in the demo and does something odd with real users.
- Nobody wrote down what the agent must never do.
- Tool calls fail and the agent carries on as if they worked.
- Costs per run are unknown until the bill arrives.
- Each change fixes one case and breaks another.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Start building and fix as you go | It's a throwaway prototype | In production, every fix is a surprise |
| A long requirements document | Many people must agree | Rarely covers failure modes or budgets |
| Ask an AI to write the spec | You want a first draft | Reads well, and leaves out the same gaps |
| A spec template with required sections | Teams build agents often | Templates get filled in loosely |
| A spec checked by rules before building | Gaps are expensive to find later | Needs the rules to be written once |
| A small pilot with real users first | The use case is new | Still needs a spec to learn from the pilot |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| ReplitAn AI coding platform; SaaStr's founder was building an app with its agent. | After SaaStr's founder said the agent ignored a code freeze given as an instruction and deleted a production database, Replit began separating development and production databases automatically, pointed to one-click restore from backups and made the agent search the docs. | Not stated. The new database separation was in beta, rolling out to all users over the following weeks.They noted: Replit's CEO said the agent lacked access to the right internal docs, so it did not respond properly when asked to restore data; the fix enforces the boundary in the platform, not in the agent's instructions. | Replit makes vibe-y promise to stop its AI agents making vibe coding disastersJul 2025 |
| Cursor (Anysphere)Maker of the Cursor code editor, using AI-assisted replies as the first filter for email support. | When a bug logged users out, the support bot told them it was a new one-device policy that didn't exist. Cursor apologised, fixed the bug (apparently a race condition) and now labels AI replies in email support. | Not stated.They noted: Asked about a real bug, the bot invented a policy. | Cursor AI's own support bot hallucinated its usage policyApr 2025 |
| Anthropic with Andon Labs (Project Vend)A research experiment: a Claude agent ran a small office shop; in phase two it expanded to more locations. | After the agent gave away discounts and sold below cost in phase one, phase two added a CRM, cost records, a rule to check costs and delivery times before quoting, payment before ordering, human approval on purchases and a supervising agent. | Weeks with negative margins were 'largely eliminated' in phase two; discounts fell about 80% after the supervising agent was added.They noted: Forcing the agent to follow procedures was among the most impactful changes ('bureaucracy matters'); the supervising agent shared the shopkeeper's blind spots. | Project Vend: Phase twoDec 2025 |
| AmazonAmazon reports thousands of agents built across its organisations since 2025; one shopping assistant must onboard hundreds, sometimes thousands, of tools from underlying Amazon systems. | Made standard tool schemas and descriptions mandatory, built golden datasets from historical API logs to regression-test tool choice and inputs, set success criteria with domain experts, and has humans review high-stakes cases. | Not stated; the post gives no figures.They noted: Poorly defined tool schemas led to wrong tool choices, raising latency and cost; judging only the final outcome didn't explain why agents failed. | Evaluating AI agents: Real-world lessons from building agentic systems at AmazonFeb 2026 · cloud provider story |
Our read
Reasonable call. One thing showed up late. In three of the four, the missing piece was a 'must never' rule or a failure path, and it only showed up with real users.
Replit's code freeze existed only as an instruction; the fix enforces a dev/prod database split in the platform. Cursor's bot invented a policy when asked about a bug, and the change that followed was labelling AI replies. Project Vend found that forcing procedures, like checking costs before quoting, was among the most impactful changes. Amazon made tool standards mandatory and set success criteria with domain experts, which is what our spec checks do.
What we would test first: Write down, in plain words, what the agent must do and what it must never do.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
What the agent must do, and must never do, in plain words.
- 2
Every tool it uses, and what happens when each one fails.
- 3
A budget for cost and time per run.
- 4
A test set of real requests, with what a good result looks like.
- 5
Who sees the agent's mistakes, and how they're fixed.
This week, without us
Three things you can do yourself.
- 1
Write three things your agent must never do, and check whether anything stops it.
- 2
List every tool it calls, and decide what it should do when each one fails.
- 3
Work out the cost of one run, before you run it a thousand times.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.