HomeFind your caseOur AI agents work in demos and fail in production
Case · AI
A common situation · not a client storyOur AI agents work in demos and fail in production.
Teams whose agents impressed in a demo and now fail quietly with real users: wrong tool calls, loops, runaway costs, or answers nobody can trace.
What you're probably seeing
If this sounds familiar, you're in the right place.
- The agent loops, or gives up halfway, with real users.
- It calls the wrong tool, or the right tool with wrong inputs.
- Failures are silent: no error, just a bad result.
- Costs per run vary wildly, and some runs are very expensive.
- Nobody can replay a failed run to see what happened.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| A better model | The failures are reasoning mistakes | Costs more, and doesn't fix missing guardrails |
| Tighter prompts | Instructions are unclear | Without traces and tests, changes are guesses |
| Tracing and observability | You can't see what runs do | Shows failures, doesn't prevent them |
| Fixed workflows for the known paths | Most requests follow a few paths | Less flexible for the rest |
| Limits on steps, cost and time | Runs sometimes run away | Needs a sensible fallback when a limit is hit |
| Human approval for risky actions | Mistakes are expensive | Slower, and needs someone on call |
| Fewer, clearer tools | The agent picks the wrong tool or loops | Only works when the data or docs underneath are clear |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| VercelD0, its internal text-to-SQL agent, which staff ask data questions in Slack; the first version had 17 specialised tools. | Removed most of the custom tools, prompt rules and guardrails, and rebuilt it as a 'file system agent' that reads a well-documented semantic layer with standard Unix commands and runs SQL. | On 5 representative queries: success went from 4/5 to 5/5, average time from 274.8s to 77.4s, average tokens from about 102k to about 61k, and average steps from about 12 to about 7. The old agent's worst case took 724 seconds and 100 steps, then failed.They noted: Every edge case had needed another patch and every model update a recalibration; they say the new approach only works because the semantic layer was already well documented. | We removed 80% of our agent's toolsDec 2025 |
| ShopifySidekick, the AI assistant merchants use to run their stores, with dozens of tools. | Moved special-case guidance out of the system prompt into instructions returned with tool results, built test sets from sampled real merchant conversations labelled by at least three experts, calibrated LLM judges against them, and replays real goals with a simulator before each release. | Judge agreement with humans (Cohen's kappa) rose from 0.02 to 0.61, against a human baseline of 0.69 (no timeframe stated).They noted: Tool boundaries blurred past about 20 tools and behaviour became hard to reason about past 50; they advise against multi-agent designs early on. | Building production-ready agentic systems: Lessons from Shopify SidekickAug 2025 |
| ManusThe Manus AI agent, where a typical task needs around 50 tool calls and input tokens outnumber output about 100 to 1. The team says its lessons came from rewrites, dead ends and real-world testing across millions of users. | Rebuilt the agent framework four times. Kept the start of the prompt stable (no timestamp) so the KV-cache could be reused. Masked tools with a state machine instead of adding or removing them mid-task. Had the agent rewrite a todo.md file to keep its goals in recent context, left failed actions and errors in the context, and added small variations so the agent did not copy its own past actions. | Not stated; the post gives no before-and-after figures.They noted: When earlier steps referred to tools that had been removed, the model got confused, which often led to schema violations or hallucinated actions. Over long tasks the agent can drift off-topic or forget earlier goals. Erasing failures from the context removes the evidence the model needs to adapt. When reviewing a batch of 20 resumes, the agent often fell into a rhythm of repeating similar actions, leading to drift, overgeneralisation or sometimes hallucination. | Context Engineering for AI Agents: Lessons from Building ManusJul 2025 |
Our read
Same call. The teams that got their agents working started from real runs, then reworked the tools and what the agent sees at each step. That's where our tests point too.
Shopify built its test sets from sampled real merchant conversations, which is the same as sorting real runs by how they failed. Vercel's old agent had runs that went to 100 steps and still failed, and Shopify found behaviour hard to reason about past about 50 tools; both cut or reworked their tools. Manus found that removing tools mid-task confused the model and that erasing failed steps took away what it needed to adapt, so it masks tools and keeps errors in view. Vercel's before-and-after rests on 5 queries, so we would still measure cost and steps per run, worst cases included, on a larger sample.
What we would test first: Take a sample of real runs and sort them by how they failed.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
A sample of real runs, sorted by how they failed.
- 2
Tool calls: which fail, and what the agent does next.
- 3
Cost and steps per run, and the worst cases.
- 4
Which requests could follow a fixed path instead.
- 5
What you can trace and replay today.
This week, without us
Three things you can do yourself.
- 1
Pull twenty failed or poor runs and sort them by cause.
- 2
Set a hard limit on steps and cost per run, and see what hits it.
- 3
Make sure every run logs its tool calls with inputs and outputs.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.