HomeFind your caseOur AI assistant gives wrong or expensive answers
Case · AI
A common situation · not a client storyOur AI assistant gives wrong or expensive answers.
Teams with an assistant or chatbot built on a large language model (for support, internal knowledge or sales) that worked in the demo and disappoints in production.
What you're probably seeing
If this sounds familiar, you're in the right place.
- Confident answers that are wrong, or that cite the wrong document.
- Answers that were right last month are wrong now, because the documents changed.
- The model bill grows faster than usage, and nobody knows what one answer costs.
- Answers are slow, especially for long questions.
- Nobody can tell whether a prompt change made things better or worse.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Switch to a bigger model | The model lacks reasoning, not information | Costs more per answer, and rarely fixes missing context |
| Rewrite the prompt | The instructions are unclear | Without a test set, you can't tell whether it helped |
| Fine-tune a model | You need a fixed style or format, at volume | Doesn't teach facts that change, and needs data and upkeep |
| Fix retrieval: chunking, hybrid search, reranking | The right document isn't reaching the model | Needs a test set of real questions |
| Route simple questions to a smaller model, and cache | Many questions repeat or are simple | Needs measurement to route safely |
| Hand off to a person when unsure | Mistakes are costly | Needs a confidence signal you can trust |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| DoorDashSupport chatbot for Dashers (delivery drivers), used by thousands of Dashers each day. The old system used fixed flows, and only a small share of issues could be resolved quickly. | Built a RAG bot over knowledge base articles. Every answer passes a guardrail: a cheap similarity check, then, if that check fails, an LLM check for grounding and compliance. An LLM judge scores quality in five areas, and Promptfoo regression tests block prompt changes that fail. | The guardrail cut overall hallucinations by 90% and potentially severe compliance issues by 99%. Timeframe not stated.They noted: The guardrail adds latency, and depending on the model and prompt size, LLM latency can range from under a second to tens of seconds. A stronger guardrail model cost too much. Some knowledge base articles used misleading terms or were incomplete and had to be fixed. The issue summary has to be accurate because it shapes what retrieval returns. | Path to high-quality LLM-based Dasher support automationSep 2024 |
| UberGenie, an internal on-call copilot that answers questions in Slack help channels. This work covered security and privacy policy questions: 40+ policy documents and a golden set of 100+ questions with expert answers. | Used the golden set to measure answer quality. Checking the processed documents showed that standard PDF loaders broke tables apart, and simple similarity search sometimes pulled in irrelevant text. They moved policies to Google Docs, kept tables intact, added summaries and FAQs to chunks, and added hybrid (vector plus BM25) search. An LLM judge scores each answer against the expert answer. | Acceptable answers rose 27% (relative). Incorrect advice fell 60% (relative). Evaluating an experiment went from weeks to minutes.They noted: Expert review time was the early bottleneck until evaluation was automated. Many experiments gave small gains and then plateaued. Better source documents gave better answers. | Enhanced Agentic-RAG: What If Chatbots Could Deliver Near-Human Precision?May 2025 |
| Intercom (Fin support agent)RAG-based customer support agent used by many businesses. The evaluation covered data from 685 to 1,000+ customer apps and an A/B test over 1.5M conversations. | Replaced a paid third-party reranker (Cohere Rerank-v3.5), which was expensive, with a reranker of their own trained on 400,000 real queries. They tested it on 3,000 real queries, backtested it on 1,500 recent conversations, then ran an online A/B test. | Reranking cost fell 80%. P50 latency was unchanged at about 150 ms. Offline ranking scores rose 13–23% (for example, MAP from 0.521 to 0.612). Resolution rate improved by a statistically significant amount (p < 0.01). The size of that gain was not disclosed.They noted: Open-source rerankers were not good enough, and an LLM-based reranker was too slow. Label quality matters, and they plan to re-label the training data with stronger models. | How We Built a World-Class Reranker for FinSep 2025 |
| HoneycombQuery Assistant, a feature that turns plain-language questions into Honeycomb queries using gpt-3.5-turbo. Released May 2023. | They traced every LLM call in production (input, output, tokens, errors, user feedback). They set an error SLO and fixed the most common failures they saw. | Success rate rose nearly 15% from launch. The error SLO target, measured over a seven-day window, was raised from 75% to 80% successful requests. Most calls took under 10 seconds. Token use reached the model's ~4k limit for teams with large schemas.They noted: Small prompt changes can cause large behavior changes. Teams with the largest schemas used the feature most, which was the opposite of what the team expected. Token data showed that schema context was being cut off. | Improving LLMs in Production With ObservabilityJun 2023 |
Our read
Same call. Most of these teams built a test set from real questions and fixed retrieval and documents before changing the model, which is the order our tests follow.
Uber's golden set measured answer quality, and checking its processed documents found a PDF loader that broke tables. The fixes were to documents and retrieval, not the model. DoorDash found misleading terms and gaps in its knowledge base, notes that its summary step shapes retrieval, and blocks prompt changes that fail regression tests. Intercom tested a new reranker on real queries, then backtests, then a live A/B test before switching, and cut reranking cost by 80%. Honeycomb is the partial exception: its error SLO catches queries that fail and user feedback flags poor results, but its write-up has no known-good answers to test against. We would also measure cost and response time per answer up front. DoorDash's write-up shows its guardrail trades latency and cost for accuracy, and Uber's gives no cost figures.
What we would test first: Build a test set of real questions from your logs with known good answers, then check whether the right document reaches the model for each one.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
A test set of real questions with known good answers, taken from your logs.
- 2
Retrieval: for each question, does the right document reach the model?
- 3
Where wrong answers come from: retrieval, the prompt, the model, or the documents themselves.
- 4
Cost and response time per answer, today.
- 5
The same test set run against each shortlisted fix.
This week, without us
Three things you can do yourself.
- 1
Collect a few dozen real questions your users asked, each with the answer they should have got.
- 2
For ten wrong answers, check whether the right document was retrieved at all.
- 3
Work out what one answer costs: tokens in and out, times the price per token.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.