HomeArticles

Chapter 7 of AI agents in production

AI Agent Scaling Production Challenges: A Field Guide

Why AI agents that work in a demo break in production, and what to fix: error handling, latency, observability, cost control and context window limits.

Table of five ways AI agents break in production: retry storms, context exhaustion, tool failures, non-determinism, cost.

Scaling agents isn't just about throwing more GPUs at the problem. It's about the messy, unglamorous layer between the model and the user: infrastructure, error handling, observability, and cost control. This guide covers the patterns that work, the mistakes that quietly drain your budget, and the tools that actually make a difference.


The Gap Between Demo and Deployment

Many teams treat an agent demo as a proof of concept. It's not. A demo proves the model can reason. Production proves the system can survive. Those are entirely different problems.

Consider what happens in a real deployment. Your agent gets a request, plans a sequence of tool calls, executes them, and synthesizes a response. In a demo, that takes one path with one set of happy-path data. In production, you get malformed inputs, rate limits from every third-party API you depend on, timeouts mid-plan, and models that occasionally decide to call tools in the wrong order. Anthropic's engineering team makes this distinction painfully clear in their Building Effective Agents guide: agents are flexible, which means they're also unpredictable. You're not deploying a deterministic service. You're deploying a system that can make its own decisions about how to behave, and sometimes those decisions are wrong.

I'm not saying that's a reason to avoid agents. I'm saying you need to engineer for the uncertainty rather than assuming it away.

The lesson: design for the environment, not just the model. The deployment roadmap published on Machine Learning Mastery makes this point well — agent production infrastructure needs to handle load balancing, failover, and queue management much like a traditional microservice architecture. But agents add a layer of non-determinism that traditional services simply don't have. Your load balancer can't predict a spike in tool-call depth when the model decides to take four extra steps to solve a problem it could've solved in two.

Where Agents Actually Break in Production

Common failure modes:

  1. Retry storms. One agent hits a rate limit, retries aggressively, and the retries pile up. Ten agents do this simultaneously and you're effectively DDoSing your own infrastructure. Google's research on agentic AI infrastructure in practice identifies this as a core hurdle — agents that aren't designed with backoff mechanisms create cascading failures that bring down everything downstream.

  2. Context window exhaustion. Agents accumulate state over the course of a run. Longer runs mean more context. More context means higher latency and cost per call. Eventually the model can't fit the full history and you start making truncation decisions that silently degrade quality.

  3. Tool call failures. Every external API your agent calls is a point of failure. The tool itself might return garbage, time out, or return a schema you didn't anticipate. Your agent's error handling — or lack of it — determines whether that becomes a minor blip or a full incident.

  4. Non-deterministic outputs. The same input can produce different tool-call sequences across runs. That makes testing, debugging, and reproducing issues much harder than with conventional software.

  5. Cost explosion. Each agent run can spawn dozens of LLM calls. Unbounded agent loops are a budget killer.

The thread connecting all of these: agents amplify small infrastructure failures. A 2% API error rate that feels negligible becomes a constant annoyance when your agent calls five tools per task. Each step multiplies the chance of at least one failure along the way.

Latency Is a System Property, Not a Code Property

A common mistake: teams optimize their agent's latency by improving the model call, then can't figure out why end-to-end response times are still terrible. The model call is often the fastest part of the equation. The slow parts are the round trips — tool calls, database queries, external API requests, and context re-processing.

A frequent cause is an architecture that re-sends the full conversation history with every step. Moving shared context into a persistent store and sending only deltas with each model call addresses it without changing the model.

The Practical Guide from arXiv covers this well: context management is one of the biggest design decisions in agent architecture. Your latency budget has to account for every step in the chain, not just the model inference. If you have a tool call that takes 3 seconds, that's 3 seconds where your user is waiting, and you can't pipeline around it because the model needs the result to decide the next step.

Be intentional about parallelism. Independent tool calls can run concurrently. Sequential dependencies can't. A simple dependency graph in the orchestrator lets independent calls run in parallel. The Toward Data Science piece on workflows vs agents argues this exact point — many agent workflows benefit from structured orchestration that controls when and where parallel calls happen, rather than letting the model decide everything.

Observability: You Can't Debug What You Can't See

Here's a hard truth: most existing observability tools weren't built for agent workloads. Traditional APM tools track requests and spans, but they don't track reasoning. When your agent makes a wrong tool call, you need to know not just that it failed, but why it chose that path. That requires tracing at the semantic level, not just the infrastructure level.

The market for ai agent observability tools for production is finally catching up. There are now serious options — Langfuse, Helicone, AgentOps, and the tracing built into LangSmith — that capture token usage, tool calls, model inputs and outputs, and latency at each step. But tools alone don't solve the problem. You need a discipline around what you trace and how you use the data.

Here's what I'd consider non-negotiable for production agent observability:

from opentelemetry import trace
tracer = trace.get_tracer("agent-runtime")

def run_agent_step(step_name, func):
    with tracer.start_as_current_span(step_name) as span:
        span.set_attribute("agent.id", agent_id)
        span.set_attribute("step", step_name)
        start = time.time()
        result = func()
        span.set_attribute("duration_ms", (time.time() - start) * 1000)
        span.set_attribute("tokens_used", result.get("usage", {}).get("total_tokens", 0))
        span.set_attribute("tool_calls", len(result.get("tool_calls", [])))
        span.set_attribute("success", result.get("success", True))
        return result

Every agent run should emit structured traces that capture not just the infrastructure metrics but the semantic ones: the model's reasoning path, the tool calls it chose, the tokens it consumed, the errors it encountered. Without that semantic layer, you're debugging in the dark.

The Google research on agentic AI infrastructure makes a critical point here: production agents fail in ways that don't show up in model evaluation. A model may achieve excellent scores on benchmark reasoning tasks, but in production it hits an edge case where it calls a tool with the wrong parameters and there's no clear recovery path. You need observability that captures those moments, not just the success path.

AI Agent Error Handling in Production: Design for the Long Tail

Agent error handling: back off, cap and cache for transient errors; validate each step and fall back for semantic errors

Most teams treat error handling as an afterthought. They wrap tool calls in try/except blocks and call it done. That's not error handling — that's just catching exceptions. Real ai agent error handling in production is about designing the agent's behavior when things go wrong, not just preventing crashes.

The key insight: an agent that fails gracefully is more valuable than an agent that never fails. Your goal is to keep the agent useful even when some of its dependencies are down. That means building fallback paths into the orchestration layer, not relying on the model to improvise.

Here's a retry pattern for outbound tool calls:

import random
import time
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=0.5, max=8.0),
    retry=lambda e: isinstance(e, (RateLimitError, TimeoutError)),
)
def call_tool_with_backoff(tool_fn, *args, **kwargs):
    """Calls an external tool with exponential backoff and jitter."""
    try:
        return tool_fn(*args, **kwargs)
    except RateLimitError as e:
        # Add jitter to prevent thundering herd on retry
        jitter = random.uniform(0, 0.5)
        time.sleep(jitter)
        raise

    except ExternalServiceError:
        # Fall back to a cached response if one exists
        cache_key = build_cache_key(tool_fn, args, kwargs)
        cached = cache.get(cache_key)
        if cached:
            return cached
        raise

But retries are only the beginning. The harder problem is the semantic errors — the cases where the tool returns a valid response that's wrong for the context. Your agent might receive a database row that's stale, an API response that's incomplete, or a model output that contradicts known facts. No retry policy fixes that.

A useful principle: every agent step should have an explicit success criteria, not just an exception handler. The agent should know what a good tool response looks like, and it should be able to reject a bad one. The model doesn't evaluate this — validation logic in the orchestrator does. For example, if a tool returns a result that's supposed to be a valid customer ID and it doesn't match the expected format, the orchestrator flags it as a failed step and triggers a fallback path.

The guide from Blaxel on deploying AI agents hits on this when discussing the orchestration layer: agents need guardrails that define what success looks like. Without them, you're relying on the model to self-correct, and models don't self-correct reliably. They generate plausible-sounding confidence that's not backed by actual validation.

Cost Engineering: When Good Agents Go Broke

I'm going to say something that might upset people: token costs are the least interesting cost problem with agents. The interesting problem is waste — tokens spent on retries, on redundant tool calls, on context that gets re-sent every step, on agent loops that don't terminate.

Here's a token budget pattern for predictable costs:

class TokenBudget:
    def __init__(self, per_run_cap=20_000, daily_cap=500_000):
        self.per_run_cap = per_run_cap
        self.daily_cap = daily_cap
        self.daily_usage = 0

    def reserve(self, run_context):
        estimated = run_context.estimate_tokens()
        if estimated + self.daily_usage > self.daily_cap:
            run_context.fallback_to_cheaper_model()
            return
        if estimated > self.per_run_cap:
            run_context.slice_work("sequential")
            return
        if run_context.complexity > HIGH:
            run_context.enable_streaming()

Consider caching. If you're calling an LLM with the same or overlapping context across user sessions, you can cache the system prompt and re-use portions of the computation. Anthropic's prompt caching and OpenAI's cached input token pricing can cut costs dramatically when your agent uses long, stable system prompts.

But caching has a subtle trade-off: cached responses are less flexible. If your agent needs to adapt its behavior based on user-specific data, a cached system prompt forces you to push that variability into the user message, which increases per-request token costs. There's a genuine tension between caching economics and personalization quality. Segmenting agents — one with a stable, cached system prompt for common tasks, another with a dynamic prompt for complex, user-specific scenarios — can work better than trying to build a single agent for everything. The cost difference between the two paths is significant, but the quality difference matters more.

Another pattern that pays off: short-circuiting. Your agent doesn't need to go through a full plan-execute-verify loop for every request. Many requests are simple lookups that can be answered with a single tool call. Set up a routing layer that identifies simple requests upfront and handles them with a lightweight path, reserving the full agent loop for genuinely complex work. The Toward Data Science comparison of workflows and agents makes a related argument: not everything needs to be an agent. Some tasks are better served by deterministic workflows that happen to use an LLM at one step, and those are dramatically cheaper to operate.

The Context Window Trap

Every agent framework pushes you toward a particular way of managing context: accumulate everything, let the model figure it out. That works until it doesn't. Context windows have limits, and those limits force judgment calls about what to keep and what to drop.

A structured memory approach helps. Instead of appending raw document text, have the agent extract structured facts into a knowledge store, then send only the relevant subset of facts with each subsequent step.

I'd also note: the context window isn't just about tokens. It's about attention. Models have finite attention spans, and while architectures like those behind long-context models have improved dramatically, there's still a quality degradation as context grows. Sending a model 50,000 tokens of mostly-irrelevant history is worse than sending it 5,000 tokens of carefully curated context — and it's 10x more expensive.

The arXiv practical guide on agent design discusses this extensively: there's a real trade-off between having complete context and having focused context. Agents that manage context deliberately — extracting, summarizing, and discarding — consistently outperform agents that just append everything.

Human-in-the-Loop Is Not a Feature

Human-in-the-loop is often treated as a feature checkbox. It's not. It's an operational requirement that's almost always more complicated than expected.

Here's why: a human review step breaks the pipeline. Your agent hits a decision point it's not confident about, it hands off to a human reviewer, and that reviewer takes 15 minutes to respond. Meanwhile, the user is waiting, the context is getting stale, and your cost per run is climbing because you're paying for idle agent state. Human-in-the-loop only works if you design the process to minimize the number of handoffs and make each one fast.

A better approach: escape hatches, not constant supervision. The agent runs autonomously by default. It only escalates when it hits a specific, narrow set of conditions — things like a legal review, a high-value transaction, or an internal validation check. Avoid open-ended escalation like "I'm not sure, please help." That type of escalation creates a flood of trivial requests that burn reviewer time and slow everything down.

The current industry consensus is converging on this. Anthropic's Building Effective Agents guide argues that autonomy should be a design choice, not a default, and that the right level depends on the task. Tasks with high error costs benefit from human checkpoints. Tasks with low error costs should run fully automated. The tricky part is that many real-world tasks sit in between, and your agent needs to judge which side it's on.

Another pattern is "provisional execution with review." The agent completes the task, produces its output, and then asks for human review — but it doesn't block on the review. The user gets their answer immediately, and a human reviewer follows up if the agent's action warrants correction. This works well for low-risk, high-volume tasks. It doesn't work for anything irreversible or high-stakes.

Four Changes to Consider

First, add a circuit breaker to every external tool call. Dependent APIs have rate limits you can't control, and aggressive retries make things worse. The circuit breaker pattern tracks consecutive failures and opens the circuit after a threshold, failing fast instead of hammering a degraded service:

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=30.0):
        self.failures = 0
        self.threshold = failure_threshold
        self.opened_at = None
        self.reset_timeout = reset_timeout

    def call(self, fn, fallback=None):
        if self.opened_at is not None:
            if time.time() - self.opened_at > self.reset_timeout:
                self.failures = 0
                self.opened_at = None
            else:
                if fallback is not None:
                    return fallback()
                raise CircuitOpenError("Circuit is open")

        try:
            result = fn()
            self.failures = 0
            return result
        except Exception:
            self.failures += 1
            if self.failures >= self.threshold:
                self.opened_at = time.time()
            if fallback is not None:
                return fallback()
            raise

Second, make the agent orchestrator stateful. Every agent run persists its state to a store — which step it's on, what it's done, what it's decided — so interrupted runs can resume instead of starting over.

Third, add continuous production evaluation instead of only running offline evals on benchmark questions. Sample a percentage of production runs, replay them through an offline eval harness, and score them against expected outcomes. This gives signal on quality that pure observability can't. The arXiv guide calls this "production-then-evaluation" and it's become a best practice for teams serious about agent quality.

Fourth, make some agents less autonomous. Production traces can show agent runs that follow predictable patterns and don't need autonomous decision-making. Convert those to deterministic workflows with LLM components, and keep full autonomy for where it genuinely adds value.

Common Questions

FAQ table on production agents: observability, runaway costs, autonomy, testing and the biggest mistake.

What's the right observability stack for production agents?

Start with your existing tracing infrastructure. If you use OpenTelemetry, instrument your agent with semantic spans that capture tool calls, token usage, and reasoning paths. Then add a dedicated agent observability tool for the semantic layer — Langfuse or Helicone are both good starting points. The goal is to have both infrastructure-level traces and semantic-level traces in one place. You want to answer: which step failed, why did the model choose that step, and what it cost. Most teams find that their existing tools cover the first and third questions; the second requires dedicated agent telemetry.

How do you prevent agent loops and runaway costs?

Budget caps at multiple levels: per run, per user, per hour, per day. And implement a maximum tool-call limit per run — something like 20-30 steps, after which the agent must synthesize its findings and stop. Setting a behavior budget, not just a token budget, also helps. The model gets told it has N steps to complete the task, and the orchestrator enforces it.

Are fully autonomous agents ready for production?

For some tasks, yes. For most, no. Agents with a bounded scope and clear constraints do well. Agents given broad autonomy and vague objectives do poorly. The difference isn't the model — it's the system design around it. Start with workflows that use agents at specific steps, measure performance, and expand autonomy as confidence grows.

How do you test agents before deploying?

Three layers: unit tests on individual tool calls, integration tests on the orchestration logic, and scenario-based evals on the complete agent. The third layer is the hardest — you need a set of representative scenarios with known-good answers, and you need to evaluate against them regularly. Run the harness regularly, for example weekly, against production-traced scenarios so it flags quality regressions.

What's the biggest mistake teams make?

Underestimating the orchestration layer. They treat the agent as the model, and everything else as plumbing. In reality, the orchestrator, the state management, the error handling,

Want to talk this through?

Sivaro AI →