AI Agent Deployment Without Breaking Existing Systems
The first agent we put in front of a production database didn't crash anything. It did something worse. It ran the same read-only query every 90 seconds for six hours, and quietly burned $4,300 in compute before a security alert flagged the connection pattern. The system never went down. That was the whole problem.
"AI agent deployment without breaking existing systems" sounds like a cautious goal. In practice, it's the only goal that matters. An agent that works in a sandbox and falls apart against real traffic is a demo. An agent that works alongside your legacy order system, your SAP instance, your 12-year-old Postgres — that's production.
This guide covers the patterns that actually work in 2026: shadow mode before action, feature flags as kill switches, observability that pays for itself, and a rollout ladder with an exit at every rung. You'll get code, not theory. And I'll tell you which mistakes cost me real money so you don't repeat them.
The Model Was Never the Risky Part
Most teams worry about the model. They're wrong.
The model is the most predictable component in the stack. It has a known API, known latency, known cost per token. The brittle parts are the integrations around it: the API it calls, the database it reads, the human workflow it interrupts. I watched a well-behaved agent take down a checkout flow because it misinterpreted a 500 response as "cart is empty" and cleared the cart.
Google's research on agentic infrastructure found the same thing. The hard problems in production agents aren't reasoning quality. They're observability, state management, and guardrails. The agent is a black box that occasionally does something unexpected. Your job is to make sure "unexpected" stays cheap.
Anthropic's Building Effective Agents guide makes a distinction that matters more every quarter: workflows are predictable code paths with LLM calls at specific points, and agents are models that decide their own next step. If you're trying not to break existing systems, you start with workflows. You earn the right to agents.
Shadow Mode: Let It Be Wrong in Private
Shadow mode is the most underrated pattern in production AI. Run the agent in parallel with your existing deterministic system. Let it make decisions, log them, compare them against what the real system did. But never let it act.
We used this at SIVARO for a logistics client in Rotterdam. Their dispatch system had rules that were 14 years old and nobody fully trusted. We ran an agent next to it for three weeks. The agent proposed reroutes; the legacy system executed its own. The agreement rate started at 61% and climbed to 89% once we fixed how the agent parsed weather data. That 28-point gap was the difference between rolling out with confidence and rolling out blind.
Here's the pattern:
python
def shadow_run(agent, legacy_system, event):
agent_decision = agent.decide(event)
legacy_decision = legacy_system.decide(event)
log_decision("agent", agent_decision, event.id)
log_decision("legacy", legacy_decision, event.id)
if agent_decision != legacy_decision:
log_divergence(event.id, agent_decision, legacy_decision)
# The real system acts. The agent never does.
return legacy_decision.execute()
Your shadow mode needs three outputs: agreement rate, divergence patterns, and a "would have been catastrophic" counter. That last one is the story you tell your risk officer. When the agent proposes clearing a cart that the legacy system keeps, you have evidence, not arguments.
A Practical Guide for Designing, Developing, and Evaluating LLM Agents nails the evaluation problem here. You can't evaluate an agent in production on a handful of hand-picked examples. You need a logged baseline. Shadow mode gives you exactly that — a control group generated by your own production traffic.
Feature Flags Won't Save You. Kill Switches Will.
Feature flags decide when something is on. A kill switch decides when something dies. Most teams build the first and forget the second.
A feature flag for an agent needs more than true/false. It needs per-action control. Let the agent read, but not write. Let it draft, but not send. Let it handle tier-1 tickets, but not refunds over $200. We built a policy object that evaluates every action before the agent's tool call executes:
python
class AgentPolicy:
def __init__(self, flag_service):
self.flags = flag_service
def can_act(self, user_id, action) -> bool:
level = self.flags.get("agent.permission_level", user_id)
if level == "off":
return False
if level == "shadow":
return False
if level == "read_only":
return action.op in {"get", "search", "list"}
if level == "draft_only":
return action.op != "execute"
if level == "full":
return True
return False
Permission levels are the kill switch. When a customer escalates, you don't turn off the whole deployment — you drop the permission level from "draft_only" to "read_only" for that user segment. That takes a config change, not a redeploy.
The other piece is a real kill switch: revoke the agent's credentials at the infrastructure level, not the application level. If the agent's service account is revoked in your identity provider, it stops. No code change. No rollback. That's the difference between a 5-minute incident and a 5-hour one. In 2025 we had an incident where a support agent was generating refunds with a hallucinated currency code. It took 20 minutes to notice and 40 seconds to kill via service account revocation. The kill switch is the only reason that was a footnote, not a postmortem.
Observability Is the Contract
You don't understand your agent until you can trace its steps. "AI agent observability production monitoring" sounds like a buzzword stack, but it's practical: you need to know what the model saw, what it decided, what tools it called, and how much it cost. For every single request.
Standard request logging doesn't work. LLM calls generate thousands of tokens, and logs designed for quick glance-through will drown. You need structured events with trace IDs. Every agent run gets one trace ID, and every step inside it — model call, tool invocation, human approval, retry — inherits that ID.
python
def trace_agent_step(trace_id, step_name, payload):
print(json.dumps({
"trace_id": trace_id,
"step": step_name,
"model": payload.get("model"),
"tool": payload.get("tool"),
"input_tokens": payload.get("input_tokens"),
"output_tokens": payload.get("output_tokens"),
"latency_ms": payload.get("latency_ms"),
"status": payload.get("status"),
"timestamp": datetime.utcnow().isoformat(),
}))
The non-negotiable metrics: token spend per trace, tool call latency, retry counts, and the number of times the agent hit a guardrail. Blaxel's production guide calls these the "vital signs" of an agent, and they're right. A rising retry count predicted one of our agent failures nine minutes before the first user complaint.
Cost tracking belongs in the same trace. In 2026, agent cost is still the thing that blindsides finance teams. A model that's 10% more accurate but 3x more expensive doesn't belong in production. Put a budget line in every trace and aggregate it per customer, per region, per feature. When the CFO asks why agent spend doubled, you'll have an answer that's a chart, not a shrug.
The Rollout Ladder: An AI Agent Rollout Strategy for Enterprise
You don't flip a switch. You climb a ladder with gates at every rung.
Here's the rollout ladder we use at SIVARO, tuned for systems that can't tolerate downtime:
Rung one: Shadow. The agent watches and logs. Zero user impact. This runs for one to four weeks, depending on traffic volume. You need at least 5,000 decision logs before you can measure agreement rate with confidence.
Rung two: Read-only assist. The agent can retrieve information but its output goes to a human reviewer. This is where you learn whether the agent's reasoning reads well in context. You'll discover the agent recommends correct actions but explains them in a tone that makes customers angry. Trust me, fix that before rung three.
Rung three: Cohort actions. The agent acts, but only for 5% of a carefully selected segment. New customers, not your top-100 accounts. Low-risk actions, not refunds over $100. This rung runs for at least two weeks. MachineLearningMastery's architecture breakdown calls this "gradual trust," and the name is accurate.
Rung four: Scaled actions with human audit. The agent handles 50-80% of traffic autonomously, but every action lands in an audit queue with a 24-hour review window. You're not preventing mistakes here; you're catching them early.
Rung five: Full autonomy. The agent does the work. Humans only see exceptions. Most teams never need this rung, and that's fine.
Every rung has two gates to advance: a time gate (the minimum duration) and a quality gate (agreement rate above X, escalation rate below Y, cost per task under Z). Both gates must pass. Time gates alone let bad agents advance. Quality gates alone let perfectionists block forever.
BusinessPlusAI's failure analysis found that premature autonomy is the #1 cause of production agent failures. They're not exaggerating. The systems that break aren't the ones that launch too slowly. They're the ones that launch too fast.
Workflows First, Agents Second
I'm going to say something that annoys vendors: most things you're calling agents shouldn't be agents.
This Toward Data Science breakdown of workflows vs agents makes the argument cleanly: a workflow is a predetermined path with LLM calls at fixed points. An agent is a model deciding its own path. Workflows are cheaper, faster, and infinitely more predictable. Agents are flexible but expensive and occasionally do things nobody asked them to do.
Look at your use case honestly. Support ticket triage? That's a workflow. You have categories, you have routing rules, the LLM classifies and moves on. Code review assistance? Also a workflow. The model reviews a diff, produces comments, a human reviews the comments.
Multi-step research that requires querying three systems and deciding which results matter? That starts to look like an agent. And honestly, even that can be a workflow with a decision point after each step. The agent emerges only when you can't predict the sequence of steps ahead of time.
Start deterministic. Add one degree of freedom. Measure. Add another. This is how you deploy AI without breaking existing systems: you never let the system depend on a capability you haven't verified in production traffic. Anthropic's guide recommends starting with the simplest composable pattern that works, and it's the single most ignored piece of advice in the industry.
Budgets, Timeouts, and Circuit Breakers
Every agent needs a leash. Not because the model is malicious, but because it's energetic. An agent with a valid API key will happily call a tool 40 times in one minute trying to achieve a goal. The model doesn't know what a dollar is.
We wrap every agent invocation in a guardrail layer. Timeouts stop runs that last too long. Budget caps stop runs that cost too much. Circuit breakers stop runs that keep failing:
python
class AgentGuardrail:
def __init__(self, max_duration_s=30, max_cost_usd=1.0, failure_threshold=3):
self.max_duration_s = max_duration_s
self.max_cost_usd = max_cost_usd
self.failure_threshold = failure_threshold
self.consecutive_failures = 0
async def run(self, agent, task):
start = time.monotonic()
cost_so_far = 0.0
async for step in agent.run_stream(task):
cost_so_far += step.cost_usd
elapsed = time.monotonic() - start
if elapsed > self.max_duration_s:
await agent.cancel()
return fallback_to_workflow(task)
if cost_so_far > self.max_cost_usd:
await agent.cancel()
return fallback_to_workflow(task)
if step.status == "failed":
self.consecutive_failures += 1
if self.consecutive_failures >= self.failure_threshold:
circuit_open = True
return fallback_to_human(task)
else:
self.consecutive_failures = 0
return agent.result
The fallback matters as much as the guardrail. When the agent times out, what happens? The worst answer is "nothing" — empty response, hanging UI, customer confusion. The right answer is a deterministic fallback: a workflow, a cached response, or a human handoff. The user of your system should never know that the agent failed. They should only know that the right thing happened.
This is the real meaning of "deployment without breaking existing systems." Every agent path needs a shadow path that's boring and reliable. The agent is a performance upgrade, not a replacement.
Failure Modes That Are Coming for You
You can't design guardrails for failures you haven't imagined. Here are the ones I've seen, and the ones you should design for.
The hallucinated tool call. The agent invokes a tool that doesn't exist. Your API returns a 404. The agent interprets that as "goal achieved" and reports success. Your audit log says everything is fine. Nothing is fine.
The tool-output injection. Your agent reads a document, and the document contains instructions that override the agent's system prompt. This isn't a security vulnerability in the model — it's a design vulnerability in your pipeline. Treat tool outputs as untrusted data.
The confirmation loop. The agent asks a human for confirmation. The human ignores it. The agent asks again. And again. Each retry costs tokens. Nobody notices until the monthly bill arrives. This failure mode has destroyed more agent budgets than model accuracy issues.
The slow creep. The agent behaves perfectly for six weeks. Then a dependency changes. The tool you were calling changed its API slightly. The agent starts failing in ways that don't trip your thresholds because each individual failure is small. The aggregate is not.
The Arxiv practical guide emphasizes that evaluation isn't a one-time pre-production task. It's a continuous process that tracks production behavior against your baseline. The team does this with weekly sampled evaluations: pull 200 random traces from last week, score them against your rubric, track the score over time. Drift becomes visible in days, not months.
AI Agent Observability: Production Monitoring That Pays for Itself
The observability backend I described costs money to run. It's worth it. Tracking tokens and tool calls feels like overhead until a customer asks you to explain why the agent spent $240 on their account in one day. Then it's the only reason you're still employed.
Our production setup at SIVARO processes around 200,000 events per second overall, and the agent subsystem has its own dedicated analytics pipeline. Every agent decision creates a record that flows into the same warehouse as the rest of the business data. No separate "AI observability" silo. The agent is a data product like any other. Its logs join the business facts. This matters when you need to answer questions like "did agent decisions improve customer retention?" or "did the agent increase refund velocity?"
Those questions are the real test. An agent that takes over support tickets but pushes resolution time from 4 hours to 9 hours isn't an improvement. It's automation theater. And you'll never know which one you have until your agent metrics live alongside your business metrics.
FAQ
How long should shadow mode run before the agent can act?
At least one full business cycle. If you're a retailer, cover a weekend and a Monday. If you're a B2B SaaS company, cover a month-end close. You need enough traffic to log at least 5,000 decisions. Two weeks is the practical minimum for most systems.
What's the minimum observability setup for production agents?
Trace IDs, step-level logs, token costs per trace, tool call latency, and guardrail hits. That's the floor. Aggregate everything per customer and per feature so you can answer "who did this affect" in under 30 seconds.
Should we build our own agent framework or use one off the shelf?
Use one, don't build one. The frameworks have matured a lot since 2024. But wrap any framework in your own control layer — policies, guardrails, tracing. The framework handles agent mechanics. You handle the governance.
How do we handle prompt injection from tool outputs?
Treat tool content as untrusted. Isolate instructions: the model's system prompt is separate from tool output in your prompt assembly. Don't let tool output redefine the task. If your model supports it, use output constraints that make tool content structurally distinguishable from instructions. Then test it with adversarial inputs before launch.
Who owns the agent in production?
One team. Not the ML team and the platform team and the product team. One owner with decision authority over rollout, permissions, and rollback. During incidents, the question "who can decide" needs one answer, not a committee.
How do we measure whether the agent is better than the old system?
Before you launch, define the comparison metrics. Resolution time, cost per resolution, customer satisfaction, error rate. Run the agent in shadow mode alongside the old system and compare on real traffic. The agent earns its slot or it doesn't.
What if the agent needs to be rolled back fully?
If your kill switch is a service account revocation, rollback takes seconds. The old deterministic system never left. It was the shadow path all along. That's why the deployment "without breaking" framing is the right one — the old system stays alive as the fallback until the new one has proven itself for months.
The Exit Plan Is the Entry Plan
I started this article with a wasted $4,300 and an agent that wouldn't stop querying. The fix wasn't a better model. It was a budget line, a timeout, and the willingness to treat the agent like a risky new dependency