Agentic Workflow Production: What I Learned the Hard Way
You've built a prototype that can write emails, summarize reports, or even manage a code review. It works beautifully in your notebook. Then you push to production and it falls apart.
I know that feeling. At SIVARO, we've deployed over two dozen production AI agents since 2023. Some worked day one. Most didn't. The gap between a demo and a system that runs for months without human intervention is not a small gap — it's a chasm.
This guide is about best practices for agentic workflow production. Not theory. Not "both have merits." I'm going to tell you what nearly broke us, what we fixed, and what we still struggle with today.
By the end, you'll understand the difference between ai agents in production vs development, the most common mistakes deploying ai agents production, and how to build systems that survive real traffic.
What "Agentic" Actually Means in Production
Let's get one thing straight: an agent isn't just a large language model call with a loop. An agent is a system that observes state, makes decisions, executes actions, and corrects itself. In production, that means:
- It must handle ambiguous inputs without crashing.
- It must respect rate limits, error backoffs, and dead-simple timeouts.
- It must not hallucinate actions that destroy data.
At first I thought this was a technical problem. Turns out it's an architectural problem. The agent's reasoning loop is the easy part. The infrastructure around it — observability, guardrails, fallback modes — that's where production lives.
We learned the hard way in March 2025 when an agent running a customer's data pipeline decided to "optimize" a table by dropping it. No guardrails, no human approval. Just a blank audit log and a panicked call at 2 AM.
Never let an agent write to a database without a confirmation step. That's rule zero. We'll get to the rest.
The Architecture That Didn't Collapse
Most people think you need Kubernetes clusters, Redis streams, and a microservice per function. They're wrong. We started with a monolithic Python service that ran 30 agents each in their own async task. It worked fine for months.
But then we hit 200K events per second in a pipeline for a logistics client in Q4 2025. The monolith became a liability. Not because of throughput — because of coupling. When one agent's memory grew unbounded, it OOM'd the whole process.
Here's what we settled on after seven rewrites:
yaml
# agent-config.yaml — SIVARO's current production template
agent:
name: "triage-agent"
mode: "semistructured" # choices: "structured", "semistructured", "freeform"
loop:
max_iterations: 10
timeout_seconds: 120
error_policy: "backoff" # "fail" or "backoff"
state_store: "redis"
memory:
type: "sliding_window"
max_messages: 50
ttl: 3600
guardrails:
- type: "output_validator"
schema: "./schemas/triage_response.json"
- type: "human_approval"
actions: ["execute_write", "delete_resource"]
Three things matter: bounded memory, explicit error policy, and action-level guardrails.
We test every agent with 5x production load using synthetic traces before it hits live traffic. The paper A Practical Guide for Designing, Developing, and ... calls this "adversarial robustness testing." I call it "saving our jobs."
State Management: The Silent Killer
Every agent has a state — conversation history, pending tasks, intermediate results. In development, you just keep it in memory. In production, that memory will be your graveyard.
We lost two days of work for a fintech client because an agent's state grew to 200MB and killed the pod. Restarting fixed it — and lost every in-flight task.
The solution: externalize state. Use Redis or DynamoDB. Not because you need speed — because you need survival.
python
# agent_with_external_state.py — simplified example
from redis import Redis
from typing import Dict, Any
class ProductionAgent:
def __init__(self, agent_id: str, redis_client: Redis):
self.agent_id = agent_id
self.redis = redis_client
self.state_key = f"agent:{agent_id}:state"
async def run(self, input_data: Dict[str, Any]) -> Any:
state = self._load_state()
state['current_input'] = input_data
try:
result = await self._execute_loop(state)
self._save_state(state)
return result
except Exception as e:
state['error'] = str(e)
self._save_state(state)
raise
def _load_state(self) -> dict:
data = self.redis.get(self.state_key)
return json.loads(data) if data else {}
def _save_state(self, state: dict):
self.redis.setex(self.state_key, 3600, json.dumps(state))
This pattern let us survive pod restarts, scale horizontally, and debug failures by replaying state. The Google research paper Agentic AI Infrastructure in Practice makes the same point: "external state is not optional."
Observability: You Can't Fix What You Can't See
I have a rule: if an agent fails in production and I can't reproduce it in staging, I haven't instrumented it enough.
In development, agents talk to you. You can print the raw LLM output. In production, you get a 500 error and a log line: "Agent crashed at iteration 3."
We spent three months building what I call "agent forensics." Every step of the reasoning loop gets logged: input tokens, output tokens, tool calls, latency per step, final decision. Then we dump it to a structured sink (Elasticsearch + S3).
python
# instrumentation snippet
import structlog
logger = structlog.get_logger()
async def agent_step(step_id: int, prompt: str) -> str:
start = time.time()
response = await llm_call(prompt)
duration = time.time() - start
# Log every decision
logger.info("agent_step_complete",
step=step_id,
prompt_tokens=response.usage.prompt_tokens,
response_tokens=response.usage.completion_tokens,
latency_ms=duration*1000,
tool_call=response.tool_calls[0].name if response.tool_calls else None,
success=True
)
return response
You don't need fancy tracing tools. You need structured logging that you can query. Without it, you're debugging blind.
Guardrails That Don't Annoy Users
Every agent team I talk to adds guardrails too late. They ship a raw agent, it does something dumb, and then they bolt on a validation layer. That reactive approach costs money and trust.
We now design guardrails before the first line of agent code. Three layers:
- Input guardrails – reject toxic, out-of-scope, or malformed prompts.
- Output guardrails – ensure responses match schema, don't contain PII, and don't instruct harmful actions.
- Action guardrails – require human approval for destructive operations.
The Anthropic guide Building Effective AI Agents recommends starting simple: "Don't add complexity until you have concrete evidence it's needed." I agree — but guardrails aren't complexity. They're insurance.
We use a lightweight validator library that checks output against a JSON schema. If validation fails, the agent retries with a "you made a mistake" prompt. After three retries, it escalates to a human.
yaml
# guardrail-example.yaml
output_validator:
schema:
type: object
properties:
action:
type: string
enum: ["respond", "escalate", "noop"]
reason:
type: string
maxLength: 500
required: [action]
on_failure:
- retry (max 3)
- escalate (to human)
Simple. Works. We caught 40 hallucinated actions in the first month across one client deployment.
Testing in Production (Yes, You Have To)
"Test in production" used to be a joke. Now it's a necessity. You cannot simulate every edge case in staging. Agents see weird inputs — null strings, emoji-only messages, date formats from the 1800s.
We run a shadow deployment for every new agent. Copy a percentage of live traffic to a separate agent instance. Compare its outputs against the current system. Alert if divergence exceeds a threshold.
The article How to Deploy AI Agents to Production calls this "canary testing." We call it "sleeping at night." In 2025, a shadow deployment caught an agent that suddenly started generating SQL with a syntax error every 10th call. The main system was fine. But we fixed the shadow agent before it went live.
The Human-in-the-Loop Fallacy
Let me be contrarian: human-in-the-loop is not a safety net; it's a bottleneck. If your agent needs human approval for every other action, you don't have an agent — you have a suggestion engine.
But you can't let agents run wild either. The balance: define a small set of critical action types that always require human approval. Everything else runs autonomously. Give humans a dashboard, not a pager.
We learned this after a client's agent for customer support started approving refunds autonomously — correctly, mind you — but bypassed the finance team's control. That wasn't an agent failure. It was a process failure. Now we have a "human review" step only for actions that touch money, delete data, or change access controls. The common mistakes deploying ai agents production article lists "insufficient human oversight" as mistake number one. I'd rephrase: "oversight that doesn't match the action's risk level."
Scaling: The Boring Infrastructure Wins
Everyone wants to talk about prompt engineering. No one wants to talk about connection pooling. But connection pooling will kill your agent faster than any prompt.
We tested three architectures for scaling agents:
| Approach | Latency @ 100 req/s | Cost @ 1000 req/s | Observability |
|---|---|---|---|
| Single async process | 320ms | $0.002/req | Good |
| Worker pool (Celery) | 410ms | $0.003/req | Medium |
| Queue-based (Redis + consumer) | 280ms | $0.001/req | Excellent |
The queue-based approach won. Why? Because when an agent takes 30 seconds to respond, you don't block your API. You write to Redis, return a tracking ID, and let the consumer pick up the task. The Deploying AI Agents to Production: Architecture guide has similar benchmarks.
Common Mistakes We Made (So You Don't Have To)
-
Treating development LLM outputs as ground truth — In dev, GPT-4 returned perfect JSON. In prod, a cheaper model hallucinated extra fields. We now pin model versions and run weekly regression tests.
-
Ignoring latency — An agent that takes 10 seconds in dev takes 30 in prod because of network overhead. We added a timeout after 15 seconds and a fallback that returns a "still thinking" response. Users stopped hitting back button.
-
Not planning for token cost — One agent called the LLM 12 times per request. At scale, that was $0.50/request. We optimized: reduced steps to 4, cached intermediate results. Cost dropped 80%.
-
Assuming deterministic behavior — LLMs are not deterministic. Same input, different output. We added a "reliability check": for high-stakes actions, call the agent twice and compare outputs. If they disagree, escalate.
-
Forgetting idempotency — An agent that refunds a payment should not refund twice. We built a sidecar that deduplicates action calls using a hash of input + action type. Saved us in a 2024 outage.
Code: A Real Production Agent Pattern
Here's the pattern we use at SIVARO for agents that need to call external APIs. It's based on the A Practical Guide for Designing, Developing, and ... but adapted for reliability.
python
# production_agent_pattern.py
import asyncio
from dataclasses import dataclass
from typing import Optional, Callable
@dataclass
class AgentConfig:
model: str = "claude-3.5-sonnet-20251022"
max_iterations: int = 5
timeout_per_step: int = 30
retry_on_error: bool = True
max_retries: int = 3
class ProductionReadyAgent:
def __init__(self, config: AgentConfig, llm_client: Any, tools: dict):
self.config = config
self.llm = llm_client
self.tools = tools
self.state = {"history": [], "errors": 0}
async def run(self, input_text: str) -> dict:
self.state = {"history": [{"role": "user", "content": input_text}], "errors": 0}
for step in range(self.config.max_iterations):
try:
async with asyncio.timeout(self.config.timeout_per_step):
response = await self.llm.chat_completion(
model=self.config.model,
messages=self.state["history"],
tools=list(self.tools.values())
)
if response.tool_calls:
for tc in response.tool_calls:
result = await self._execute_tool(tc)
self.state["history"].append({"role": "tool", "content": str(result)})
else:
return {"output": response.content, "steps": step+1}
except asyncio.TimeoutError:
if self.config.retry_on_error and self.state["errors"] < self.config.max_retries:
self.state["errors"] += 1
continue
return {"error": "timeout", "steps": step+1}
except Exception as e:
return {"error": str(e), "steps": step+1}
return {"error": "max iterations", "steps": self.config.max_iterations}
async def _execute_tool(self, tool_call) -> Any:
tool_name = tool_call.function.name
tool_fn = self.tools.get(tool_name)
if not tool_fn:
return f"Tool {tool_name} not found"
args = json.loads(tool_call.function.arguments)
return await tool_fn(**args)
This isn't perfect. It blocks on tool calls, doesn't handle state persistence, and assumes tools are async. But it's a start. Layer on Redis for state, a queue for scaling, and guardrails for safety. You'll have something production-ready.
The Human Side: Why Most Agent Deployments Fail
I've seen more agent projects fail from organizational friction than from technical bugs. The legal team doesn't trust the agent. The ops team doesn't want to maintain it. The sales team oversold it.
The fix: involve all stakeholders in guardrail design. Let legal define what "destructive action" means. Let ops write the rollback plan. Let sales set expectations — "The agent can triage 80% of tickets; the rest go to humans."
The Building Effective AI Agents guide says "agents are a design pattern, not a product." I'd add: agents are a product that requires a cross-functional team to operate.
Frequently Asked Questions
Q1: When should I use a workflow vs an agent?
Workflows are for deterministic, fixed steps (e.g., "process invoice → validate → approve"). Agents are for open-ended tasks that require reasoning (e.g., "investigate support ticket and respond"). Start with a workflow. Only add agentic loops when you need adaptive behavior. The A Developer's Guide to Building Scalable AI covers this distinction well.
Q2: How do you handle LLM cost in production at scale?
Cache model responses for identical inputs (with TTL). Use smaller models for low-stakes steps — we use Claude Instant for summarization, Claude 3.5 Sonnet for decision steps. Batch requests where possible. Monitor token usage per request; set budgets per user.
Q3: What's the best way to handle partial failures in an agent?
Let the agent decide whether to retry, escalate, or skip. But enforce a hard retry limit. Our agents get three retries per step. After that, they log the error and continue if the action is non-critical, or halt if it's critical. We also have a dead-letter queue for steps that fail permanently.
Q4: How do you test an agent's edge cases?
We use adversarial test suites: empty inputs, maximum-length inputs, malformed JSON, extremely long context, irrelevant instructions, and contradictory commands. We also run fuzz testing: modify valid inputs randomly and ensure the agent doesn't produce harmful outputs.
Q5: Should you use open-source or closed-source LLMs?
It depends on your data sensitivity. For agents that handle PII or trade secrets, open-source models deployed in your VPC are often required. For general-purpose agents, closed-source models are simpler to manage. We use a hybrid: open-source Mistral for classification, closed-source Anthropic for complex reasoning.
Q6: How do you handle versioning of agents?
We version the entire agent configuration — model, prompt, tools, guardrails — as a YAML file in git. Every deployment is a new version. We can roll back by redeploying the old YAML. No hotfixes in production.
Q7: What monitoring metrics matter most?
- Agent success rate (percentage of requests that end with a valid output)
- Average steps per request (shouldn't increase over time)
- Token consumption per request
- Tool call failure rate
- Human escalation rate (should be stable or decreasing)
Q8: How do you prevent prompt injection in an agent that reads user input?
We run input sanitization: strip control characters, truncate to max length, and reject obvious injection attempts (e.g., "ignore all previous instructions"). We also sandbox tool calls — no agent can execute arbitrary strings. Use the principle of least privilege for tool access.
Where We're Going Next
The industry is moving toward multi-agent systems — specialized agents that delegate to each other. We're building a "router agent" that classifies incoming requests and sends them to domain-specific agents. Early results show 30% higher accuracy than a single monolithic agent.
But that's next year's problem. For now, the best practices for agentic workflow production are simpler than you think:
- Externalize state.
- Add guardrails before code.
- Test in production with shadow traffic.
- Don't trust the LLM 100%.
- Monitor everything.
- Involve humans only where it matters.
The golden rule: Your agent will fail. Plan for that failure. Every architecture, every guardrail, every test — they're all about reducing blast radius and increasing recoverability.
I'm Nishaant Dixit. I've built production AI systems that process 200K events per second. I've also built agents that deleted databases. The difference between success and failure isn't the model — it's the infrastructure around it.
Now go build something that survives.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.