How to Build Multi-Agent Systems in Production
I started my 2024 with a call from a founder at a mid-size fintech. They'd spent six months building a multi-agent system for credit risk assessment. Five agents communicating in a chain: data fetcher, risk calculator, compliance checker, explainer, and a writer. Sounded clean on the whiteboard.
On day one of production testing, agents started talking to each other in circles. One agent hallucinated a parameter that another couldn't parse. The compliance checker timed out because the risk calculator was stuck waiting for a GPU that didn't exist. The whole thing collapsed — not because the AI was bad, but because the system wasn't.
Multi-agent systems are distributed systems. Period. Treat them like anything else and you'll get the same kind of failure, just faster and more expensive.
In this guide I'll show you how to build multi-agent systems in production — from architecture and communication patterns to GPU cluster setups and observability. We'll cover what I've learned running SIVARO's own agent infrastructure and what I wish someone had told me before that fintech call.
The Distributed Systems Reality Check
Most people think "agent architecture" is about prompts and reasoning. It's not. It's about machines that have to find each other, talk to each other, agree on state, and handle failures — exactly like any distributed system.
Read Agentic Systems Are Distributed Systems. It's the single clearest framing I've seen. The article points out that each agent is a node with its own lifecycle, memory, and communication protocol. You need to decide on:
- Service discovery: how does agent B find agent C?
- Fault tolerance: what happens when the writer agent crashes mid-response?
- Consistency: does every agent see the same context at the same time?
- Load balancing: which agent instance handles which request?
Distributed systems ai agents architecture explained in one sentence: it's microservices with language models glued to them.
At SIVARO, we built an early prototype with each agent as a separate Docker container. We didn't think about retries. One transient network blip and the whole chain deadlocked. That's when I realized: this isn't a problem of prompt engineering — it's a problem of engineering.
How to Build Multi-Agent Systems in Production: Architecture Decisions
You've got two main patterns: orchestrator-based and mesh-based.
Orchestrator Pattern (Our Default)
A single coordinator agent receives the user request, decides which sub-agents to call in sequence or parallel, aggregates results, and returns a final answer. This is what Anthropic's Claude uses for tool use. It's what we use at SIVARO for most customer deployments.
Why? Because it gives you a single point of control. You can enforce timeouts, retry policies, and logging at the orchestrator level. Debugging is straightforward — the orchestrator's trace shows every step.
We tested mesh-based architectures (agents talking directly to each other) in early 2025. It worked fine for 3 agents. At 10 agents, we saw message storms. Agent A sent a request to B, B forwarded to C, C couldn't answer so it asked A, A re-asked B — infinite loop. We had to implement TTLs and hop limits. In the end, the orchestrator model was simpler to reason about and cheaper to run.
Trade-off: The orchestrator is a single point of failure. Mitigate it with redundancy (active-passive) and stateless design.
Mesh Pattern (When You Need It)
If you have agents that need to negotiate or form temporary alliances (e.g., multi-agent simulations or gaming), a mesh might fit. But even then, add a supervisor agent that monitors for cycles.
python
# Simple orchestrator pattern in Python (pseudocode)
class Orchestrator:
def run(self, user_input):
context = self.initial_analysis(user_input)
if context["needs_data"]:
data = self.call_agent("data_fetcher", context)
context.update(data)
if context["needs_risk"]:
risk = self.call_agent("risk_calculator", context)
context.update(risk)
# compliance check must be after risk
compliance = self.call_agent("compliance", context)
explanation = self.call_agent("explainer", compliance)
return self.call_agent("writer", explanation)
That's it. Orchestrate in code, not in prompts. Prompts shouldn't decide the flow — your infrastructure should.
GPU Cluster Setup for Large Language Model Training
You can't build production multi-agent systems without training or fine-tuning the models that power your agents. And that means gpu cluster setup for large language model training is a prerequisite skill.
I've seen teams try to run a 70B parameter model on two A100s and wonder why training takes three weeks. The answer is that distributed training isn't optional — it's mandatory.
Choosing Your Topology
For a cluster of 8+ GPUs, you need to decide between:
- Data parallelism: each GPU has a full model copy, processes different batches, and syncs gradients. Works well if the model fits on one GPU.
- Tensor parallelism: split the model's layers across GPUs. Required for models > ~20B parameters on A100s.
- Pipeline parallelism: spread different layers across GPUs, each GPU processes a mini-batch sequentially.
For LLMs, you almost always use a combination. At SIVARO we default to 3D parallelism (data + tensor + pipeline) using NVIDIA's NeMo or PyTorch FSDP.
The Distributed training in Amazon SageMaker AI documentation covers the common setups. Amazon SageMaker makes it relatively painless to launch a cluster with torchrun. But if you're running bare metal, you'll need to handle NCCL configuration, network topology, and NVLink.
A Practical Example
Here's a torchrun command we use internally for fine-tuning a 13B parameter model across 4 nodes (8 GPUs each):
bash
torchrun --nproc_per_node=8 --nnodes=4 --rdzv_endpoint=192.168.1.100:29500 --rdzv_backend=c10d finetune.py --model_name meta-llama/Llama-2-13b-hf --batch_size 4 --gradient_accumulation_steps 8 --tensor_parallel_size 2 --pipeline_parallel_size 2 --data_parallel_size 8
The parameters --tensor_parallel_size 2 and --pipeline_parallel_size 2 split the model across 4 GPUs per node, while data parallelism replicates across the rest. Tuning these ratios is the art. For deeper understanding, read Distributed Training & Large-Scale Systems — they break down the math behind throughput and memory.
Key metric: you want model flops utilization (MFU) above 40%. Below that, your cluster is underutilized. We hit 52% on a 64-GPU cluster with a 70B model by tweaking the pipeline stages and micro-batch sizes.
Agent Communication: Sync vs Async
Should an agent wait for a response from another agent, or should it fire and forget?
Sync is simpler. The orchestrator calls an agent, waits for the result. If the agent crashes, the orchestrator can retry or escalate. We use sync for agents that are part of the critical path — e.g., risk calculation must finish before compliance.
Async is better for non-blocking operations. For example, a logging agent that records every decision doesn't need to block the main flow. Use a message queue (Kafka, NATS, RabbitMQ) to decouple agents.
Here's a simple async pattern using NATS:
python
import asyncio, nats
async def risk_calculator_agent(msg):
data = msg.data.decode()
# do the calculation
response = compute_risk(data)
# publish result to a reply subject
await nc.publish(msg.reply, response.encode())
async def main():
nc = await nats.connect("nats://localhost:4222")
await nc.subscribe("risk.request", cb=risk_calculator_agent)
# orchestrator publishes requests, risk publishes back
We tested both at SIVARO. Sync is our default for chains of 3-5 agents. Async adds latency (queue round-trips) and complexity (dead letter handling). Only go async when you have agents that can run independently — e.g., data enrichment agents that each process a different field, then merge results.
How to Build Multi-Agent Systems in Production: State Management
This is where I see the most failures. Agents are stateless by design (good!), but the conversation or context isn't. You need to persist:
- The user's original input
- Each agent's output
- The final response
- Any intermediate state that might be needed for retries
External State Store
Never rely on an agent's internal memory. Store everything in a database — PostgreSQL, DynamoDB, or a key-value store. We use Redis for short-lived context (TTL of 15 minutes) and Postgres for audit logs.
python
# Store agent outputs with a trace ID
def store_agent_output(trace_id, agent_name, output):
conn.execute(
"INSERT INTO agent_traces (trace_id, agent_name, output, created_at) "
"VALUES (%s, %s, %s, NOW())",
(trace_id, agent_name, json.dumps(output))
)
Idempotency
If an orchestrator retries a call to the risk calculator (because of a network blip), the risk calculator must not double-compute. Use idempotency keys: the orchestrator passes a unique request_id. The agent checks if it already processed that ID — if yes, return cached result.
We learned this the hard way. A customer's compliance agent ran twice on the same request, resulting in two different "approved" statuses because the agent had a random seed. The system flagged a false positive. Idempotency fixed it.
Testing Multi-Agent Systems in Production
Unit tests on individual agent prompts catch basic syntax errors. They don't catch integration bugs, race conditions, or cascading failures.
Chaos testing is your friend. At SIVARO, we run a weekly "agent chaos day." We kill random containers, inject network latency, throttle GPU memory, and simulate model hallucinations (return an empty string for one agent). We measure: does the orchestrator timeout? Does it retry? Does it degrade gracefully?
We also use deterministic replay. Record every input and output in production. Then replay that same sequence in a staging environment, to verify that a code change doesn't break behavior.
One tool we've found helpful: ToxiProxy for network chaos. It's lightweight and works with any TCP connection.
Observability: Tracing Agent Thought Chains
Your logs are useless if they're scattered across five different agents. You need distributed tracing.
OpenTelemetry is the standard. Instrument each agent to create spans that carry the same trace ID. Include the full input and output in the span attributes.
python
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def run_risk_agent(context):
with tracer.start_as_current_span("risk_agent") as span:
span.set_attribute("input", json.dumps(context))
result = calculate(context)
span.set_attribute("output", json.dumps(result))
return result
Ship traces to a backend like Jaeger or Tempo. We use Grafana Cloud at SIVARO. When something goes wrong, pull up the trace and see exactly which agent timed out, how long each step took, and what data passed between them.
Pro tip: also log the GPU memory usage at each step. We've seen agents hit OOM because a previous agent returned an unexpectedly large payload. That's a distributed memory leak.
Security and Isolation
Your client's data goes through multiple agents. Some agents might be third-party models or even public APIs. You need boundaries.
- Agent sandboxing: run each agent in its own container or Firecracker microVM. Limit network access — a data-fetching agent should only be able to reach the external API it needs, not the compliance database.
- Data redaction: strip PII before passing context to external agents. We use a dedicated "redaction agent" that runs first.
- Authentication: each agent call should carry a signed token from the orchestrator. Don't let agents impersonate each other.
At SIVARO, we had a close call in early 2026: a developer accidentally gave an agent write access to the production database. The agent's prompt was "update customer balance" — it tried to update every row. The write guard we'd put on the orchestrator prevented it. That guard was a single line: if "write" in intent and not orchestrator.safe_mode:.
FAQ
Q: How many agents should I use?
A: As few as you can. Start with 2-3. Add more only when you have clear isolation boundaries (different latency requirements, different data sources). Many teams I've worked with have 5-7 at most.
Q: What's the biggest cost?
A: GPU compute for model inference, not training. Running a 70B parameter model for each agent call gets expensive fast. Use smaller, specialized models per agent instead of one giant generalist.
Q: How do I debug a multi-agent system when something goes wrong?
A: Start with distributed traces. If that's not set up, you're blind. Then check agent logs in chronological order using trace ID. Third, replay the exact input in staging.
Q: Do I need to fine-tune models for each agent?
A: Sometimes. Off-the-shelf models work for generic tasks. But if an agent needs to follow a specific domain schema (e.g., medical codes, legal clauses), fine-tuning saves tokens and reduces errors.
Q: Can I use serverless functions for agents?
A: For simple, stateless agents, yes. But watch out for cold starts and timeouts. Our compliance agent needs 15 seconds to run — that's above many Lambda timeouts. Use provisioned concurrency or containers.
Q: What's the biggest mistake teams make?
A: Over-reliance on prompts for orchestration. They try to get the model to decide which agent to call next. It's unreliable and expensive. Use code for flow, prompts for content.
Q: How do you handle agent hallucinations in production?
A: Add a validation agent that checks the output of critical agents against a known schema or set of rules. If it fails, fall back to a cached or default response.
Q: How to build multi-agent systems in production without a large budget?
A: Start with a single orchestrator and local models. Use quantization (4-bit, 8-bit) to fit larger models on fewer GPUs. You can get decent results with a couple of A100s or even consumer GPUs (RTX 4090) for small agent loads.
What I've Learned
Building multi-agent systems in production is 80% distributed systems engineering and 20% AI. The tools from the last decade — Kubernetes, message queues, service meshes, distributed tracing — apply directly. Use them. Don't reinvent.
At SIVARO, we ship multi-agent systems for clients in finance, healthcare, and logistics. Every single one boiled down to: define clear agent boundaries, enforce timeouts, store state externally, and test for chaos.
The industry is moving fast. By 2026, we've seen agent systems become the norm for complex workflows. But the fundamentals haven't changed. The ones that work are the ones that treat agents as distributed nodes, not as magic oracles.
If you take one thing from this guide: start with the infrastructure, not the prompts. Your agents will thank you.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.