Agent to Agent Architecture: A Production Example

Free Technical Audit

Expert Review

July 21, 2026. Two weeks ago I watched an agent swarm we built for a supply chain client hit a deadlock that cost them $45,000 in idle inventory. Not because the agents were dumb. Because they couldn’t talk to each other fast enough.

Agent-to-agent architecture is the hot term. Every startup claims their framework makes agents “collaborate seamlessly.” Most demos work on a laptop with three agents and no load. Production is a different animal. You need real protocols, real timeouts, real observability. And you need to pick the right pattern before you write a single line of code.

In this guide I’ll walk through a agent to agent architecture production example — an inventory forecasting system we deployed at SIVARO for a midwestern retailer. I’ll show you the code that worked, the mistakes we made, and the protocols that held up under load. You’ll learn how to design, test, and deploy multi-agent systems that don’t fall over the second you add latency.

Let’s start with the dirty truth no one tells you.

Why Most Agent-to-Agent Demos Fail in Production

Three years ago I thought agent orchestration was a brand problem. Turns out it’s a protocol problem.

Most frameworks like LangGraph, CrewAI, or AutoGen let you define agents that pass messages. They work fine in a notebook. Then you put them behind an API and suddenly agents start talking over each other, timing out, or sending malformed JSON. The AI Agent Frameworks survey from IBM calls out that 68% of teams who build multi-agent prototypes hit integration failures when moving to production. I believe that number.

The root cause: ad-hoc messaging. Teams define custom JSON schemas for agent communication, then hardcode endpoints and retry logic. It works until you need to swap an agent or add a new one. The coupling is too tight. You need a contract — a protocol — that every agent agrees to at the network level.

We learned this the hard way in early 2025. We had three agents: a demand forecaster, a supplier matcher, and a risk assessor. They communicated over HTTP with hand-crafted payloads. When we scaled from 10 SKUs to 10,000, the agents started dropping messages. We spent two weeks debugging a race condition in our custom queue. Never again.

The Protocol Problem: A2A vs MCP vs Custom

You have three choices today for agent-to-agent communication:

Agent-to-Agent Protocol (A2A) — Google’s standard, now backed by 30+ companies.
Model Context Protocol (MCP) — Anthropic’s model for tool/agent interaction.
Custom — roll your own.

I’ve used all three. Here’s my take.

A2A is the right choice for agent-to-agent messages in multi-agent systems. It defines a standard envelope with agentId, messageId, metadata, and payload. It supports streaming, idempotency keys, and error codes. We adopted it in Q4 2025 and it cut our integration time by 40%. The AI Agent Protocols report on arxiv confirms that A2A reduces coupling and improves observability — exactly what we saw.

MCP is fine when an agent talks to a tool (a database, a search API). It’s not built for agent-to-agent peer communication. The agentic framework comparison from LangChain makes this distinction explicit: MCP for agent‑tool, A2A for agent‑agent.

Custom is a trap. Yes, you can build exactly what you want. But you’ll spend 80% of your time writing message routing, error handling, and serialization that someone else already got right. The open source agentic frameworks list from AIMultiple shows that the vast majority of serious multi-agent projects now use A2A or MCP. We are fully A2A for agent-to-agent. Period.

A Real Production Example: Inventory Forecasting System

Let me show you a concrete agent to agent architecture production example. This is the system we deployed in May 2026 for a retailer with 200 stores and 15,000 SKUs.

Architecture Overview

We have four agents:

Forecaster Agent — predicts demand per SKU using a time series model.
Inventory Agent — checks current stock and lead times.
Supplier Agent — queries supplier catalogs and prices.
Orchestrator Agent — routes messages, manages state, and handles failures.

All agents communicate over A2A. Each agent exposes an HTTP endpoint that accepts A2A envelopes. The orchestrator is the message hub — it sends tasks and collects results. But agents can also talk directly (peer‑to‑peer) if the orchestrator delegates.

Here’s the A2A envelope schema we use:

json
{
  "agentId": "forecaster-v2",
  "messageId": "msg-20260721-001",
  "timestamp": "2026-07-21T14:30:00Z",
  "metadata": {
    "idempotencyKey": "req-abc123",
    "priority": "high",
    "correlationId": "corr-8899"
  },
  "payload": {
    "type": "forecast_request",
    "data": {
      "sku": "A1001",
      "storeId": "store-42",
      "dateRange": {"start": "2026-08-01", "end": "2026-08-31"}
    }
  }
}

The orchestrator sends this to the forecaster agent. The forecaster responds with a forecast_result envelope. Then the orchestrator sends a stock_check to the inventory agent. Simple.

But simple is not enough for production. You need a deployment guide.

AI Agent Deployment Challenges and Solutions

We hit every common problem. Let me walk through three: latency, state, and observability.

Latency: The Silent Killer

Agents are slow. Not the inference — the network. Our forecaster agent runs a LightGBM model that predicts in 12ms. But the HTTP round trip between agents averages 85ms. Add retries (three attempts), and you’re at 300ms per agent call. With four agents in a chain, that’s over a second just for coordination.

Solution: We switched to persistent connections using HTTP/2 multiplexing and kept alive. That cut latency by 35%. We also added a local cache at the orchestrator for common queries (like store metadata). The Agentic AI Frameworks guide from Instaclustr recommends exactly this — inter‑agent communication should be over long‑lived connections, not one‑off requests.

State: Where Was That Agent?

Agents are stateless by design. But the orchestrator needs to track which messages are in flight. Without state, lost messages mean deadlocks.

Solution: We use a Redis‑backed state store. Every message gets a correlationId and a state field. The orchestrator logs sent messages in Redis with a TTL of 5 minutes. If a response doesn’t arrive in time, the orchestrator re‑sends (idempotency keys prevent duplication).

Here’s the core logic in Python:

python
import redis, uuid, json

class Orchestrator:
    def __init__(self):
        self.redis = redis.Redis(host='localhost', port=6379, decode_responses=True)
    
    def send_message(self, agent_url, payload):
        msg_id = str(uuid.uuid4())
        envelope = {
            "agentId": self.agent_id,
            "messageId": msg_id,
            "timestamp": "2026-07-21T14:30:00Z",
            "metadata": {"idempotencyKey": msg_id, "correlationId": self.correlation_id},
            "payload": payload
        }
        # store in redis with 300s TTL
        self.redis.setex(f"msg:{msg_id}", 300, json.dumps(envelope))
        response = requests.post(agent_url, json=envelope)
        if response.status_code == 200:
            self.redis.delete(f"msg:{msg_id}")
        return response

This pattern is simple. It works. We learned it from building our agent to agent protocol deployment guide for the team — always assume the network will lose a message.

Observability: Who Dropped the Ball?

When a forecast comes back late, you need to know why. Standard logging isn’t enough. You need distributed tracing across agents.

Solution: Every envelope carries a correlationId. We instrument every agent with OpenTelemetry. Each agent’s span includes the messageId and the previous agent’s trace ID. In production, we use Jaeger to visualize the full chain. The AI Agent Protocols survey on arxiv mentions that 45% of multi-agent systems lack proper tracing — and those systems fail more often. We saw that first hand.

Designing for Failure: Timeouts, Retries, Rate Limits

You can’t design an agent system without assuming everything will break. Here’s our failure‑handling checklist.

Timeouts: Each agent call gets a configurable timeout. We start at 2 seconds for forecast, 5 seconds for inventory (which queries a DB), 10 seconds for supplier (which calls an external API). If it times out, we retry once after 200ms, then escalate to the orchestrator’s dead‑letter queue.

Retries: Use exponential backoff with jitter. Don’t retry immediately — you’ll just amplify the failure. We use backoff=min(2^n, 30) seconds with 50% jitter. After three retries, the orchestrator marks the message as failed and logs it to a Dead Letter Queue (DLQ). A human or a supervisor agent reviews the DLQ.

Rate Limits: Agents can get overwhelmed. Our forecaster can handle 100 requests per second. The inventory agent only 20 (due to database connection limits). We implement a token bucket at the orchestrator. Each agent registers its capacity. The orchestrator queues or throttles messages accordingly.

Here’s the throttler:

python
import time, threading

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity
        self.lock = threading.Lock()
    
    def consume(self, tokens=1):
        with self.lock:
            now = time.time()
            self.tokens = min(self.capacity, self.tokens + (now - self.last_refill) * self.rate)
            self.last_refill = now
            if self.tokens >= tokens:
                self.tokens -= tokens
                return True
            return False

This is basic but necessary. Without rate limiting, the supplier agent would get hammered by retries and permanently degrade. The modern agent protocol standards list from SSONetwork explicitly recommends rate limiting at the protocol level. We enforce it at the orchestrator, not the agent — that way the agent stays simple.

Agent Coordination Patterns: Pipeline, Mesh, Hub-and-Spoke

You need a pattern. Don’t wing it.

Pipeline

Agents in sequence: Forecast → Inventory → Supplier. Simple. Good for predictable flows. Bad for fault tolerance — one slow agent blocks the whole chain.

Mesh

Every agent talks to every other agent. Maximum flexibility. Maximum chaos. We tried this. It’s a nightmare to debug. One agent sends a request to the wrong peer and you’re hunting through logs. Use only if you have a small, fixed set of agents that need tight coupling.

Hub-and-Spoke (the winner)

One orchestrator agent routes all traffic. Agents only talk to the hub. This is what we use. It centralizes state, tracing, rate limiting, and retries. The tradeoff is a single point of failure — you need to run multiple orchestrator instances (we run three behind a load balancer). But the clean architecture is worth it. The LangChain guide on agent frameworks argues that hub‑and‑spoke is the most production‑proven pattern. I agree.

Testing Agent-to-Agent Communication in Production

You can’t unit test a network. You need integration tests that simulate real conditions.

We wrote a test harness that spins up three instances of each agent in Docker, then pumps synthetic A2A messages. We measure latency, success rate, and recovery time when we kill an agent mid‑request.

Here’s the test for idempotency:

python
import unittest, requests

class TestIdempotency(unittest.TestCase):
    def test_dedup(self):
        payload = {"type": "forecast_request", "data": {"sku": "A1001"}}
        env = build_envelope(payload, idempotency_key="same-key")
        r1 = requests.post("http://forecaster:8000/agent", json=env)
        r2 = requests.post("http://forecaster:8000/agent", json=env)
        self.assertEqual(r1.json()["data"]["result"], r2.json()["data"]["result"])

If the agent returns different results for the same idempotency key, the test fails. We also run chaos engineering — randomly drop network packets, delay responses by 500ms, kill a container. The orchestrator must recover within 10 seconds.

FAQ

Q: What’s the simplest A2A library for Python?
Use Google’s a2a-sdk (pip install a2a-sdk). It handles envelope creation, validation, and retries. We combine it with FastAPI for the agent endpoints.

Q: Can I mix A2A and MCP in the same system?
Yes. Use A2A between agents, MCP when an agent calls a tool (database, search, calculator). The orchestrator acts as the bridge.

Q: How do I handle versioning of agent APIs?
Each envelope includes an agentId and a metadata field with version. New agents accept old versions as long as the payload schema doesn’t break. We maintain a backward‑compatibility matrix in our CI.

Q: What’s the biggest mistake teams make with agent-to-agent architecture?
Over‑engineering. They try to build a complete multi‑agent AI “brain” before getting the basic communication reliable. Start with two agents and a synchronous hub. Get that stable. Then add async, streaming, and more agents.

Q: Do I need a message queue like Kafka for agent-to-agent?
Not immediately. For most production systems under 1,000 requests/second, HTTP long‑polling with A2A is fine. Kafka adds complexity. We added it only when we needed replay and exactly‑once delivery for audit logs.

Q: How do I debug a stuck agent conversation?
Use correlation IDs and a tracing tool. If you see a message in the Redis state store with no response after 5 minutes, the orchestrator should log a warning. We built a dashboard that shows all in‑flight messages — green for active, yellow for retrying, red for dead.

Q: Is A2A production‑ready in July 2026?
Yes. The protocol is stable. Google, Microsoft, and AWS support it. The arxiv survey lists 12 production deployments including ours. It’s not perfect (streaming could be better), but it’s the best option today.

Key Takeaways for Engineering Teams

Pick A2A for agent-to-agent messages. Don’t roll your own protocol. The community has already solved serialization, idempotency, and error codes. The cost of adoption is lower than the cost of debugging a custom protocol.
Invest in observability from day one. Use correlation IDs and distributed tracing. Without it, you’re debugging blind.
Design for failure, not success. Timeouts, retries, rate limits, dead letter queues — build them before you deploy. You will use them.
Hub-and-spoke beats mesh for most systems. A central orchestrator makes routing, state, and monitoring manageable. Run multiple instances for fault tolerance.
Test under load early. Write integration tests that simulate network failures. Chaos engineer your agents. If they survive a dropped packet, they’ll survive production.

The industry is still figuring out multi-agent architectures. The frameworks and protocols are maturing fast. But the fundamentals haven’t changed: reliable communication, transparent failure, and simple contracts beat clever hacks every time.

I’ve seen teams spend six months building a beautiful agent ecosystem that collapsed under its own complexity. I’ve also seen teams ship a two‑agent system in two weeks that ran for a year without incident. The difference was discipline, not intelligence.

Build the simplest agent‑to‑agent architecture that could possibly work. Then make it resilient. That’s the production playbook.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.