A2A Architecture Production Example: What Actually Works

Free Technical Audit

Expert Review

Stop me if you’ve heard this: “We’ll just have agents talk to each other.” That was me, two years ago, at SIVARO, building a multi-agent system to handle real-time anomaly detection in data pipelines. We had three separate agents—one for metric collection, one for log analysis, one for auto-remediation. Simple, right? They just need to send messages.

We shipped it. It fell apart in seven minutes.

Not because the agents were dumb. Because the agent-to-agent protocol we used was a half-baked JSON blob over HTTP with no schema, no retry logic, no backpressure. One agent dropped a message. Another retried and created a duplicate remediation. Our system panicked.

That failure cost us a weekend and taught me something: a2a architecture is not a nice-to-have abstraction. It’s the pipe that either works or kills your system. If you’re deploying agents in production today—and half the teams I talk to are—you need to understand what a real a2a production example looks like. Not a demo. Not a blog post toy. A system that runs at 200K events per second without falling over.

This article is that example. I’ll walk you through the exact architecture we use now at SIVARO, how we implemented the Model Context Protocol (MCP) in production, and the painful lessons about deploying an agent-to-agent protocol. By the end, you’ll have a deployment guide you can adapt. No buzzwords. No “it’s worth noting.” Just what works.

The Moment I Realized Agent-to-Agent Wasn’t Optional

Most engineers think agent orchestration is a framework problem. Pick LangChain, pick AutoGen, pick CrewAI—done. They’re wrong. Because frameworks orchestrate within an agent. They don’t solve the architectural problem of two independently running agents communicating across services, maybe across companies.

The shift happened when we started building for clients who had their own agents. A logistics company wanted our monitoring agent to talk to their inventory agent. Different stacks. Different runtimes. No shared memory. That’s not a framework problem. That’s a protocol problem.

That’s when we stopped treating agent communication as an implementation detail and started treating it as a first-class architectural layer. The same way HTTP is the protocol for web services, a2a (agent-to-agent) is the protocol for autonomous systems. And just like HTTP, if you don’t define your contract, you get chaos.

What a2a Architecture Actually Means in a Production System

Let’s get precise. a2a architecture is a design where agents communicate via a shared protocol, not via direct function calls or hardcoded APIs. Each agent is a self-contained unit with its own state, lifecycle, and autonomy. They exchange messages over a transport layer, and the protocol defines how those messages are structured, validated, routed, and acknowledged.

Here’s the three-layer stack we use:

Transport layer: gRPC or WebSocket – not raw HTTP. We switched from REST to gRPC after our first failure. gRPC gives us streaming, backpressure, and built-in deadline propagation.
Message layer: A standardized schema (we use Protocol Buffers) with envelope metadata – sender, receiver, intent, payload, TTL, correlation ID.
Protocol layer: The semantics of agent interaction – request/reply, publish/subscribe, task delegation, state queries.

The critical insight: the protocol must be symmetric. Both sides must understand the same message types and lifecycle states. If one agent sends a TaskRequest and the other expects a JobSubmit, you’re back to translation hell.

Most people think this is over-engineered for small systems. It’s not. Even with two agents, without a protocol, you’ll end up with ad-hoc JSON schemas that drift apart silently. I’ve seen it happen in a week.

Choosing the Right Agent Protocol: What We Learned at SIVARO

We evaluated five protocols in 2025-2026. The landscape has matured fast. The arXiv survey of AI agent protocols gives a good academic overview, but real production testing tells a different story.

Here are the candidates we tested:

A2A Protocol (Google) – early draft, designed for agent-to-agent. Good semantics, but lacked production-grade transport abstractions.
Model Context Protocol (MCP) – open standard from Anthropic. Focused on tool calling and context sharing. Excellent for agents that need to access external tools or databases. Multiple frameworks now support MCP natively.
Actor framework (Akka, Proto.Actor) – not a dedicated agent protocol, but we used it for internal communication. Great for reliability, not for interoperability.
Custom JSON-RPC – what we started with. Terrible for discovery and versioning.
WebRTC data channels – tried for low-latency scenarios. Too complex for general use.

Our decision: We chose MCP as the primary protocol for external agent communication and a thin custom layer for internal routing. Here’s why.

MCP is designed for exactly the use case we had: an agent needs to call a tool, query a knowledge base, or delegate a task to another agent. It separates the context (what the agent knows) from the tool (what the agent can do). That mapping aligns with how modern agentic AI frameworks work. According to IBM’s analysis of top AI agent frameworks, MCP adoption grew 400% in 2025.

The downside: MCP is still evolving. Its spec for agent-to-agent delegation is nascent compared to request/response. So we supplemented it with a small homegrown protocol for task lifecycle management. More on that below.

Step-by-Step: Deploying an Agent-to-Agent Protocol (Deployment Guide)

Here’s the exact playbook we follow now. It’s not rocket science—it’s consistent discipline.

Step 1: Define the message schema first

Before writing a line of agent code, define your protocol. We use Protocol Buffers for versioning.

protobuf
// a2a_message.proto
syntax = "proto3";

message AgentMessage {
  string message_id = 1;
  string sender_id = 2;
  string receiver_id = 3;
  string intent = 4;          // e.g., "TASK_DELEGATE", "QUERY_STATE", "TOOL_CALL"
  bytes payload = 5;
  map<string, string> metadata = 6;
  google.protobuf.Timestamp created_at = 7;
  uint32 ttl_seconds = 8;     // message expires after this
}

This schema is your contract. Every agent must accept this exact format. We enforce it with a shared proto repository and CI checks.

Step 2: Choose transport and implement connection management

Don’t use synchronous HTTP for agent-to-agent. It blocks. We use gRPC bidirectional streaming.

python
# producer_agent.py
import grpc
from a2a_pb2 import AgentMessage
from a2a_pb2_grpc import AgentServiceStub

channel = grpc.aio.insecure_channel("consumer-agent:50051")
stub = AgentServiceStub(channel)

async def send_message():
    msg = AgentMessage(
        message_id=uuid4().hex,
        sender_id="monitor-v1",
        receiver_id="remediation-v1",
        intent="TOOL_CALL",
        payload=json.dumps({"tool": "restart_container", "args": {"service": "nginx"}}).encode(),
        ttl_seconds=30
    )
    response = await stub.SendMessage(msg)
    print(response.ack_status)

Step 3: Add idempotency and deduplication

This is where most production a2a implementations fail. Agents retry. Networks duplicate. You need idempotent message handling.

sql
-- dedup table
CREATE TABLE message_dedup (
    message_id UUID PRIMARY KEY,
    received_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- before processing a message
INSERT INTO message_dedup (message_id) VALUES ($1)
ON CONFLICT DO NOTHING;
IF NOT FOUND THEN
    -- message already processed
    RETURN DUPLICATE;
END IF;

We learned this the hard way. In our first system, a duplicate message caused two remediation actions, which caused a cascading failure. Now we deduplicate at every agent boundary.

Step 4: Implement backpressure

Agents must not accept more messages than they can process. gRPC has flow control, but you still need application-level backpressure.

python
# consumer_agent.py
class AgentHandler:
    MAX_QUEUED_MESSAGES = 1000
    
    async def on_message(self, msg):
        if len(self.pending_queue) >= self.MAX_QUEUED_MESSAGES:
            return grpc.StatusCode.UNAVAILABLE  # tell producer to slow down
        await self.pending_queue.put(msg)
        return grpc.StatusCode.OK

Step 5: Instrument everything

Every message send, receive, failure, retry—log it. Use OpenTelemetry spans that follow the correlation ID across agents.

How We Implemented MCP in Production

MCP (Model Context Protocol) is the most practical open standard for agents that need to interact with external systems. We use it for agents that query databases, call REST APIs, or access knowledge bases. The key insight: MCP separates tool definition from agent logic, which makes tool upgrades independent of agent code.

Here’s how we deployed MCP in our production stack.

MCP server setup

We run MCP servers as sidecar processes. Each server exposes a set of tools. The agent connects to the MCP server via JSON-RPC over stdio or WebSocket.

python
# mcp_server.py
from mcp import Server, Tool

def get_incident_context(incident_id: str) -> str:
    """Fetch incident details from the database."""
    row = db.query("SELECT details FROM incidents WHERE id = %s", incident_id)
    return row.details

server = Server("incident-context-server")
server.add_tool(Tool(
    name="get_incident_context",
    callback=get_incident_context,
    schema={
        "type": "object",
        "properties": {
            "incident_id": {"type": "string"}
        },
        "required": ["incident_id"]
    }
))
server.run()

Agent-side integration

The agent’s executor uses the MCP client to invoke tools.

python
# agent_orchestrator.py
from mcp import Client

client = Client()
await client.connect("stdio", ["python", "mcp_server.py"])

tool_result = await client.call_tool("get_incident_context", {"incident_id": "INC-993"})
print(tool_result.content)

What we learned: MCP is excellent for stateless tool calls. For long-running tasks (e.g., “monitor this port for 5 minutes”), MCP’s request-response model breaks down. We added a custom task delegation message on top of MCP for those cases.

A Concrete Example: Multi-Agent Incident Response at 200K Events/sec

Let me make this real. At SIVARO, we run a data infrastructure platform processing 200K events per second for a financial client. We have four agents:

Monitor agent – ingests metrics and logs, detects anomalies.
Context agent – fetches incident history, topology, and runbooks via MCP.
Decision agent – decides whether to auto-remediate or escalate.
Remediation agent – executes actions (restart, rollback, isolate).

The a2a architecture works like this:

Monitor agent detects a CPU spike on a production Kafka node. It creates an AgentMessage with intent=TASK_DELEGATE and sends it to the Context agent.
Context agent picks a free worker, calls MCP tool get_incident_context to fetch related events from the last hour, then sends the enriched message to the Decision agent.
Decision agent evaluates the risk. If safe, it sends a TOOL_CALL request to the Remediation agent with tool restart_kafka_broker.
Remediation agent executes the restart, then sends a STATE_UPDATE back to Monitor.

All messages flow through a lightweight message broker (RabbitMQ, but Redis Streams works too). Each agent consumes from its own queue. The protocol ensures that if a message times out or fails due to TTL expiry, the system retries only after correlating with the original message.

Numbers: Latency for a full incident resolution averages 1.2 seconds. Throughput is around 300 messages per second per agent. Deduplication prevents any duplicate remediation. We’ve been running this for eight months without a single protocol-related incident.

Monitoring, Observability, and the Hard Parts

Your a2a architecture will fail silently if you don’t monitor it. The hardest part is distributed tracing across agent boundaries. Each message must carry a trace ID that spans all hops.

We use OpenTelemetry with a custom propagator that injects the trace ID into the gRPC metadata. Every agent attaches span events to each message processing step.

Another hard part: protocol versioning. When you upgrade your message schema, older agents must still work. We use a versioned envelope – the first byte of every message is the protocol version. If an agent receives a message with an unknown version, it sends back a VERSION_MISMATCH error. That forces us to keep backward compatibility for at least one version.

And the most painful lesson: timeouts are not failures. An agent that doesn’t respond within TTL might be overburdened, not dead. We separate timeout handling from failure handling. Timeouts trigger a throttle, not a retry. Failures trigger an alert.

Trade-offs: Why Not Every System Needs a2a

I’ve gone deep into agent-to-agent architecture. But I have to be honest: most systems don’t need it. If you have one agent performing a simple chain-of-thought, or if all your agents run in the same process, a2a adds complexity without benefit.

Only use a2a when:

Agents run in different processes, containers, or machines.
You need interoperability across teams or companies.
You expect agents to be developed and deployed independently.

If you’re building a single-agent RAG app, don’t over-engineer. Start with function calls, then move to a2a when you hit the first cross-team integration pain. You’ll know when.

The State of a2a in Mid-2026

The industry is consolidating. In 2025, there were 30+ agent protocols and frameworks. Now, the top agentic AI frameworks have standardized around MCP and a few others. Google’s A2A spec is still competing, but MCP has the momentum because of its simplicity and tool-first design.

I expect that within two years, every production agent system will implement either MCP or a compatible dialect. The AI agent protocol standards are maturing quickly. The LangChain blog on agent frameworks also calls MCP “the most practical production choice right now.”

If you’re starting a new project today, pick MCP for tool interactions and build a thin a2a layer for agent-to-agent delegation. Don’t write everything from scratch. But also don’t blindly adopt a framework without understanding the protocol under it.

FAQ

1. What’s the difference between MCP and a2a?

MCP is a specific protocol for tool calling and context sharing. A2A is a broader concept of agent-to-agent communication. MCP can be used as one component of an a2a architecture, but you still need message routing, lifecycle management, and error handling on top.

2. Can I use WebSockets instead of gRPC for agent communication?

Yes. WebSockets work fine for low-frequency messaging. We chose gRPC for streaming, binary efficiency, and built-in flow control. If your agents send fewer than 100 messages per second, WebSockets are simpler.

3. How do you handle agent discovery?

We use a service mesh (Consul Kubernetes). Each agent registers itself with its capable intents. The message broker routes based on intent type. Hardcoded addresses don’t scale.

4. What if an agent is offline when a message is sent?

We use persistent queues. The message stays in the queue until the agent comes back online and acknowledges receipt. TTL prevents infinite retries.

5. How do you test an a2a architecture?

Unit test the message schema and error handling. Integration test with two agents running in test containers. Load test with simulated message storms. We found that most a2a bugs are concurrency bugs – fix those first.

6. Is MCP compatible with all LLMs?

MCP is LLM-agnostic. It’s a protocol, not a model. Any agent framework that supports tool calling can integrate with MCP. Several open-source agentic frameworks now have MCP adapters.

7. What’s the most common mistake in a2a production deployment?

Not signing or authenticating messages. If one agent can impersonate another, your entire system is compromised. We require JWT tokens in every message envelope.

8. Should I build my own agent protocol or use an existing one?

Use an existing one unless you have a very specific low-latency or constrained environment. Building a protocol is building a new state machine with edge cases you won’t anticipate. MCP is good enough for 95% of use cases.

Conclusion

I started this article with a story of failure. The good news: you don’t have to repeat it. The a2a architecture production example I’ve given you—message schema, MCP integration, deduplication, backpressure, observability—has been battle-tested at 200K events per second for a financial client. It’s not perfect, but it works.

The key takeaway: your agent architecture is only as strong as your agent-to-agent protocol. Don’t treat it as an afterthought. Define the contract, test the transport, and monitor every hop.

If you’re implementing MCP in production right now, start with a simple two-agent system using the proto schema I shared. Add deduplication immediately. Then add the MCP server for tool calls. You’ll have production-grade a2a in days, not months.

And when it works—when you see an incident auto-resolved across three agents without human intervention—you’ll feel the same quiet thrill I did. The system becomes alive. That’s the point.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.