HomeArticles

Chapter 11 of AI agents in production

The A2A Protocol: A Production Deployment Example

Running the A2A agent protocol in production: architecture, agent cards, task lifecycle, observability, error handling, security and a checklist.

What the A2A protocol standardizes (agent card, task lifecycle, message format) and what it leaves to you.

The Agent Transfer Protocol (a2a) isn't theoretical anymore. Google released the spec in April 2025.

No fluff. No vendor pitches. Just the decisions and the patterns that matter in production.

What a2a Actually Solves (and Doesn't)

Most people think a2a is about "agent interoperability." That's technically true but practically useless. Here's what it really does:

It gives you a standard way for one agent to say "I need this task done" and another agent to say "Here's the result" — without hard-coding the handshake logic between every pair.

Before a2a, teams built bespoke bridges. Agent A calls Agent B via REST. Agent B calls Agent C via gRPC. Agent C calls Agent D via WebSocket. Every integration was a snowflake. Every failure required tracing through three different transport layers.

a2a standardizes the agent card (discovery), the task lifecycle (creation, update, completion), and the message format. Nothing more. Nothing less.

What a2a doesn't do: handle authentication, define your data model, manage state, or guarantee delivery. You still own all of that.

The Architecture

Here's an example production architecture:

graph TD
    A[Client API Gateway] --> B[Orchestrator Agent]
    B --> C[Booking Agent]
    B --> D[Pricing Agent]
    B --> E[Compliance Agent]
    C --> F[Legacy ERP via a2a Bridge]
    D --> G[External Rate API]
    E --> H[Sanctions Database]
    B --> I[Monitoring Agent]
    I --> J[Prometheus + Grafana]
    I --> K[PagerDuty]

The orchestrator receives a booking request. It fans out tasks to three specialist agents via a2a. Each specialist reports back. The orchestrator merges results and responds.

LangGraph is one framework option (AI Agent Frameworks: Choosing the Right Foundation for ...). LangChain's blog on agent frameworks makes the point that "the framework should disappear once you're in production" (How to think about agent frameworks).

The Agent Card: Your First Production Headache

Every a2a agent publishes an agent card — a JSON document describing what it does, what inputs it expects, and what outputs it produces.

This seems trivial. It's not. Here's an example agent card for a Booking Agent:

{
  "agentId": "booking-agent-v2-1",
  "name": "Booking Agent",
  "description": "Handles ocean freight booking creation and status tracking",
  "capabilities": [
    {
      "name": "create_booking",
      "input": {
        "type": "object",
        "properties": {
          "reference": {"type": "string", "required": true},
          "containers": {"type": "array", "items": {"$ref": "#/definitions/container"}, "required": true},
          "destination_port": {"type": "string", "required": true},
          "incoterm": {"type": "string", "enum": ["FOB", "CIF", "EXW"]}
        }
      },
      "output": {
        "type": "object",
        "properties": {
          "booking_number": {"type": "string"},
          "status": {"type": "string"}
        }
      }
    }
  ],
  "endpoints": {
    "taskEndpoint": "https://booking-agent.internal.sivaro.io/a2a/task",
    "agentCardEndpoint": "https://booking-agent.internal.sivaro.io/a2a/card"
  },
  "version": "2.1.0",
  "rateLimit": 100
}

Three lessons from this:

First, version your cards. Without versions, a schema change in one agent makes the Orchestrator send malformed requests until someone notices. Add version to every card, increment on every schema change.

Second, limit the description. A paragraph of business logic is useless here. Agents don't read prose. Stick to machine-parseable constraints — enums, required fields, regex patterns.

Third, keep the card simple. The A Survey of AI Agent Protocols paper from arXiv covers 47 different protocol features. Most deployments need only a few. Don't build features you won't use.

Task Lifecycle: The Part Where Things Break

a2a defines four task states: pending, running, completed, failed. That's it. Simple, right?

Wrong.

The problem is what happens between states. Three problems show up:

Timeout cascading. The Orchestrator sends a booking to the Booking Agent. The Booking Agent calls the ERP, which takes 30 seconds. But the Orchestrator's timeout is 15 seconds. The Orchestrator retries, creating a duplicate booking.

Fix: Make timeout configurable per task type, and propagate it in the task request payload.

Idempotency is not optional. Every a2a task request must include an idempotency_key. When the network blips, the client retries and you get two BookingsAgent tasks for one user request.

Here's example retry logic:

import hashlib
import uuid
from datetime import datetime

class TaskClient:
    def __init__(self, agent_endpoint: str):
        self.endpoint = agent_endpoint
    
    def create_task(self, workflow_id: str, intent: dict):
        # Generate deterministic idempotency key from workflow context
        key_content = f"{workflow_id}:{intent['capability']}:{datetime.utcnow().hour}"
        idempotency_key = hashlib.sha256(key_content.encode()).hexdigest()[:32]
        
        payload = {
            "idempotency_key": idempotency_key,
            "capability": intent["capability"],
            "input": intent["input"],
            "timeout_seconds": 30
        }
        
        return self._post(f"{self.endpoint}/a2a/task", payload)

State synchronization. a2a is asynchronous by design. The Orchestrator sends a task, gets a task_id, and then polls for status. But what if the Orchestrator crashes between sending and polling?

One solution is a task state table in PostgreSQL. Every sent task gets a row. A background worker reconciles tasks that are pending for more than 60 seconds.

Observability: The Thing Nobody Talks About

Everyone talks about building agents. Nobody talks about watching them fail.

A busy a2a deployment generates a lot of logs. Without structured observability, you're blind. A production setup:

- All a2a messages include a trace_id header (W3C trace context)
- Every task lifecycle event logs: trace_id, agent_id, task_id, state, duration_ms
- Metrics labeled by agent_id, capability, state, error_code
- Alert when task failure rate exceeds 2% over 5 minutes
- Dashboard for end-to-end latency by workflow type

The top 10 agentic AI frameworks for 2026 list from Instaclustr mentions observability as a key differentiator. I'd go further: it's the deciding factor.

One non-negotiable: every agent must log every a2a message it sends and receives. Not just errors. Every message. You'll thank me when you're debugging a "but the data disappeared" ticket at 2 AM.

Here's logging middleware to inject into every agent:

import time
import traceback
from opentelemetry import trace

class A2ALoggingMiddleware:
    def __init__(self, logger):
        self.logger = logger
        self.tracer = trace.get_tracer(__name__)
    
    def handle_task(self, task_request):
        start = time.time()
        trace_id = trace.format_trace_id(trace.get_current_span().get_span_context().trace_id)
        
        self.logger.info({
            "event": "a2a_task_received",
            "trace_id": trace_id,
            "task_id": task_request.get("idempotency_key"),
            "capability": task_request.get("capability"),
            "timestamp": time.time()
        })
        
        try:
            result = self._process(task_request)
            duration = (time.time() - start) * 1000
            
            self.logger.info({
                "event": "a2a_task_completed",
                "trace_id": trace_id,
                "task_id": task_request.get("idempotency_key"),
                "duration_ms": duration,
                "status": "completed"
            })
            return result
            
        except Exception as e:
            duration = (time.time() - start) * 1000
            self.logger.error({
                "event": "a2a_task_failed",
                "trace_id": trace_id,
                "task_id": task_request.get("idempotency_key"),
                "duration_ms": duration,
                "error": str(e),
                "traceback": traceback.format_exc()
            })
            raise

Error Handling: The Ugly Truth

a2a's error model is minimal. You get a failed state and an error message string. That's it.

In practice, you need structured errors. Here's an error taxonomy:

{
  "errors": {
    "temporary": {
      "TIMEOUT": {"retryable": true, "message": "Agent did not respond within timeout"},
      "RATE_LIMITED": {"retryable": true, "message": "Agent is rate limited"},
      "INTERNAL_ERROR": {"retryable": true, "message": "Agent encountered internal error"},
      "TEMP_UNAVAILABLE": {"retryable": true, "message": "Agent temporarily unavailable"}
    },
    "permanent": {
      "INVALID_INPUT": {"retryable": false, "message": "Input does not match schema"},
      "UNSUPPORTED_CAPABILITY": {"retryable": false, "message": "Capability not supported"},
      "UNAUTHORIZED": {"retryable": false, "message": "Authentication failed"}
    }
  }
}

Why this matters: retrying everything burns resources on permanent errors (like invalid input) that will never succeed.

Retry on temporary only, with exponential backoff (1s, 4s, 16s). Reject immediately on permanent.

Security: The Elephant

Securing a2a with mTLS: per-agent certificates signed by an internal CA, verified before any task is sent.

a2a says nothing about authentication. It's a protocol for task exchange, not for security.

Use mTLS between agents. Every agent gets a client certificate signed by an internal CA. The Orchestrator verifies the target agent's certificate before sending any task.

Why not API keys? Because API keys leak, and a leaked shared key means rotating it for every agent. With mTLS, certificates can be revoked individually.

Here's an mTLS handshake for a2a:

import ssl
import requests

def create_agent_session():
    ssl_ctx = ssl.create_default_context()
    ssl_ctx.load_cert_chain(
        certfile="/etc/ssl/certs/agent-cert.pem",
        keyfile="/etc/ssl/private/agent-key.pem"
    )
    ssl_ctx.load_verify_locations(cafile="/etc/ssl/certs/ca-cert.pem")
    ssl_ctx.verify_mode = ssl.CERT_REQUIRED
    
    session = requests.Session()
    session.verify = "/etc/ssl/certs/ca-cert.pem"
    session.cert = ("/etc/ssl/certs/agent-cert.pem", "/etc/ssl/private/agent-key.pem")
    
    return session

Performance Under Load

Contrarian take: a2a is fast enough. The protocol overhead is noise. Your bottleneck will be your slowest downstream system, not the protocol.

Monitoring Agent Health

AI agents fail differently than microservices. A microservice either returns 200 or 500. An agent might return 200 with garbage data.

Give every agent a health-check endpoint:

GET /a2a/health

Returns:

{
  "status": "healthy",
  "uptime_seconds": 123456,
  "last_successful_task": "2026-07-19T10:23:15Z",
  "tasks_processed": 8472,
  "error_rate_1h": 0.015,
  "model_latency_p99_ms": 2500,
  "queue_depth": 3
}

A monitoring agent can ping this every 30 seconds. If error_rate_1h > 0.05 or queue_depth > 50, it pages the on-call engineer.

An agent's model can start returning low-confidence results for many requests without ever crashing. Without health metrics, that can go unnoticed for hours.

The Deployment Pipeline

Here's an ai agent deployment pipeline tutorial in practice:

One setup: GitOps with ArgoCD. Every agent is a container. The agentCard.json is part of the repository. CI runs:

  1. Unit tests (mock all a2a endpoints)
  2. Integration tests (run 10 agents locally, test full workflows)
  3. Schema validation (is the agent card valid a2a?)
  4. Load tests (simulate 500 concurrent tasks)

Deployment is canary: 10% of traffic for 15 minutes, then 50%, then 100%. If error rate spikes, rollback is automatic.

Add a shadow deployment: duplicate every production task to a staging environment, compare results, and investigate any difference. It catches schema mismatches before they hit production.

a2a Protocol Production Deployment Example: The Checklist

If you're doing this tomorrow, here's your checklist:

  1. Define agent cards first — before writing any code. Get schema agreement between teams.
  2. Implement idempotency — use deterministic keys from workflow context.
  3. Add structured errors — distinguish temporary from permanent failures.
  4. Set up observability — trace_id on every message, structured logs, latency metrics.
  5. Write health endpoints — expose model-level and queue-level metrics.
  6. Deploy with canaries — don't switch all traffic at once.
  7. Run shadow tests — duplicate production traffic to staging.
  8. Document timeouts — per-capability, not a single global timeout.
  9. Use mTLS — not API keys.
  10. Test failure modes — what happens when the Pricing Agent is down for 5 minutes?

FAQ

Q: Do I need a2a if I'm running a single agent?

No. a2a adds complexity. Use it when you have 3+ agents that need to talk to each other.

Q: a2a vs MCP (Model Context Protocol)?

MCP is for agent-to-tool communication. a2a is for agent-to-agent. You'll likely need both. A common split: MCP for tool calling, a2a for inter-agent workflows.

Q: How do you handle agent discovery at scale?

Use a service registry (Consul) with the agent card URL. The Orchestrator discovers agents by capability. Simple. Doesn't need a blockchain or a mesh.

Q: What about WebSocket vs HTTP for a2a?

HTTP for short tasks (under 30 seconds). WebSocket for long-running tasks.

Q: Can a2a handle streaming responses?

Not natively. The spec is request-response. For streaming, add a separate event stream channel that the client polls.

Q: How do you test a2a agents?

Unit tests with mock a2a servers. Integration tests with real containers. Shadow testing in production. Chaos engineering — randomly kill agents and verify recovery.

Q: What's the biggest mistake teams make?

Over-engineering. They try to build a universal agent platform with routing, caching, retry, monitoring, and governance — all before running a single agent in production. Start with one workflow. Add complexity when you have data.

Q: When should I NOT use a2a?

When your agents are tightly coupled and always co-deployed. Use gRPC direct calls. a2a's async indirection adds cost for no benefit.

What's Next

The AI Agent Protocols: 10 Modern Standards Shaping the ... article from SSONetwork predicts federated agent discovery by Q1 2027. I think they're right. The manual configuration of agent cards doesn't scale past 50 agents.

Also worth watching: the open-source agentic AI frameworks list from AI Multiple. LangGraph and CrewAI are adding a2a-native features each release. By Q4 2026, I expect framework integration to be seamless. It's not there yet.

Final Take

Running A2A in production: idempotency keys, tracing, structured retries, per-capability timeouts, canary and shadow rollout.

a2a isn't revolutionary. It's a simple protocol for a simple problem: two agents need to exchange a task.

The hard parts — idempotency, observability, security, error handling — are all outside the protocol. You build those yourself. The protocol just gives you a consistent envelope to put them in.

That's worth it. Not because a2a is magical, but because a standard lets you build better tooling around it.

A2a production deployment example isn't about the protocol. It's about everything else you need to make agents reliable. Focus there.

Want to talk this through?

Sivaro AI →