SIVARO
AI Agent Orchestration with AWS: The 2026 Buyer's Guide
The Orchestration Trap
Last quarter, a fintech client in Singapore called me with a familiar problem. They'd built five AI agents — one for KYC, one for fraud scoring, one for customer onboarding, one for document verification, and one for compliance reporting. Each worked beautifully in isolation. Together, they were a disaster. Agents were calling each other in infinite loops, hitting rate limits on shared services, and burning through $40K/month in Lambda invocations because nobody had thought about state persistence.
The problem wasn't the agents. It was the orchestration layer.
And honestly? This is the same story I've seen at eleven companies since 2024. Everyone builds the agents first. Nobody builds the coordination layer until it's too late.
So let's talk about ai agent orchestration with aws best practices — what actually works in production, what's just vendor marketing, and how you should think about buying or building your orchestration stack in 2026.
What AI Agent Orchestration Actually Means
Here's the definition I use with my teams: orchestration is the layer that decides which agent does what, when, and how they share context. It's not the agents themselves. It's the traffic controller, the memory system, and the failure handler rolled into one.
The key insight? Agent orchestration is fundamentally a ai agents distributed systems architecture explained problem. Each agent is a node. The orchestration layer is the distributed system that keeps those nodes coherent. Once you see it that way, everything from the AWS docs starts making sense.
The Four Orchestration Patterns That Work
I've tested every major pattern over the past two years. Here's my honest assessment:
Pattern 1: Sequential Pipeline
Agent A finishes, hands output to Agent B, then C. Simple, predictable, and you can trace exactly where things break.
We used this for a logistics client's shipment exception handling. Label extraction → route analysis → customer notification. Each step had a clear input schema and output schema.
Works great when steps have hard dependencies. Fails when agents need to backtrack or when two tasks can run in parallel.
Pattern 2: Router Pattern
A central router agent examines the request and decides which specialized agent handles it.
This is what Anthropic's Claude 3.5 Opus used for GitHub Copilot in 2025. We built something similar for an insurance claims processor last spring.
The hard part is the router itself. It becomes the bottleneck, the single point of failure, and the most complex piece of code you'll maintain.
Pattern 3: Supervisor / Worker
A supervisor agent delegates tasks to worker agents, collects results, and makes decisions about next steps.
This is the pattern AWS's Multi-Agent Orchestrator framework supports natively. It's also the closest to how I'd design a ai agents distributed systems architecture for anything beyond twenty agents.
Pattern 4: Dynamic Graph
The most flexible pattern. Agents can call other agents, spawn new ones, and the execution path isn't known ahead of time.
This is where things get dangerous. In 2025, I watched a startup burn through $200K in compute credits because their dynamic graph generated infinite recursion loops. No cycle detection. No budget caps.
AWS Services Comparison: The Current Stack
Here's the honest rundown of what's actually available on AWS for agent orchestration as of August 2026. I've used all of these in client work.
Amazon Bedrock Agents
Bedrock agents is AWS's native agent runtime. It handles the prompt loop, tool calling, and memory for individual agents. For orchestration, you get the Multi-Agent Orchestrator — released GA in October 2025.
What works: tight integration with Bedrock Foundation models, built-in memory, and the orchestration SDK supports the supervisor/worker pattern out of the box.
What doesn't: the orchestration logic is Python-only right now. If your team is TypeScript-first, you're maintaining a parallel codebase. And debugging distributed agent invocations in CloudWatch is still miserable.
AWS Step Functions
Step Functions is the distributed state machine service that's been around since 2016. It's not AI-specific, but it's the backbone for most serious orchestration.
Why it matters: durable execution. If a Lambda fails mid-flight, Step Functions retries with same-state guarantees. For agent orchestration, that means your agents don't lose their memory when something crashes.
We built a claims processing pipeline with Step Functions and Bedrock Agents in March. The Step Functions state machine handles the routing, retries, and timeouts. The Bedrock agent handles the actual reasoning and tool calls.
json
{
"Comment": "Agent Orchestration State Machine",
"StartAt": "RouteRequest",
"States": {
"RouteRequest": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:router",
"Next": "InvokeAgent",
"Retry": [
{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2.0
}
],
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"Next": "FallbackHandler"
}
]
},
"InvokeAgent": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeAgent",
"Parameters": {
"AgentId": "${agentId}",
"AgentAliasId": "${agentAlias}",
"SessionState": {
"promptSessionAttributes": {
"userId.$": "$.userId",
"context.$": "$.context"
}
}
},
"End": true
},
"FallbackHandler": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:fallback",
"Payload": {
"input.$": "$"
}
},
"End": true
}
}
}
The tradeoff: Step Functions has a steep learning curve for developers who've never worked with state machines. And for simple two-agent workflows, it's overkill.
Amazon EKS + Ray
For serious scale — think hundreds of agents running concurrently — Ray on EKS is what I'd pick.
Ray's actor model maps beautifully to agent orchestration. Each agent is a Ray actor. The Ray distributed scheduler handles placement and fault tolerance. Ray Serve gives you HTTP endpoints for each agent.
We used this for a real-time trading analysis system in June. Twenty-seven agents processing market data streams. Each agent was a Ray actor with its own state. The orchestration layer handled fan-out/fan-in with zero dropped messages.
python
from ray import serve
import ray
@ray.remote
class SentimentAgent:
def __init__(self, agent_id: str):
self.agent_id = agent_id
self.context_window = []
async def process(self, message: dict):
# Agent-specific reasoning logic
self.context_window.append(message)
# Truncate to last 20 messages
self.context_window = self.context_window[-20:]
return {"agent_id": self.agent_id, "result": "processed"}
@serve.deployment
class Orchestrator:
def __init__(self):
self.agents = [SentimentAgent.remote(f"agent-{i}") for i in range(27)]
async def __call__(self, request):
payload = await request.json()
# Fan-out to all agents
results = await asyncio.gather(
*[agent.process.remote(payload) for agent in self.agents]
)
# Fan-in aggregation
return {"status": "complete", "results": results}
orchestrator = Orchestrator.bind()
But here's the catch: you're now running a Kubernetes cluster. That's an operational burden that most teams underestimate. If you don't have a platform engineer who's intimately familiar with EKS, Day 2 operations will eat you alive.
Amazon SageMaker + LangGraph
LangGraph is the orchestration framework from LangChain. It's model-agnostic, supports the graph pattern natively, and has better debugging tools than most alternatives.
For SageMaker deployment, you can host the LangGraph server as a SageMaker endpoint. This gives you auto-scaling, model monitoring, and integration with SageMaker's experiment tracking.
The tradeoff: LangGraph is opinionated about how you structure your graph. If your workflow fits its patterns, it's great. If it doesn't, you're fighting the framework.
I've seen teams try to force their orchestration into LangGraph because they like the ecosystem — and end up with worse performance than if they'd written plain Python with asyncio.
Cost Comparison: What You'll Actually Pay
Here's the thing nobody tells you about agent orchestration costs: the orchestration layer is usually 15-30% of your total infrastructure spend.
For a production system handling 10,000 agent invocations per day:
| Service | Monthly Cost | Notes |
|---|---|---|
| Bedrock Agents (Orchestrator + 5 sub-agents) | $1,200-$2,500 | Includes token costs for orchestration prompts |
| Step Functions (10K state transitions/day) | $250-$400 | Cheap until you add retries |
| EKS + Ray (3-node cluster) | $1,500-$3,000 | Base cluster cost, before GPU instances |
| SageMaker + LangGraph (ml.g5.12xlarge) | $4,000-$6,000 | Highest upfront, best for heavy GPU workloads |
Look, cost-per-invocation is the wrong metric. What matters is cost per successfully completed workflow. An orchestration layer that retries 30% of the time costs more than one that gets it right the first time, even if the per-invocation price is lower.
Reference Architecture: What I Actually Deploy
Let me show you the architecture I've settled on after two years of iterating. It's not the sexiest, but it works.
Client Request
↓
[API Gateway] → [Auth Lambda]
↓
[Step Functions: OrchestrationStateMachine]
↓
[Router Lambda] → decides agent path based on intents
↓
[Bedrock Agents: Sub-agent A] ←→ [Shared Memory (ElastiCache)]
[Bedrock Agents: Sub-agent B] ←→ [Shared Memory (ElastiCache)]
[Bedrock Agents: Sub-agent C] ←→ [Shared Memory (ElastiCache)]
↓
[Aggregator Lambda] → merges outputs, checks constraints
↓
[Response] → back through API Gateway
Why Step Functions and not Bedrock's native orchestrator? Because I need the state machine's built-in retries, timeouts, and human-in-the-loop approval steps. Bedrock's orchestrator doesn't give me fine-grained control over failure recovery.
The shared memory layer is critical. We use ElastiCache for Redis to maintain conversation state and tool call results across agents. This prevents the "agent amnesia" problem where each sub-agent forgets what the others said.
python
# Shared state management with Redis
import redis.asyncio as redis
import json
class AgentStateStore:
def __init__(self, redis_url: str):
self.redis = redis.from_url(redis_url)
async def save_state(self, session_id: str, state: dict):
"""Persist agent state with TTL for cleanup."""
key = f"agent_state:{session_id}"
await self.redis.setex(key, 3600, json.dumps(state))
async def get_state(self, session_id: str) -> dict | None:
"""Retrieve agent state, returning None if expired."""
key = f"agent_state:{session_id}"
raw = await self.redis.get(key)
return json.loads(raw) if raw else None
async def append_to_context(self, session_id: str, agent_id: str, message: dict) -> None:
"""Append message to a per-agent transcript."""
key = f"transcript:{session_id}:{agent_id}"
await self.redis.rpush(key, json.dumps(message))
# Keep only last 50 messages
await self.redis.ltrim(key, -50, -1)
This is the ai agents distributed systems architecture behind the scenes. The state store is the durability layer. The Step Functions state machine is the execution layer. The Bedrock agents are the reasoning layer.
Security and Guardrails
Each agent has its own IAM role with least-privilege permissions. The orchestration layer can only invoke agents, not access their internal data stores.
But here's the security issue nobody's talking about: prompt injection through the orchestration layer.
In September 2025, I saw a client's orchestration system compromised because a malicious user embedded instructions in document text that got routed to a sub-agent. The sub-agent was following the document's embedded instructions instead of the orchestrator's.
Best practices I now enforce:
- Every orchestration message includes a
system_boundarytoken that sub-agents are trained to treat as immutable - All external input is sanitized before entering the orchestration frame
- IAM policies restrict which Bedrock models agents can invoke
- CloudTrail is enabled on all orchestration functions with alarms on anomalous invocation patterns
json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeAgent"
],
"Resource": [
"arn:aws:bedrock:us-east-1:123456789012:agent-alias/*"
],
"Condition": {
"StringEquals": {
"aws:RequestedRegion": "us-east-1"
}
}
},
{
"Effect": "Deny",
"Action": "bedrock:InvokeAgent",
"Resource": "*",
"Condition": {
"StringLike": {
"aws:SourceArn": [
"arn:aws:lambda:us-east-1:*:function:untrusted-handler"
]
}
}
}
]
}
The Tools Comparison: Ready-Built vs. Custom
I get asked constantly: "Should we use an orchestration framework or build our own?"
The honest answer: start with a framework, plan to build when you hit its limits.
We went through this with SIVARO's own platform. Started with LangGraph in 2024. Then hit performance issues with our event-driven architecture. Moved to Step Functions, then built a custom orchestrator on top of Amazon Simple Workflow Service (SWF) for our production workloads.
Each framework has a ceiling. LangGraph's is around 100 concurrent agents with complex interactions. Step Functions' is around 10,000 state transitions per minute. Once you hit those ceilings, you're either throttling or writing workarounds that are worse than what you'd build from scratch.
But building from scratch from Day 1 is suicide. You'll spend months on basic features like retries and state persistence instead of solving your actual domain problem.
Monitoring and Observability
This is where most teams fall apart.
Our agent orchestration stack generates three telemetry streams: trace data, evaluation scores, and cost metrics. We route everything to CloudWatch with structured JSON logging.
Here's a sample instrumentation:
python
import json
import time
import boto3
def trace_agent_invocation(agent_id, session_id, start_time, status, metadata=None):
"""Emit structured trace data to CloudWatch."""
cloudwatch = boto3.client('cloudwatch')
duration_ms = (time.time() - start_time) * 1000
metrics = [
{
'MetricName': 'AgentInvocationDuration',
'Value': duration_ms,
'Unit': 'Milliseconds',
'Dimensions': [
{'Name': 'AgentId', 'Value': agent_id},
{'Name': 'Status', 'Value': status}
]
}
]
cloudwatch.put_metric_data(
Namespace='SIVARO/AgentOrchestration',
MetricData=metrics
)
# Structured log for correlation
print(json.dumps({
"timestamp": time.time(),
"agent_id": agent_id,
"session_id": session_id,
"duration_ms": duration_ms,
"status": status,
"metadata": metadata or {}
}))
The critical metric to watch: successful workflow completion rate. Not latency, not token count. If your completion rate drops below 95%, your orchestration layer is broken, and no amount of model tuning will fix it.
Cost Optimization: Where Your Money Actually Goes
Let's break down the largest cost drivers in order of impact:
Model Inference Tokens
This is the biggest lever. Orchestration prompts (the system prompts that coordinate agents) consume tokens every time an agent is invoked. For a five-agent system doing three steps per workflow, that's 15 orchestrator-level calls that burn tokens.
Fix: Cache orchestrator prompts aggressively. We use SageMaker's prompt caching to reduce 40% of this token spend.
Lambda Cold Starts
Every Lambda that invokes an agent has a cold start penalty. In agent orchestration, this compounding effect can add 3-5 seconds of latency to each agent call.
Fix: Provisioned Concurrency on your hot path. It costs more, but your user experience is dramatically better.
State Storage
Redis and DynamoDB costs are usually ignorable, but if you're storing full conversation transcripts for every agent session, the cost adds up. We store 50 messages per session max, then compress older context into embeddings.
The Vendor Lock-In Question
Everyone worries about vendor lock-in with AWS for agent orchestration. Honestly? The lock-in concern is overblown, but not imaginary.
The orchestration patterns (supervisor-worker, router, graph) are framework-agnostic. If you build your orchestration as a clean abstraction layer, you can swap out Bedrock for Vertex AI or Azure OpenAI with minimal effort.
What will trap you: Bedrock-specific features like the native memory calls, the special agent invocation syntax, and integration with Amazon's knowledge base. Those are genuinely difficult to replicate elsewhere.
My recommendation: keep your orchestration logic in a separate module with a stable interface. The orchestration shouldn't directly call Bedrock's SDK — it should call an abstraction interface.
Buy vs. Build: My Honest Take
Buy if: You're in the first six months of an agent project, you have fewer than five agents, and you don't have a dedicated infrastructure team.
Build if: You're past proof of concept with more than ten agents, you require custom failure recovery, or you need fine-grained cost controls per tenant.
Hybrid if you're serious: Use Step Functions for your state machine, Bedrock Agents for individual agents, and write your orchestration logic in a thin Lambda. This is what I'd do for any production system, regardless of scale.
FAQ: Questions I Get Every Week
Q: Do I need Kubernetes for agent orchestration?
No. Kubernetes is necessary only if you're running hundreds of agents, need GPU scheduling, or require specific horizontal scaling characteristics. For most systems, Step Functions plus Lambda or Bedrock Agents will handle your load fine.
Q: What's the best way to handle agent retries?
Use Step Functions for retry logic. Don't implement retries inside your agent code — you'll end up with infinite loops. Step Functions has built-in exponential backoff, jitter, and max attempts, which is exactly what you need.
Q: How do I prevent agents from hitting rate limits?
We use a distributed semaphore pattern with ElastiCache. Each agent acquires a lease before executing. If it can't get a lease within a specified window, it waits and retries.
Q: What about state management for long-running sessions?
Use Step Functions for the orchestration state and ElastiCache for the agent-specific state. Store session data separately from agent internal state. When a workflow spans days, Step Functions keeps the orchestration state alive, and ElastiCache holds the agent transcripts.
Q: Is the AI Agent Orchestrator from AWS worth the premium over Step Functions?
It's a different tool for a different job. The Orchestrator handles agent-specific stuff — routing, tool calls, model selection. Step Functions handles workflow-level stuff — retries, timeouts, human touchpoints. You might need both.
Q: How do I handle testing with orchestration?
Testing orchestration is the hardest part. We built a mock agent framework that simulates agent responses with realistic latency. We can then test failure modes and edge cases without invoking real models. It's not perfect, but it catches 70% of issues before they reach production.
Q: What's your rule of thumb for when to split an agent vs. keep it combined?
If two tasks share 80% of their context, keep them as one agent. If they share less than 50%, split them. Between 50-80%, test and measure. We've seen teams over-split and end up with more orchestration overhead than actual reasoning work.
What Changed in 2026
I want to close with something I noticed last month at AWS Summit in New York.
The emphasis on agent orchestration shifted from tools to outcomes. People aren't asking "what's a step function?" anymore — they're asking "how do I get my agents to stop self-loops?" The questions have gotten more mature.
That said, the tooling still has rough edges. Bedrock Agents' orchestration SDK is closer to version 0.8 than 1.0. Debugging distributed agent backends in CloudWatch is legitimate pain.
But it's getting better. The trajectory is good.
And honestly? The teams that succeed with AI agent orchestration aren't the ones with the best tools. They're the ones who started with a clear ai agents distributed systems architecture explained to their team, built the state management first, and made the orchestration layer boring.
Boring is good. Boring means it works.
At SIVARO, we've helped migrate six Fortune 500 clients onto this architecture in the past year. Every single one tried to skip the orchestration layer at first. Every single one came back.
Build the orchestration layer first. Your agents will thank you.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.