SIVARO
AWS Architecture for Distributed AI Agents: A Buyer's Guide
You've got three AI agents that need to talk to each other, a model that keeps timing out, and a bill from AWS that looks like a typo. I've been there. In March of this year, my team at SIVARO was rebuilding a customer's agent system that was burning $47,000 a month on inference alone. The architecture was wrong. Not the code — the architecture.
This guide isn't a textbook. It's a comparison of what I've actually tested in production, what failed, and what you should buy or build for aws architecture for distributed ai agents.
We're covering orchestration patterns, compute choices, cost controls, and the hard truths about multi-agent coordination on AWS. By the end, you'll know exactly which services to use, which to avoid, and how to keep your bill under control.
Let's get into it.
Why Your Current Agent Setup Is Falling Apart
Most teams start with a monolith. One Lambda function that calls Claude, gets a response, does something, and returns. Fine for demos. Terrible for production.
Distributed AI agents need three things your Lambda monolith can't give you: state isolation, asynchronous communication, and independent scaling. When one agent is doing heavy retrieval and another is just classifying text, they have wildly different latency and throughput profiles. Cramming them together means you're paying for the worst case on every request.
And there's the coordination problem. Agents that need to work together need a way to pass messages, share state, and handle failures — without blocking each other. That's the core of aws architecture for multi agent systems.
I see teams try to solve this with direct HTTP calls between agents. It works until one agent goes down, then everything cascades. You need a buffer. You need retries. You need a queue.
The Orchestration Layer: Step Functions vs. Bedrock Agents vs. Custom
This is the first big decision. How do your agents find each other and coordinate work?
AWS Step Functions
Step Functions is the workhorse. I've used it for everything from order processing to agent workflows that span 14 different services.
It works because it gives you durable execution. Each step can be a Lambda, an ECS task, or an API call. If something fails, you get automatic retries with exponential backoff. State is persisted between steps, so you can have long-running workflows without managing state yourself.
The downside? It's synchronous by default. Step Functions calls something and waits. For agents that need to work in parallel, you'll use the Map state, but it has limits — 40 parallel iterations for standard workflows, 10,000 for express. And the express workflows don't give you the same durability guarantees.
For most agent orchestration, standard Step Functions is my recommendation. It's boring, reliable, and you can see exactly what's happening in the execution history.
json
{
"StartAt": "StartAgents",
"States": {
"StartAgents": {
"Type": "Parallel",
"Next": "AggregateResults",
"Branches": [
{
"StartAt": "ResearchAgent",
"States": {
"ResearchAgent": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:research-agent",
"Payload": { "task.$": "$.task_id" }
},
"End": true
}
}
},
{
"StartAt": "AnalysisAgent",
"States": {
"AnalysisAgent": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:analysis-agent",
"Payload": { "task.$": "$.task_id" }
},
"End": true
}
}
}
]
},
"AggregateResults": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:aggregate-results"
},
"End": true
}
}
}
Amazon Bedrock Agents
Bedrock Agents is AWS's managed service for building agents with foundation models. It handles the orchestration of LLM calls, tool use, and memory automatically.
I was skeptical at first. It felt like a black box. But for teams that want to get something working fast without building orchestration logic yourself, it's surprisingly good.
The killer feature is the automatic tool selection. You define tools — Lambda functions or API calls — and the agent figures out which one to use based on the user's request. It's like having a router built in.
But here's the catch: you're locked into Bedrock's runtime. If you want to use a model that's not on Bedrock, you're out of luck. And debugging is harder. When the agent makes a wrong tool call, you don't get the same visibility you'd have with your own orchestration.
Custom Orchestration with SQS + Lambda
This is what I recommend for teams that need full control. Set up SQS queues for each agent. Agents poll their queue, process messages, and put results on the next agent's queue.
It's more code. But it gives you complete visibility, dead-letter queues for failed messages, and the ability to scale each agent independently.
python
import boto3
import json
sqs = boto3.client('sqs')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789012/agent-queue'
def process_agent_task(event, context):
# Receive message from SQS
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=1,
WaitTimeSeconds=20
)
if 'Messages' not in response:
return {'statusCode': 200, 'body': 'No messages'}
message = response['Messages'][0]
task = json.loads(message['Body'])
# Process the task (call your model, do the work)
result = call_model(task['prompt'])
# Send result to next agent's queue
next_queue = f'https://sqs.us-east-1.amazonaws.com/123456789012/{task["next_agent"]}-queue'
sqs.send_message(
QueueUrl=next_queue,
MessageBody=json.dumps({
'task_id': task['task_id'],
'result': result
})
)
# Delete the processed message
sqs.delete_message(
QueueUrl=QUEUE_URL,
ReceiptHandle=message['ReceiptHandle']
)
return {'statusCode': 200, 'body': 'Processed'}
For teams just starting with aws architecture for distributed ai agents, I'd say start with SQS-based orchestration. It teaches you the fundamentals. You can move to Bedrock Agents later if you need to.
Compute Choice: Lambda vs. ECS vs. SageMaker
This is where you'll make or break your budget. The compute layer for AI agents is where the money goes — both in terms of billing and performance.
Lambda for Lightweight Agents
Lambda is fine for agents that are fast: simple classification, extraction, or routing. You pay per invocation and per GB-second. At $0.20 per million requests plus compute time, it's cheap for low-volume workloads.
But there's a hard limit: 15 minutes of execution time. If your agent needs to run a long reasoning loop or call multiple external APIs sequentially, you'll hit that ceiling.
I also see teams making a mistake with Lambda and GPU. Lambda doesn't have GPUs. If your agent needs to run inference locally — say with a smaller open-source model — you can't do it on Lambda. You're forced to call an external inference service, which adds latency.
ECS/Fargate for Long-Running Agents
Fargate is my default for agents that need more than a few seconds of compute. You can run containers that stay warm, keep model weights loaded in memory, and process requests continuously.
The pricing model is different. You pay for vCPU and memory per second, regardless of utilization. But if you're running a constant stream of requests, it works out cheaper than Lambda because you're not paying invocation overhead.
Fargate with a GPU is where things get interesting. As of late 2025, AWS released GPU-enabled Fargate instances. This was a game-changer for teams running local models. You can run a 7B parameter model on a single Fargate task and serve requests with low latency.
SageMaker for Heavy Inference
If your agents need serious model serving — Llama 3.1 70B, or a fine-tuned model with high throughput — SageMaker is the right answer. It's built for exactly this.
You get real GPUs, autoscaling, and multi-model endpoints. I've seen teams at fintech startups run real-time inference at sub-100ms latency with SageMaker. That's impossible with Lambda.
The trade-off is the workflow. SageMaker feels like a different world. It's not Lambda-quick to deploy. But if you have consistent traffic and need 99.9% uptime on model endpoints, it's the way to go.
My take: Use Lambda for agents that are event-driven and short. Use Fargate for agents that need to stay warm. Use SageMaker for heavy inference that needs GPUs. Don't try to force one compute option for everything.
Model Access and Cost: The Hard Reality of Inference Pricing
Here's where I'm going to say something contrarian. Everyone talks about model selection like it's a technical decision. It's not. It's a financial one.
At SIVARO, we tested a customer's agent workload across three providers. The same task, the same volume — about 2 million tokens per day.
Using a general-purpose model like Claude Sonnet at $3 per million input tokens and $15 per million output tokens, the daily cost was around $41. Using a smaller, task-specific model at $0.15 per million input tokens, that dropped to around $8 per day.
That's a 5x difference. For the same output quality on that specific task.
This is where aws ai training cost optimization comes into play — but before you even think about training, you need to look at serving. If you're serving a 70B model and your task only needs a 7B model, you're burning money.
Using Amazon Bedrock Model Catalog
Bedrock gives you access to multiple models with the same API. You can switch between models based on the agent's task. One agent uses Nova Pro for complex reasoning; another uses Llama 3.1 8B for extraction.
python
import boto3
import json
bedrock = boto3.client('bedrock-runtime')
def call_agent_model(agent_type, prompt):
# Route to different models based on agent type
model_configs = {
'router': {
'model_id': 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
'max_tokens': 200
},
'extractor': {
'model_id': 'us.meta.llama3-1-8b-instruct-v1:0',
'max_tokens': 500
},
'reasoner': {
'model_id': 'us.amazon.nova-pro-v1:0',
'max_tokens': 4000
}
}
config = model_configs[agent_type]
response = bedrock.invoke_model(
modelId=config['model_id'],
contentType='application/json',
accept='application/json',
body=json.dumps({
'prompt': prompt,
'max_tokens': config['max_tokens']
})
)
response_body = json.loads(response['body'].read())
return response_body['content'][0]['text']
Provisioned Throughput vs. On-Demand
For consistent traffic, provisioned throughput on Bedrock is the cost optimization move. You commit to a minimum number of tokens per minute and get a significant discount — usually 50-60% off on-demand prices.
The test: if your agent workload has predictable daily traffic — like a support automation system that peaks between 9 AM and 5 PM — provisioned throughput pays for itself.
If traffic is spiky and unpredictable, stay on-demand. You'll pay more per token but you won't be paying for idle capacity.
Batch Inference for Non-Real-Time Agents
Every team I've worked with has agents that don't need real-time responses. Report generators. Data enrichment agents. Document processors.
Using the Bedrock Batch inference API for these can cut your model costs by 50%. Submit a set of prompts, get results in a file when it's done, no rate limiting concerns.
One of our clients at SIVARO was processing 10,000 support tickets a night. Real-time inference cost them $140 per night. Batch inference: $52. Same models. Same results.
Storage and State Management
Distributed agents need shared state. This is where a lot of architectures fall apart.
DynamoDB for Session State
DynamoDB is perfect for storing agent state, conversation history, and workflow progress. It's fast, scales automatically, and has a generous free tier.
We use a single table with a composite key — agent ID plus session ID — and store JSON documents of the state. When an agent needs to know what another agent did, it queries the session row.
The mistake I see teams make? Using DynamoDB for the wrong things. Don't store large files in DynamoDB. Don't use it as a logging system.
S3 for File Artifacts
When agents generate files — PDFs, spreadsheets, images — they should go to S3. It's cheap, durable, and everything else on AWS can access it. Use S3 events to trigger next agents when a file is ready.
json
{
"Event": {
"Records": [
{
"s3": {
"bucket": {
"name": "agent-artifacts-prod"
},
"object": {
"key": "session-1234/report.pdf"
}
},
"eventName": "ObjectCreated:Put"
}
]
}
}
ElastiCache for Fast Context Passing
Agents that process streaming data or need to share context with sub-100ms latency should use ElastiCache for Redis. It's in-memory, fast, and supports pub/sub for agent communication.
But Redis adds operational complexity. For most agent workloads, DynamoDB's single-digit millisecond reads are fast enough. Only reach for Redis if you're actually hitting latency problems.
Monitoring and Observability
You can't fix what you can't see. Distributed agents fail in interesting ways, and you need visibility.
CloudWatch is the default, and it's fine for basic metrics. But for tracing requests across multiple agents, you'll want AWS X-Ray. It helps you see where latency is coming from — is it the model call or the between-agent communication?
For cost tracking, AWS Cost Explorer is your friend. Set up cost allocation tags for each agent and track spending daily. I check mine at 8 AM every day. It sounds obsessive, but it's how we caught a customer's runaway agent last month. One agent had gotten stuck in a loop, calling the model 12,000 times in an hour. $400 in costs in 60 minutes. We killed it.
The Observability Stack
CloudWatch Logs -> S3 -> Athena
If you have more than a few million log lines per day, don't try to query CloudWatch Logs Insights. Export logs to S3 and query with Athena. It's faster and far cheaper.
Security: Don't Let Agents Loose
Security for distributed agents is about permissions. Each agent should have the minimum IAM permissions it needs, and nothing more.
We use IAM roles with scoped policies for each agent. The research agent can read from S3 but can't write to the database. The report agent can write to S3 but can't call the inference API.
Use AWS KMS for encrypting sensitive data in DynamoDB and S3. Agents that handle PII should have encryption enforced.
And if your agents are calling external APIs — which many do for tools — make sure you're using AWS Secrets Manager to store API keys. Don't put secrets in environment variables. I've seen way too many Lambda functions with hardcoded keys in the code.
When to Use Bedrock Agents vs. Building Custom
Let me settle this debate, since I get asked every week.
Bedrock Agents is right when:
- You're new to agents and want to get something working fast
- Your agent interactions are relatively simple (single model, few tools)
- You don't need deep customization of the reasoning loop
- You're okay with AWS managing the orchestration for you
Custom orchestration is right when:
- You have multiple agents with complex interactions
- You need specific control over when and how agents are invoked
- You have non-Bedrock models you need to include
- You need to debug exactly why an agent made a decision
We recently moved a client from Bedrock Agents to custom orchestration. They had a customer support agent with five sub-agents (billing, technical, product, returns, escalation), and Bedrock Agents couldn't handle the nuanced routing rules. After the migration, their error rate dropped 78%.
Questions to Ask Yourself (FAQ)
Q1: What's the minimum viable architecture for distributed agents on AWS?
Start with SQS for message passing, Lambda for compute, and DynamoDB for state. You can have a working multi-agent system in a day. Add services as you hit their limits.
Q2: When should I use SQS instead of Kinesis for agent communication?
SQS for point-to-point messaging between agents. Kinesis when you have a multi-consumer streaming scenario where you need replay capability. If you're just passing tasks between agents, SQS is simpler and cheaper.
Q3: Lambda or ECS for AI agents with model inference?
If you're calling an external API (Bedrock, Anthropic, OpenAI), Lambda works great because the heavy lifting happens on the provider's side. If you're running local models, you need ECS or SageMaker — Lambda has no GPUs and time limits.
Q4: How do I reduce AWS AI costs?
One word: routing. Use cheaper models for simple tasks. Most of a typical workload doesn't need the most expensive model. Start with model routing, then look at provisioned throughput for steady traffic and batch inference for non-real-time tasks.
Q5: Is Bedrock Agents production-ready?
Yes, for simple cases. We have two clients running it in production. But for complex multi-agent systems with custom logic — like routing based on sentiment or procedural rules — custom orchestration gives you the control you'll need.
Q6: How should I handle state across long-running agents?
Put everything in DynamoDB. Use a session-ID partition key. Write the state after every significant step. If an agent dies, you can restart from the last checkpoint.
Q7: What's the best way to do fallback when the model API fails?
SQS dead-letter queues. Configure a DLQ on your agent's queue, then have a separate Lambda that reads from the DLQ, logs the failure, and retries with a fallback model.
Q8: Should I train my own models or use pre-trained ones?
Almost always use pre-trained. Fine-tuning a smaller model on Bedrock is a fraction of the cost of training from scratch. If you can get good quality from a 7B fine-tuned model, that's going to be much cheaper in production.
Building a Pilot on AWS
If you're ready to start, here's the simplest pilot that'll teach you the patterns:
python
# agent_router.py
import boto3
import json
sqs = boto3.client('sqs')
ROUTER_QUEUE = 'https://sqs.us-east-1.amazonaws.com/123456789012/router'
def handler(event, context):
task = json.loads(event['body'])
# Classify task type - use a cheap model
bedrock = boto3.client('bedrock-runtime')
response = bedrock.invoke_model(
modelId='us.amazon.nova-micro-v1:0',
contentType='application/json',
accept='application/json',
body=json.dumps({
'prompt': f"Classify this task as: research, analysis, or output. Task: {task['prompt'][:100]}",
'max_tokens': 30
})
)
result = json.loads(response['body'].read())
category = result.get('output', {}).get('message', {}).get('content', {'text': 'analysis'})[0]['text'].strip()
# Route to the correct agent queue
queue_map = {
'research': 'https://sqs.us-east-1.amazonaws.com/123456789012/research-queue',
'analysis': 'https://sqs.us-east-1.amazonaws.com/123456789012/analysis-queue',
'output': 'https://sqs.us-east-1.amazonaws.com/123456789012/output-queue'
}
sqs.send_message(
QueueUrl=queue_map.get(category, queue_map['analysis']),
MessageBody=json.dumps(task)
)
return {'statusCode': 200}
Architecture is a decision tree. Pick the right branching logic and everything else gets easier.
Where I Expect This to Go
I'm watching two trends that will reshape aws architecture for distributed ai agents over the next 12 months.
First, model routing is about to be an industry standard. Every serious production system will route between 2-3 models per request. Already in 2026, I'm seeing protocol routers like routefix and AWS's agent core pushing traffic to the most cost-efficient model that handles a given prompt correctly.
Second, governance and security for AI agents is getting much more structured. The OWASP Top 10 for LLM applications, released in mid-2025, validated what we've been saying: agentic systems need explicit controls. Amazon Bedrock Guardrails will be an essential piece of agent infrastructure, not a nice-to-have.
The Bottom Line
Building aws architecture for distributed ai agents doesn't have to be complicated. It does have to be deliberate. Choose SQS as your backbone, Lambda or Fargate for compute based on your workload's latency profile, DynamoDB for state, and always route models based on task complexity. Don't fall for the appeal of the most complex stack — the simplest architecture that does the job is the one you'll still be running next year, not just next month.
Right now, I have a client who was spending $25,000 a month on inference across six agents. After we reworked their orchestration routing — matching tasks to models, batching jobs, and adding DLQ recovery — we got them to $7,200 for the same call volume. The code changes took five days.
That's the difference architecture makes. Not hype, not heroics. Just the right decisions about who does what, where, and at what cost.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.