SIVARO
What Is the Most Cost Efficient Architecture for LLM Inference
Last November, a fintech client came to us with a $47,000 monthly OpenAI bill. They'd built a document processing pipeline the "smart" way — serverless functions calling GPT-4o for every request. Clean architecture. Zero ops. And bleeding cash.
We rebuilt it on a self-hosted Llama 3.3 70B running on two H100s. The new bill? $6,200/month. Same throughput. Better latency.
That's the thing nobody tells you when you ask what is the most cost efficient architecture for LLM inference. The answer isn't a single architecture. It's a decision tree that depends on your request volume, latency tolerance, and how much ops pain you're willing to eat.
I've built inference systems at SIVARO for the last three years. Some processed 200K events/sec. Others ran a single model for a 12-person startup. This guide is everything I've learned, boiled down into a buying decision. No vendor pitch. Just what actually works.
By the end, you'll know exactly which architecture fits your workload — and where most teams waste 70% of their budget.
The Real Cost Equation Nobody Shows You
Most teams look at cost-per-token and call it a day. That's wrong.
The real equation has five variables:
- GPU utilization — what percentage of the time your hardware is actually doing work
- Cold start overhead — how long before a request hits a warm model
- Batching efficiency — how many requests share a single forward pass
- Ops cost — the human hours and tooling to keep it running
- Egress + storage — tiny for inference, brutal for fine-tuning loops
A $0.50/M-token API looks cheap until you realize you're paying it on every call. A $3/hr GPU looks expensive until you run it 24/7 at 80% utilization and realize you're at $0.04/M tokens.
The most cost efficient architecture for LLM inference minimizes total cost of ownership per useful token, not sticker price. That distinction matters more than anything else in this article.
Serverless: When It's the Wrong Answer (And When It Isn't)
Let's get the definition straight, because what is serverless architecture vs container architecture trips up a lot of engineers.
Serverless means you don't manage servers. You upload code, the platform runs it on demand, you pay per invocation. AWS Lambda, Cloudflare Workers, Modal, RunPod Serverless, Replicate.
Container architecture means you run persistent processes on machines you (or a managed service) provision. You pay for uptime, not invocations. EKS, ECS, Kubernetes, bare metal, Fly.io.
For LLM inference, serverless has one fatal flaw: cold starts.
A Lambda cold start is 200ms. An LLM cold start is 30 to 120 seconds. You can't spin up a 70B model on demand and expect a user to wait. So "serverless LLM inference" in practice means warm serverless — the platform keeps your model loaded on a pool of GPUs and routes requests to hot instances.
Modal, RunPod, and Baseten all do this well. You get per-second billing on warm containers. Sounds perfect. It isn't.
The catch: you pay a premium of 2–4x over raw GPU rentals. Modal's H100 pricing runs around $3.95/hr as of September 2026. A raw H100 on Lambda Labs is $2.49/hr. That gap is the convenience tax.
When serverless wins:
- Spiky traffic (blogging tools, internal chatbots, prototype products)
- Under 100K requests/day
- Teams without a platform engineer
- Anything with fewer than 40 hours/week of GPU demand
When it loses:
- Steady traffic above 50% GPU utilization
- Latency-critical paths under 200ms
- Multi-model routing (serverless platforms charge per model warm)
My rule of thumb: if you're spending more than $8K/month on serverless inference, you're overpaying. Move to containers.
The Container Playbook: Self-Hosted That Doesn't Suck
Here's the contrarian take. Most teams shouldn't self-host. But the ones who should, save 60–85%.
Self-hosting used to mean racking GPUs and writing CUDA. Not anymore. You rent H100s or L40Ss from Lambda Labs, RunPod, or Vast.ai and run vLLM or SGLang. Both are mature. Both handle continuous batching, paged attention, tensor parallelism.
Here's a minimal vLLM setup:
python
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.3-70B-Instruct",
tensor_parallel_size=2, # 2 GPUs for 70B
gpu_memory_utilization=0.92,
max_model_len=8192,
enable_prefix_caching=True, # huge win for RAG
)
params = SamplingParams(temperature=0.2, max_tokens=512)
outputs = llm.generate(["Summarize this contract: ..."], params)
Prefix caching alone cut our RAG latency 40% on a legal-tech deployment last spring. If your prompts share a system message or retrieved context, this is free money.
You run that inside a Docker container. You scale with Kubernetes or a simple autoscaler script. Here's what a Kubernetes deployment looks like trimmed down:
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama70b
spec:
replicas: 2
template:
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args: ["--model", "meta-llama/Llama-3.3-70B-Instruct",
"--tensor-parallel-size", "2"]
resources:
limits:
nvidia.com/gpu: 2
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 180 # model load time
Note that readiness probe delay. 180 seconds. That's real. If you don't set it, Kubernetes restarts your pod mid-load and you spiral.
The Cost Math That Actually Decides
Let me run the numbers I ran for that fintech client.
Their workload: 4.2M tokens/day in, 1.1M out. Roughly 5.3M tokens daily. 160M/month.
Serverless (Modal H100 warm): $3.95/hr × 2 replicas × 720 hrs = $5,688/month. Plus per-token overage $0.0004/M. Total: **$6,300/month**.
Wait — that seems fine. But they were on API, not warm serverless. OpenAI GPT-4o at $2.50/M input and $10/M output: (4.2M × $2.50 + 1.1M × $10) × 30 = ~$645/day = ~$19,350/month. The $47K figure included embeddings, retries, and a fine-tuned variant.
Self-hosted (Lambda Labs, 2× H100 reserved): $2.49/hr × 2 × 720 = $3,585/month. Plus $1,200 for the ops overhead (a part-time engineer's allocation). Total: **$4,800/month**.
So: self-hosted beat warm serverless by 24% and OpenAI by 75%.
But here's the honest part — that fintech had a platform engineer already. If you don't, the ops overhead doubles or triples. At $9,600/month ops, self-hosting loses to serverless.
The tipping point: $10K/month in serverless spend, or a full-time ML platform person on payroll. Below that, rent. Above that, own.
how to reduce cloud infrastructure costs Without Breaking Things
You want the leverage points. Here they are, ranked by impact:
1. Continuous batching. If your inference server processes one request at a time, you're lighting money on fire. vLLM, TGI, and SGLang all batch by default. Verify yours does.
2. Quantization. FP8 or INT8 cuts memory in half and often doubles throughput. Quality loss is under 1% on most benchmarks. We ran Llama 3.1 405B on 4× H100s in FP8 last year — it would have needed 8 in BF16.
bash
# vLLM with FP8 quantization
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--quantization fp8 \
--tensor-parallel-size 2 \
--max-num-seqs 256
3. Spot instances for batch work. Lambda Labs spot is 50–70% off. For async jobs (nightly summarization, offline embedding), you don't need on-demand.
4. Right-size your model. Most teams use a 70B when a 8B fine-tune would do the job. Qwen 3 8B and Llama 3.3 8B are shockingly good after task-specific tuning.
5. Cache aggressively. Semantic caching with Redis or GPTCache cut our support-bot token usage 34% in Q1 2026. Same question, same answer, no GPU touch.
6. Kill idle replicas. A lot of teams run min 2 replicas 24/7 for a product that peaks 9am–6pm. Scale to zero with a 90-second cold-start budget if your SLA allows.
For a deeper operational checklist, AWS's Well-Architected cost pillar is dry but thorough.
The Hidden Costs Nobody Talks About
GPU time isn't your bill. Here's what shows up in month three:
Egress. Pulling model weights (70–140GB) on every cold start costs you. On AWS, egress is $0.09/GB. A 70B model pulled 200 times/month is $2,500. Cache weights locally or use a shared volume.
Storage. Fine-tune artifacts, checkpoints, LoRA adapters. Small per file, big in aggregate. We've seen a 400GB S3 bill from a team that never cleaned up intermediate checkpoints.
Idle GPU time. The single biggest leak. A team runs 2 replicas, traffic drops 80% at night, nobody scales down. That's 12 hours × 30 days = 360 wasted GPU-hours. On H100s, that's $900/month per replica.
Retries on failure. If your inference server OOMs and the client retries blindly, you pay twice. Fix with proper backoff and health checks.
Model licensing. Llama is free. Mistral small is free. GPT-4 is not. Per-token licensing on some models dwarfs GPU costs.
Choosing Your Architecture: The Decision Framework
Here's the tree I walk clients through.
Step 1: What's your monthly token volume?
- Under 50M → serverless. Stop reading.
- 50M–500M → warm serverless or single-node containers.
- Above 500M → self-hosted with autoscaling.
Step 2: What's your latency SLA?
- Under 300ms P99 → self-hosted, always-warm.
- 300ms–2s → warm serverless works fine.
- Above 2s → batch jobs, spot instances.
Step 3: Do you have ML platform engineering?
- Yes → self-host and save 60%+.
- No → serverless, or hire one if you're above $10K/month.
Step 4: Is your traffic predictable?
- Yes → reserved GPU capacity (Lambda Labs reserved, RunPod committed).
- No → serverless with autoscaling.
I've had teams argue steps 1 and 2 should swap. They're wrong. Volume determines whether you can amortize hardware. Latency determines how.
Open Models vs Proprietary APIs: The 2026 Reality
Two years ago, self-hosting meant a quality gap. Today, it doesn't for 80% of use cases.
Llama 3.3 70B matches GPT-4o on most reasoning benchmarks. Qwen 3 32B is competitive with GPT-4-turbo. DeepSeek V3 punches above its weight class. The LMSYS Chatbot Arena leaderboard updates continuously — check it before assuming closed models win.
What you lose self-hosting:
- Frontier reasoning (o3, Claude Opus still lead on hard math)
- Multimodality polish
- Vendor-managed safety guardrails
What you gain:
- 70–85% cost reduction at scale
- Data never leaves your VPC
- Zero rate limits
- Full control over version pinning
For regulated industries — healthcare, finance, legal — the data residency alone justifies self-hosting. We've helped three fintechs move off OpenAI purely because their compliance teams couldn't get past the data processing agreement.
A Real Deployment: What We Built Last Quarter
A healthcare RAG system. 12M tokens/day. HIPAA-bound.
Architecture:
- 2× H100 80GB on Lambda Labs reserved, us-east-1
- vLLM serving Llama 3.3 70B in FP8
- Redis semantic cache (Redis Vector)
- FastAPI gateway with request queuing
- Prometheus + Grafana for GPU metrics
Here's the gateway's core batching logic, trimmed:
python
import asyncio
from fastapi import FastAPI
from vllm import AsyncLLMEngine, SamplingParams
app = FastAPI()
engine = AsyncLLMEngine.from_engine_args(engine_args)
async def generate(prompt: str, request_id: str):
results = engine.generate(prompt, SamplingParams(max_tokens=512), request_id)
async for result in results:
if result.finished:
return result.outputs[0].text
@app.post("/generate")
async def handler(req: PromptRequest):
return {"text": await generate(req.prompt, req.id)}
Cost breakdown:
- GPUs: $3,585/month
- Redis: $140/month
- Egress/storage: $220/month
- Ops allocation: $2,400/month
- Total: $6,345/month
Previous OpenAI bill: $31,000/month. Savings: 79%.
Latency P99: 780ms. Previously 1,100ms. We got faster and cheaper. That's not typical — but it happens more than vendors want you to know.
FAQ
Why isn't serverless always cheaper?
Because per-invocation pricing hides a GPU premium. Warm serverless platforms charge 2–4x raw GPU rates. Below ~$8K/month, that premium is worth it for the convenience. Above it, you're subsidizing the platform's idle capacity.
what is serverless architecture vs container architecture for LLM workloads specifically?
Serverless runs your model on-demand in a managed pool, billing per second of warm runtime. Containers run persistent processes you scale yourself, billing for uptime. For LLMs, the key difference is cold starts: serverless can't cold-start a 70B model in under 30 seconds, so it keeps pools warm — which means you're paying for uptime anyway, just with markup.
Can I really self-host a 70B model on a single GPU?
Only quantized. A 70B in 4-bit fits in ~40GB, so an A100 80GB or H100 80GB works. Quality drops 2–5% on reasoning tasks. For most production use, FP8 with 2 GPUs is the safer bet.
How much does continuous batching actually save?
3–8x throughput depending on request concurrency. If you're not using it, you're the reason your GPU bill is high.
Do I need Kubernetes for self-hosting?
No. Two H100s and a systemd service works for many teams. Kubernetes helps past 4 nodes or if you need multi-region failover.
What about batching across users?
That's what vLLM does. Concurrent requests share forward passes. The only catch is per-request latency grows slightly as batch size increases — usually a good trade.
Is there a hybrid model?
Yes, and it's underused. Route latency-tolerant batch jobs to spot self-hosted GPUs and interactive traffic to serverless. We've seen 45% savings with this split.
How often should I re-evaluate?
Every quarter. GPU pricing moves fast, and so do open models. A decision that was right in January can be wrong by June.
The Verdict
The most cost efficient architecture for LLM inference isn't one thing. It's a sliding scale tied to your volume, latency needs, and team.
Below $8K/month: serverless. Warm, managed, don't think about it.
$8K–$30K/month: single-node containers on reserved H100s, vLLM or SGLang, one platform engineer.
Above $30K/month: multi-node K8s, autoscaling, quantized models, aggressive caching. You should be saving 70%+ versus API alternatives.
The trap most teams fall into is a middle ground — paying serverless premiums on workloads that justify containers. I've audited twelve companies this year. Eleven were overpaying. Nine could cut 60% by moving to self-hosted.
The math isn't subtle. But the decision requires honesty about your team, your traffic, and your tolerance for ops pain. If you don't have a platform engineer and can't hire one, stay serverless. Pay the tax. It's cheaper than the alternative.
For everyone else: the GPUs are waiting. Go own them.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.