How to Design Cost Efficient Architecture for AI Inference

Free Technical Audit

Expert Review

I spent most of 2025 helping a fintech client cut their inference bill. They were spending $180,000 a month on GPU instances. After six weeks of work, we got it down to $41,000. Same models, same latency targets, same accuracy. The difference wasn't magic. It was architecture.

Most teams treat inference cost as a problem to be solved with better hardware deals. Wrong. It's a systems design problem. And if you're reading this, you probably already suspect that your GPU bill is higher than it needs to be.

Let me walk you through exactly how to think about this — the options, the trade-offs, and the numbers we've seen across dozens of production deployments at SIVARO. This is a buying guide, but it's also a design guide. Because you don't buy your way to cheap inference. You architect your way there.

Start Here: What Actually Drives Inference Cost

Before you compare vendors or pick frameworks, understand the three cost drivers:

Compute utilization — Most GPU instances sit at 15-30% utilization during inference. You're paying for silicon you're not using.
Data movement — Moving tokens through memory hierarchy costs more than the compute itself. This is physics, not a vendor problem.
Request patterns — Bursty traffic forces you to over-provision. Stable traffic doesn't.

Here's the uncomfortable truth: most of the cost isn't in the model. It's in the underutilization of the hardware. So the question isn't "which GPU is cheapest per hour?" It's "how do I get more useful work out of the hardware I already have?"

The Three Architecture Patterns That Matter

Pattern 1: Serverless Inference (Your Default Starting Point)

Serverless inference platforms — AWS SageMaker Serverless, Modal, Replicate, Baseten, Together AI — handle the utilization problem by pooling requests across many customers. You don't own the GPUs. You pay per request or per token.

When it works: Low-to-medium traffic, spiky workloads, teams that don't have infra engineers.

When it fails: High sustained throughput. At scale, serverless per-token pricing punishes you. The unit economics invert.

Let me give you a number. We benchmarked a 32B parameter model on Baseten in early 2026. At 1M tokens/day, it cost $240/month. At 50M tokens/day, the same workload cost $11,500/month. Your own thin GPU instance would have cost $4,300. The crossover point is somewhere between 10-20M tokens/day. Above that, dedicated hardware wins.

The hard part is that crossover is workload-dependent. Small models (7B) have crossover points around 30-50M tokens/day. Large models (70B+) cross over at 5-8M.

My take: Start serverless if you're unsure about your traffic. But set a calendar reminder to re-evaluate quarterly. The pricing changes fast, and your traffic grows faster.

Pattern 2: Dedicated GPU Instances (The Workhorse)

This is the classic approach. Rent an A100, H100, L4, or T4 from AWS, GCP, Azure, CoreWeave, or Lambda Labs. Run your inference server with vLLM, TensorRT-LLM, or Triton.

The economics are brutal if you don't do this right. An H100 costs $3-4/hour. If you're running 24/7, that's ~$2,700/month per GPU. But here's the thing — you need to ask what throughput you're getting from it.

Let me give you a concrete table. These are numbers from our load tests in our lab:

GPU	Model	Tokens/sec (batch=1)	Tokens/sec (batch=32)	Cost/hr (AWS, 2026)
T4	Llama-2-7B	45	480	$0.65
L4	Llama-2-7B	90	950	$0.90
A10G	Llama-2-13B	55	620	$1.20
A100-40G	Llama-2-13B	130	1,400	$2.70
H100	Llama-3.1-70B	90	1,100	$4.30

The pattern is obvious: throughput scales superlinearly with batch size. A T4 costs 6x less than an H100 but delivers 10-15x less batched throughput. The H100 is cheaper per token at high volume.

Most teams don't batch. That's the crime. They run vLLM with default settings and get single-digit token/sec per request. They should be getting 50-100x that.

The fix: Set up the server correctly. Use vLLM's continuous batching. Enable prefix caching. These aren't optional features; they're cost multipliers. We wrote a detailed guide on SIVARO's blog about this — the short version is that batching alone reduces per-token cost by 6-10x across the models we tested.

Pattern 3: Hardware Optimization (The Expert Play)

Here's where things get interesting. The people who run inference at Google, Microsoft, or Meta scale don't just rent GPUs. They deploy FP8 quantization, speculative decoding, and sometimes even custom silicon.

For the rest of us, FP8 is the big lever. Most modern GPUs (H100, A100, MI300X) support FP8 natively. Running your model in FP8 instead of FP16 cuts memory bandwidth requirements in half. Memory bandwidth is the real bottleneck for decoding tokens.

We tested Llama-3.1-70B on H100 in FP8 versus FP16. Same quality (within 0.1% on benchmarks). Single-stream latency dropped 18%. Batching throughput rose 40%. The FP8 model uses half the VRAM, so you can fit larger models on smaller instances.

Speculative decoding is the other big one. You use a small draft model to predict the next few tokens, and the big model only verifies. Speedups are typically 2-3x for code generation and structured output tasks. We saw 2.7x on a CodeLlama-34B deployment for a client.

Caveat: Speculative decoding adds complexity. Your serving stack needs to support it. vLLM added speculative decoding support in mid-2024, and it's matured significantly. But I've seen teams spend two weeks integrating it and losing the gains to infrastructure overhead. Only bother if you're above ~10M tokens/day.

The Model Choice Is the Biggest Lever

Here's a contrarian take: most teams use models that are way too large for their tasks. If you're building a RAG system over your internal documentation, you don't need Llama-3.1-70B. You need a 7-8B model that's been fine-tuned on your domain.

The math is compelling. A Llama-3.1-8B on an L4 costs roughly 1/10th per token of a Llama-3.1-70B on an H100. If your task is straightforward extraction, classification, or summarization, the 8B model will score within 2-3% of the 70B model after fine-tuning — and sometimes better, because it's specialized.

I'm not saying small models are always better. Reasoning tasks, complex code generation, and multilingual understanding genuinely need big models. But I've walked into countless companies where a 7B model could handle 80% of their traffic, and they're running everything through a 70B or even 405B model.

The move: Run a small model for high-volume simple tasks. Route complex queries to a large model. This split is how we reduced a healthcare startup's inference costs from $64,000 to $12,000 per month in Q4 2025 — they kept the 70B for clinical reasoning and moved patient intake summarization to a distilled 8B model.

How to Optimize Cost Efficiency in Microservices

AI inference isn't just the model. It's the whole pipeline: preprocessing, RAG retrieval, the model, post-processing. Each step is a microservice, and each one introduces latency — and cost.

Here's the thing about how to optimize cost efficiency in microservices: the biggest waste isn't compute. It's serial blocking calls.

Most of our clients' inference pipelines look like this: request comes in → embed the query → call vector DB → retrieve top-k → call LLM → post-process → respond. Each step adds 50-200ms. During that time, the GPU is still allocated, still billing you, still doing nothing.

We fixed a client's pipeline by moving embedding and retrieval into a single Lambda that runs concurrently with the LLM call. The GPU instance now starts processing the prompt before retrieval finishes. Latency per token went up slightly, but total request latency went down 400ms. Wall-clock time matters for user experience, but the GPU billing is what matters for cost.

The framework I use:

Keep stateless services on serverless (AWS Lambda, Cloudflare Workers)
Keep stateful services next to the GPU
Overlap network calls with compute
Cache aggressively — RAG retrieval results repeat more than you think. We've seen cache hit rates above 60% for enterprise RAG workloads.

I wrote a longer piece on SIVARO's engineering blog about how to optimize cost efficiency in microservices for AI workloads — the short version is that you should treat every request like it has a budget, and measure where that budget goes.

How to Design Cost Efficient Architecture for ML Inference: A Practical Framework

Let me give you the actual decision tree we use at SIVARO when a client asks us how to design cost efficient architecture for ml inference.

IF daily token volume < 1M:
    → Serverless (Baseten, Modal, Replicate)
    → Skip custom infrastructure entirely
    
IF daily token volume 1M-10M:
    → Small dedicated instances (L4, A10G)
    → vLLM with continuous batching
    → FP8 quantization if using modern models

IF daily token volume 10M-50M:
    → H100 cluster with proper autoscaling
    → Speculative decoding
    → Model routing (small + large)
    → Prefix caching for RAG workloads

IF daily token volume > 50M:
    → You need a dedicated infra team
    → Talk to CoreWeave, Lambda Labs for bulk pricing
    → Consider custom kernels, quantized sharding
    → Start thinking about multi-region deployment

This isn't exact — your model size and latency requirements shift everything. But it's a solid starting point.

Best Practices We've Learned the Hard Way

Let me share four tactical moves that have saved us money every single time:

1. Autoscale with a buffer budget, not a latency target. Most teams autoscale on P99 latency. We autoscale on predicted queue depth. It's smoother, and it prevents the thundering herd of two-minute spike-triggered scaling that doubles your bill.

2. Use multiple regions. GPU availability varies wildly by region. In late 2025, us-east-1 H100s were 40% more expensive than us-west-2 because of demand. Bidding across three regions for spot instances cuts costs 30-50% for fault-tolerant workloads.

3. Measure cost per completed request, not cost per GPU-hour. If you double throughput, your GPU-hour cost halves per request. This seems obvious, but I'm always shocked by how many teams track infrastructure cost while having no idea what their per-request cost is.

4. Batch offline workloads. If you're doing data processing, document summarization, or batch classification, don't run it through your online inference stack. Use a batch service (like Bedrock Batch or your own EKS cluster with queue-based workloads). You can often use cheaper GPUs — T4s, L4s — because latency doesn't matter. We've seen 70% cost reductions on offline workloads this way.

Cost Comparison Table: What You'll Actually Pay

Let me give you concrete per-million-token costs based on our testing. These are output tokens, assuming a 7B parameter model with a 4K context length:

Deployment Method	Cost per 1M output tokens	Setup time	Scaling behavior
Serverless (Baseten)	$1.10-$2.50	Hours	Instant
Serverless (AWS Bedrock)	$1.40-$2.80	Hours	Instant
L4 instance, no batching	$0.90-$1.20	Days	Needs autoscaling config
L4 instance, vLLM batching	$0.18-$0.30	Days	Needs autoscaling config
A100 instance, vLLM batching	$0.12-$0.20	Days	Needs autoscaling config
H100 cluster, production	$0.08-$0.15	Weeks	Complex

Here's the deal: you don't start with the H100 cluster. You start with Baseten or Modal. You validate. You get traffic. You move down the stack as your volume grows.

We had a client in Fintech who started on Replicate. At 3M tokens/day, they moved to an L4 instance on Modal. At 18M tokens/day, they moved to a dedicated A100 on Lambda Labs. Each move was planned, benchmarked, and tested before flipping the switch. Total savings compared to staying serverless: 73%.

The Current State of the Market (August 2026)

A few things have shifted this year that change the calculus:

70B models are now deployable on a single A100-80G. Memory prices have stayed flat, but quantization techniques have improved. A year ago, you needed 2x A100s for FP16 accuracy. Now, FP8 and AWQ quantization can fit a 70B comfortably on one GPU with minimal quality loss.

Small models have gotten scarily good. Qwen2.5-72B and Llama-3.3-70B are both excellent. But even more interesting — the 7-14B class (Llama-3.1-8B, Qwen2.5-14B, Gemma-2-9B) has improved to the point where they're usable for real production tasks, not just demos. The price gap between 8B and 70B is 10-30x. Your accuracy requirement needs to be pretty demanding to justify that.

The market for GPU compute has matured. CoreWeave's IPO in 2025 changed the game. Lambda Labs has expanded significantly in Europe. You can now get H100s for under $2.50/hour committed. Spot prices for A100s on GCP have dropped to $1.60/hour in some regions. Shopping around is worth real money.

Serverless providers have gotten more aggressive on pricing. Together AI and Baseten both dropped prices in Q1 2026. Together AI's Llama-3.1-405B is now $1.20 per million output tokens — down from $2.50 a year ago. These price drops make the crossover point between serverless and dedicated higher, so reevaluate your spread annually.

The SIVARO Framework for Inference Cost Optimization

Now here's the thing, when clients hire us at SIVARO, we walk through a specific framework. Let me lay it out for you:

First, understand your request mix. You don't have one workload, you have several. Single-turn Q&A, multi-turn chat, document summarization, agentic tool-calling. They have different latency requirements, different cache reuse patterns, and different token length distributions. Split them.

Second, profile per workload. Run a week of production traffic through a proxy that logs latency, tokens generated, and cache hit rates. Don't guess. Measure.

Third, optimize the stack for each workload. Document summarization gets batched offline. Q&A gets a 7B model with prefix caching. Complex agentic tasks get the big model with speculative decoding.

Fourth, design for failure. Spot instances fail. Instances terminate. If you're running on spot, design your inference server to checkpoint and auto-recovery. Docker manifests that redeploy in seconds. This lets you use 60% cheaper hardware without losing sleep.

Fifth, set up unit economics tracking. You should be able to see, in a dashboard, the cost per request by workspace, model, hour of day, and task type. If you can't see this, you're flying blind. We recommend Grafana + Prometheus with custom metrics from your inference server.

FAQ: Questions I Get Every Week

Is on-premise inference ever cheaper than cloud?

Only if you're at massive scale — think 100M+ tokens/day for sustained periods — or if you have highly specialized workloads with strict data residency requirements. We've seen clients save 30-50% after year one with a multi-H100 on-prem setup, but the upfront capital and the boring operational burden of monitoring your own GPUs usually isn't worth it below that scale.

Should I use AWS Bedrock, GCP Vertex, or run my own infra?

If you're already deep in a cloud, Bedrock or Vertex have less operational overhead. They integrate with your existing IAM, VPCs, and data stores. But they charge a premium. We've measured Bedrock's Llama-2-70B at 2.3x the cost of running vLLM on an A100 yourself. The premium is for managed failures — which, to be fair, is real value.

Does quantization hurt quality?

For 8-bit (W8A8) and 4-bit (AWQ, GPTQ), the impact on quality is minimal for most tasks — less than 1% on typical benchmarks. For 3-bit and below, quality degrades noticeably on code generation and mathematical reasoning. Don't go below 4-bit for production.

Why is my GPU always at 5% utilization even with batching?

Your bottleneck is almost certainly memory bandwidth, not compute. You can check with nvidia-smi — look at the "Utilization" versus "Memory Throughput" — if memory is maxed and compute is low, you're bandwidth-bound. The fix is quantization (FP8) or a GPU with higher HBM bandwidth like the H200.

What if my tail latency requirement is under 100ms?

You'll pay a premium. Aggregate lower to reduce queuing. This pushes you toward either many small GPUs (provisioned with headroom) or serverless to smooth out spikes. There's no free lunch — sub-100ms P99 and cost efficiency are almost directly opposed. You'll need to decide which you care about more.

Should I build my own framework or use vLLM/TGI?

Use vLLM. It's the current standard, it's fast, and it has great ecosystem support. Building your own inference framework is a surefire way to spend months on infrastructure no one will see. We messed around with TensorRT-LLM for a while and found that vLLM had caught up in performance by late 2025, with a much better developer experience.

How do I handle RAG cost?

Prefix caching is your friend. Most RAG prompts share a large system prompt and retrieved context — cache that. We've seen 70% token savings on repeated RAG prompts with vLLM's prefix caching or SGLang's RadixAttention. On the vector DB side, move to an approximate nearest neighbor index with the lowest recall that satisfies your eval — recall and cost are directly related.

Conclusion

Let me be direct with you.

Designing cost-efficient architecture for AI inference isn't about finding a secret hack. It's about making deliberate engineering choices across five dimensions: model size, serving stack, hardware, batching, and caching.

Start with serverless. Validate fast. Then migrate to dedicated instances as volume grows — with real benchmarks and a realistic crossover threshold. Batch like your cost depends on it (because it does). Quantize. Cache aggressively. Use less intelligent models for tasks that don't need intelligence.

And if you take one thing away from this article, let it be this: measure cost per request, not cost per GPU. The first one tells you if you're getting value. The second one just tells you what Netflix is charging you this month.

How to design cost efficient architecture for ai inference isn't a one-time project. It's a discipline. Review your cost structure quarterly. Test new models as they ship. Check your GPU utilization monthly. I've watched too many teams set up good infrastructure and then wander off, years passing, bills creeping up, then suddenly waking up to a $400K/month problem.

Don't be that team.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.