Which Architecture Is Best for ML Inference

Free Technical Audit

Expert Review

I spent the last six months helping a healthcare analytics company re-platform their inference stack. They had a clear question: which architecture is best for ML inference when you need sub-100ms responses but also run overnight batch jobs on terabytes of claims data?

The answer wasn't a single architecture. It was a split-brain design. And that's almost always the real answer.

Here's what I've learned building and running production inference systems at SIVARO since 2018 — what works, what burns money, and how to decide for your specific latency and cost constraints. We'll compare serverless, dedicated GPU endpoints, batch pipelines, and edge deployment. Real numbers, not vendor marketing.

The Hardest Question: What Does "Best" Even Mean?

Let me ask you directly: What's your p99 latency requirement? Not the average. The p99.

Because if you can tolerate 2 seconds of latency, then which architecture is best for ML inference is a totally different answer than if you need 30 milliseconds. And if you're running fraud detection on credit card swipes, you need the 30ms answer. If you're generating nightly risk reports, you need the batch answer.

I've seen teams burn $40,000/month on dedicated GPU clusters for a recommendation engine that could have run on batch jobs overnight. I've also seen teams try to force a real-time fraud model through a batch pipeline and get caught with a 4-hour lag — that's how you lose money.

The first rule: latency requirements dictate architecture. Period.

Real-Time Inference: Where the Money Goes

Real-time inference is where most of the confusion lives. Everyone wants "real-time" until they see the bill. Let me break down the actual options you have.

Dedicated GPU Endpoints: When You Need Consistency

This is the classic setup. You rent an A100, H100, or L4, deploy your model behind a load balancer, and call it an endpoint.

At SIVARO, we ran production LLM serving for a legal tech client on dedicated H100s. They needed p99 latency under 800ms for document summarization. The dedicated endpoints delivered that consistently. A serverless function would have given them 25-second cold starts on GPU initialization.

python
# Pseudo-config for a dedicated endpoint
model_config = {
    "model": "our-finetuned-llama-3.1-8b",
    "deployment": {
        "gpu_type": "H100",
        "min_replicas": 2,
        "max_replicas": 6,
        "autoscaling_metric": "custom:requests_per_second",
        "target_utilization": 0.7,
        "max_batch_size": 16,  # dynamic batching is essential
    }
}

The problem? Cost. An H100 costs roughly $34/hour on-demand. Run two replicas 24/7 and that's $49,000/month before storage, networking, and the engineers to babysit it.

The rule of thumb I use: if your traffic is steady and you need consistent sub-100ms latency, dedicated endpoints win. If your traffic is spiky or seasonal, serverless architecture is best for ML inference cost optimization.

But there's a nuance most people miss: autoscaling on custom metrics. Don't scale on CPU. Scale on request queue depth or GPU utilization. We learned this the hard way when a client's CPU-based autoscaler kept spinning up replicas during a memory-bound workload. They were paying for 12 idle GPUs that were at 5% compute but 95% memory bandwidth saturation.

Serverless Inference: The Cost-Aware Choice

Serverless is the architecture that makes financial sense for spiky workloads. But it's got real limitations for ML. Specifically:

Cold starts on GPU instances are painful — sometimes 20-60 seconds
Long-running requests hit timeout walls
You can't do session-based inference or maintain connection pools easily

If you can live with that, serverless saves you enormous money during idle periods.

python
# Example: AWS Lambda + SageMaker Serverless Inference
import boto3
import json

client = boto3.client('sagemaker-runtime')

response = client.invoke_endpoint_async(
    EndpointName='our-model-serverless',
    InputLocation='s3://bucket/requests/input.json',
    InferenceId='request-12345'
)

# For sync inference, use invoke_endpoint instead
# response = client.invoke_endpoint(
#     EndpointName='our-model-serverless',
#     ContentType='application/json',
#     Body=json.dumps({"text": "summarize this deposition"})
# )

For a retail client we worked with in 2025, we moved their product recommendation model from a dedicated A10G endpoint (4 replicas, running 24/7) to AWS SageMaker Serverless Inference. Their traffic pattern was Mall of America busy on weekends and ghost-town quiet on Tuesdays.

The old setup: $28,000/month.
The serverless setup: $4,200/month.

The catch? P95 latency went from 45ms to 190ms. Cold starts on the first request of a spike took 8 seconds. Their frontend had to pre-warm the endpoint during predicted traffic peaks — an ugly but workable hack.

The lesson: serverless is best for ML inference when you accept the latency hit. If your user clicks "recommend" and doesn't need the answer for 3 seconds anyway, you're leaving money on the table by running dedicated GPU.

Streaming and Micro-Batching: The Middle Ground

Rick on my team calls this "the architecture nobody considers."

Instead of pure request-response or pure batch, you push requests into a queue, process them in micro-batches of 8-64 items, and stream responses back. Each batch gets processed on a GPU instance with optimal utilization.

We built this for a fintech fraud detection system in early 2026. They had Kafka streams of transactions arriving at variable rates — 50 per second at 3 AM, 5,000 per second during lunch rushes. Each transaction needed a decision within 250ms.

A dedicated endpoint would idle at 3 AM. Serverless would choke during lunch.

Micro-batching handled both:

python
# Pseudo-code for a micro-batching inference service
import asyncio
from collections import deque

class MicroBatcher:
    def __init__(self, model, max_batch_size=32, max_wait_ms=50):
        self.model = model
        self.queue = deque()
        self.lock = asyncio.Lock()
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms

    async def infer(self, input_data):
        """Submit a request and await its result"""
        future = asyncio.get_event_loop().create_future()
        async with self.lock:
            self.queue.append((input_data, future))
        return await future

    async def _process_batches(self):
        while True:
            # Wait for at least one item or until max_wait_ms elapsed
            await asyncio.sleep(0.01)
            async with self.lock:
                if len(self.queue) == 0:
                    continue
                batch = []
                while self.queue and len(batch) < self.max_batch_size:
                    batch.append(self.queue.popleft())
            if batch:
                inputs = [item[0] for item in batch]
                results = self.model.predict_batch(inputs)  # GPU-optimized batch inference
                for (input_data, future), result in zip(batch, results):
                    if not future.done():
                        future.set_result(result)

This kept GPU utilization above 80% even during variable traffic, and real time inference vs batch inference architecture cost analysis changed completely — the cost per prediction dropped by 11x compared to their old always-on GPU group.

The downside: engineering complexity. You're now building queue mechanisms, backpressure handling, and dead-letter logic. That's not a science project; that's real infrastructure work.

Batch Inference: The Forgotten Workhorse (Until Budget Review Season)

Here's the contrarian take: for most ML workloads, batch inference architecture is the best architecture. Most people think everything needs real-time. They're wrong.

Consider what your model actually does. Does a customer wait for the result? Or does your team look at predictions after the fact?

If you're generating product recommendations for a weekly email, you don't need real-time inference. If you're scoring credit risk on new applications and the user expects a decision in 60 seconds, batch-ish processing (micro-batch with 30-second cutoff) is perfectly fine. If you're detecting anomalies in manufacturing sensor data, you have minutes to respond — not milliseconds.

Batch processing on GPUs is dramatically more cost-efficient because you can pack your jobs during low-demand periods, use spot instances, and achieve near-100% GPU utilization.

Real numbers: pricing for H100 spot instances on AWS in September 2026 is about $1.80 per GPU-hour, versus $34 on-demand. For a batch job that runs 8 hours a day, that's a difference of $14.40/day versus $272/day. Month-after-month, that's $4,300 versus $81,600.

The Batch Code Pattern

python
# Using Ray for distributed batch inference
import ray
import pandas as pd

ray.init()

@ray.remote(num_gpus=1)
class BatchPredictor:
    def __init__(self, model_uri):
        self.model = load_model(model_uri)  # HuggingFace or equivalent
            
    def predict(self, chunk: pd.DataFrame) -> pd.DataFrame:
        # Fast enough on GPU, no need for streaming
        predictions = self.model.predict(chunk["text"].tolist())
        chunk["prediction"] = predictions
        return chunk

# Load your full dataset
all_data = pd.read_parquet("s3://bucket/raw_input.parquet")

# Split into chunks for processing
chunks = np.array_split(all_data, 100)
chunks = [ray.put(chunk) for chunk in chunks]

# Execute
predictors = [BatchPredictor.remote("s3://bucket/model.pt") for _ in range(4)]
futures = []
for i, chunk in enumerate(chunks):
    predictor = predictors[i % 4]
    futures.append(predictor.predict.remote(chunk))

results = ray.get(futures)
result_df = pd.concat(results)
result_df.to_parquet("s3://bucket/predictions.parquet")

The key insight from our work at SIVARO: batch inference isn't an architecture. It's a scheduling strategy. You still use GPU instances; you just use them efficiently. Whether you schedule those batch jobs on Kubernetes CronJobs, Ray, or Airflow depends on your orchestration stack. The cost difference comes from the ability to use spot instances and to pack more work into fewer seconds of compute.

I've said it before and I'll say it again: real time inference vs batch inference architecture cost isn't a valid comparison. The valid comparison is whether you need real-time responses at the product level. If yes, you pay the real-time tax. If no, you're subsidizing a latency requirement no user asked about.

Edge Inference: Not Dead, Just Misunderstood

Everyone got excited about edge inference in 2023. Apple's Core ML, TensorFlow Lite, ONNX Runtime on mobile. The idea: no network latency, no cloud cost, full privacy.

Then the LLM wave hit and suddenly the most interesting models — ones handling open-ended language and reasoning — can't fit on a phone. They're 70 billion parameters using 140GB of memory.

So what's the role of edge inference now?

For small models doing specific tasks, edge is the only architecture that makes sense. We built an on-device anomaly detection model for industrial equipment monitoring — 18MB model, running on a Raspberry Pi-class device at the factory floor. Cloud inference would have added latency and required unreliable industrial internet connections.

But for generative AI and even moderately complex classification, the cloud architecture wins for ML inference. The models are too large, and the infrastructure — NVIDIA's latest GPUs, InfiniBand interconnects, and tensor parallelism — lives in the datacenter.

This is going to change when smaller and more efficient model architectures mature. But as of September 2026, if you need MoE models or any model above 30B parameters, edge won't cut it.

The Decision Framework I Actually Use

Okay, we're deep in the weeds. Let's pull back.

When a client asks me which architecture is best for ML inference, I force them through four questions:

1. What's the actual latency SLA your business needs?

Not what the infrastructure team wants. What does the product need?

For search suggestions, that's under 200ms. For fraud detection, that's under 100ms. For a weekly churn prediction report, that's "next Monday morning." Being honest here saves you hundreds of thousands of dollars.

2. What's your traffic pattern?

Steady and predictable → dedicated endpoints
Spiky with cold periods → serverless
Variable with a solid baseline → micro-batching with autoscaling

3. What's your model size?

Under 1B parameters → consider edge or standard CPU instances
1B to 13B → a single L4 or A10G handles it with dynamic batching
13B to 70B → you need A100s or H100s with tensor parallelism
70B+ → you're in multi-node territory. Specialized architecture required

4. What's your data privatization requirement?

If you can't send data to a public cloud because of HIPAA or GDPR constraints, you're doing on-prem or VPC-only deployments. That changes the economics completely. A dedicated on-prem GPU cluster amortized over 3 years can be cheaper than cloud EKS with reserved instances — if you actually use it 24/7. If you only use it 10 hours a day, cloud wins.

The Hybrid Approach: What We Actually Run at SIVARO

Here's the split-brain answer I alluded to at the beginning.

We run most production systems with a hybrid design: a real-time inference path for requests that need immediate responses, and a batch pipeline for everything else.

That healthcare company I mentioned? They now route urgent diagnostic queries to a dedicated H100 instance (two replicas, autoscaled with queue-depth-based metrics). They route the nightly risk stratification to a Ray cluster on spot instances processing 14 million patient records in about 75 minutes. Cost dropped 78% while meeting all latency SLAs.

An analogy: you don't use a Porsche to deliver weekly groceries. It's not that the Porsche is bad at delivering groceries. It's simply the wrong tool for the cost. Same here.

The "and" is what you should remember. The tension between real-time and batch inference architecture cost is real, but choosing either is usually a mistake. The right architecture for ML inference design is one that:

Handles synchronous requests with minimal latency
Processes offline workloads asynchronously
Shares model artifacts and weights between the two paths
Scales the real-time path using queue depth, not CPU
Uses spot instances for the batch path to keep costs low

Quantifying Costs: A Real Example from January 2026

Let me give you a concrete pricing example so this isn't theoretical.

We have a client running a classification model (BERT-large, about 340M parameters) that processes user-submitted text from a mobile app. The usage is: 10 million requests/month, spiky (5 AM traffic is 10% of 8 PM traffic).

Architecture A: Dedicated GPU

3x A10G instances, 24/7
50% average utilization
Cost: ~$8,100/month

Architecture B: Serverless

AWS SageMaker Serverless Inference with A10G (configurable concurrency)
No cost when idle, $0.0012 per request during peak
Cold starts hidden via pre-warming during high-traffic windows
Cost: ~$3,400/month

Architecture C: Micro-Batching + Autoscaling

1x A10G baseline, scaling to 4 at peak
Request time out at 300ms, batched at 10ms granularity
Cost: ~$4,200/month
P99 latency: stable at 120ms

You might look at serverless and say it's the clear winner. But at peak traffic, the micro-batching architecture had only 9% failure rate while serverless hit 20% timeout rate. When your p99 is 300ms and your server has to load a 500MB model, you get failures.

The decision isn't purely financial. It's financial plus risk tolerance. If your inference failures mean a blank page for a user, the extra $800/month for the micro-batching isn't a cost — it's insurance.

How to Actually Run the Comparison for Your Workload

No article can tell you the definitive answer, because it depends on your model size, your request size, and your existing infrastructure. What I can give you is a methodology.

python
# Cost simulation script structure
# Assumptions:
# - Model: BERT-large (340M params, ~1.4GB in FP16)
# - You know your request rate per hour across a week

import math

gpu_costs = {
    "a10g": {"on_demand": 1.25, "spot": 0.28},
    "l4": {"on_demand": 0.80, "spot": 0.19},
    "h100": {"on_demand": 34.0, "spot": 1.80},
}

requests_per_hour = [  # from your metrics
    120, 130, 150, 200, 350, 800, 1200, 2000, 1800,
    1500, 1400, 1600, 1800, 1900, 2100, 2400, 3000,
    3500, 3000, 2200, 1800, 1500, 900, 500
]

# True hourly distribution
def dedicated_cost(days):
    # Fixed 3x A10G, 24/7
    return 3 * gpu_costs["a10g"]["on_demand"] * 24 * days

def serverless_cost(days):
    # Assume compute time is 0.02s per request on A10G
    compute_per_request_seconds = 0.02
    total_hours_compute = sum(requests_per_hour) * compute_per_request_seconds / 3600
    # Typical serverless inference pricing is per compute second with a minimum
    price_per_second = 0.00015  # example rate
    return total_hours_compute * 3600 * price_per_second * days

# Run simulation for 30 days
print(f"Dedicated GPU: ${dedicated_cost(30):,.0f}/mo")
print(f"Serverless: ${serverless_cost(30):,.0f}/mo")

This is the kind of spreadsheet modeling we do for every client before recommending an architecture. We plug in actual traffic from their monitoring dashboards, not hypotheticals.

Model Optimization Changes the Equation

Before you build a massive GPU cluster, ask another question: can you make your model smaller and faster?

Quantization, pruning, distillation — these aren't academic ideas. For an e-commerce client, we reduced an 8-billion parameter LLM to 3.5GB with GPTQ quantization and a custom kernel optimization (using vLLM for serving). The 45% performance drop wasn't acceptable, so instead we:

Used 4-bit quantization (losing 2% accuracy)
Used dynamic batching which gave us 6x throughput
Switched from a transformer to a mixture-of-experts layer for the final classification head

The result: the running architecture went from dedicated H100s to a single L4 GPU. We reduced the monthly bill from $49,000 to $2,700 for the same workload.

This decision tree matters:

Can I use ONNX Runtime or TensorRT to optimize my model?
Can I distill or prune it?
Can I cache common inference results (semantic caching)?
Can I move part of the pipeline offline?

The Future: What Changes by 2027?

Three trends that will change inference architectures over the next 18 months:

Smaller models, same capability. The work on distillation and Mixture-of-Experts keeps pushing capabilities into small parameter counts. By 2027, we'll likely see models with 10B effective parameters reaching today's 70B quality for specific tasks. That changes which architecture is best for ML inference.

Hardware specialization. NVIDIA's Hopper architecture brought FP8 and tensor cores to mainstream. The next generation of accelerators — plus Intel's Gaudi, AMD's MI300 — will give buyers more price-performance options. Locking yourself into one vendor now may be premature if your workload is batch-heavy and you can wait for the next silicon generation.

Inference-as-a-service consolidation. Instead of buying GPU instances, you'll increasingly buy "inference tokens" from providers. This is what Databricks, Modal, Baseten, and Replicate are effectively offering. The price-per-token model is the architecture.

FAQ: What Clients Ask Me

What's the cheapest architecture for low-traffic ML inference?

Let me be direct: for under 100k requests a month, you don't need GPUs. A model quantized to 4-bit running on an 8-core CPU instance (30-50ms inference time for BERT-size models) will cost under $50/month. Add a small caching layer and you're done. You don't need to overthink it.

Do I really need GPU for inference?

No. If your model is under 500M parameters and you can accept 100-300ms latency, modern CPUs (including Graviton instances) running an optimized runtime often hit comparable throughput at lower cost. We proved this with a sentiment analysis model in 2025 — the CPU endpoint handled 1,200 requests/second at 180ms p95 with zero GPU spend.

How do I handle cold starts on serverless GPU inference?

Options:

Keep one "warm" instance running always (costs ~$300/month but ensures sub-second response)
Pre-warm the serverless function 5 minutes before predicted traffic spikes (get your traffic forecasts right)
Accept a 5-20 second cold start on first request after idle — this is often fine for internal tools but not user-facing ones

When is multi-node inference necessary?

When you have models that don't fit on one GPU, period. A 70B model at FP16 uses about 140GB. An H100 has 80GB. You're splitting layers across GPUs with tensor parallelism. When you're in that mode, the architecture conversation stops mattering — you just need a GPU cluster and the orchestration to handle data parallelization.

Which architecture is best for ML inference for LLM chatbots?

For typical consumer chatbots that experience daily cyclical patterns, micro-batching plus dynamic batching with continuous request streaming wins. This is what vLLM, TensorRT-LLM, and NVIDIA Triton offer. Serverless LLM is getting better, but still doesn't handle long context windows reliably.

What's the cost difference between batch and real-time exactly?

Real numbers from a client we migrated: their batch processing cost was $0.0004 per prediction. Real-time inference costs $0.0021 per prediction — about 5x more. Both ran on the same model size and similar hardware — the difference was scheduling efficiency and spot instance usage.

Should I use my cloud provider's managed inference service or build on raw EC2?

Managed services (SageMaker, Vertex AI, Azure ML) abstract away scaling, hardware selection, and deployment. You pay 20-30% more for that.

Raw EC2 or GKE nodes give you control, better price, but push the operational burden onto you. Starting at ~50K requests per day, build on raw infrastructure. Under that, managed services win.

What I'd Do If I Were Building From Scratch Tomorrow

If you came to me in September 2026 and said, "Nishaant, I'm building a new product and it needs ML inference," here's where I'd start:

Categorize your workload. Does the model make a decision that's part of the user's interactive loop, or not?
For non-interactive workloads: Use an orchestration engine (Airflow or Dagster) plus a Ray cluster. Run on spot instances with checkpointing for fault tolerance. Expect up to 90% cost reduction.
For interactive workloads: Use a dedicated endpoint with dynamic batching and custom autoscaling metrics. Don't use serverless until your request rate is very spiky and your latency budget allows 500ms+ p99.
For both: Share model weights. Version them carefully and ensure the batch and real-time deployment paths use identical artifacts. Track model version per prediction so you can diagnose drift issues.
Monitor everything. Track GPU utilization, queue depths, p50/p95/p99 latency, and cost per 1K requests. Set thresholds to alert when cost-per-request deviates 10% from baseline.

Start small. Test with 10% of traffic. Scale once you trust your numbers.

Final Take: The Best Architecture Isn't an Architecture — It's a Decision Process

Stop Googling "which architecture is best for ML inference." Start measuring your traffic patterns and latency requirements. Build a spreadsheet comparing options with your real numbers. Run a pilot for two weeks on each candidate.

You'll likely discover that the answer isn't choosing one architecture. It's choosing a set of architectures, each solving the latency and cost constraints of a specific workload.

Real-time inference architecture costs more per prediction. Always. Batch inference costs less. Always. The entire strategy is figuring out which workloads can tolerate batch processing and which truly can't.

That's the real architecture. Everything else is just renting GPUs.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.