HomeArticles

Chapter 10 of LLMs in production: choosing a model, speed and cost

Why Is LLM Inference Slow? A Practitioner’s Guide to What’s Actually Going On

Why LLM inference is slow: memory bandwidth limits, attention, serving architecture, quantization, GPU use and batch size, with the order to fix them in.

Numbers behind slow LLM inference: 7 ms FP16 floor per decode step, 1.75 ms at INT4, ~2 GB KV cache for one 4K sequence

A 7B model on an A100 can still take many seconds for a short response, even with a solid pipeline and a quantized model.

The common mistake is thinking inference speed is one problem. It’s not. It’s a stack of problems, layered like an onion, and each layer has its own bottlenecks that compound into the slowdown you’re feeling.

This guide walks through why LLM inference is slow — the real reasons, not the hand-wavy explanations — and what actually works to fix it.

The Core Problem: It’s Not Just Model Size

Most people think big model = slow inference. That’s true, but it’s not the full story. The fundamental issue is that LLM inference is memory-bandwidth bound, not compute-bound for most practical scenarios.

Here’s the math:

A single forward pass of a 7B parameter model in FP16 requires loading ~14 GB of parameters from GPU memory into the compute units. An A100 has ~2 TB/s of memory bandwidth. That means the absolute fastest you can do one forward pass is:

14 GB / 2 TB/s = 7 milliseconds

That’s the floor. You can’t go below it without reducing model size or increasing bandwidth. And that’s for one token. For a 2000-token response, you’re looking at ~14 seconds of pure memory transfer — before you even compute anything.

This is why quantization works so well. Moving from FP16 (14 GB) to INT4 (3.5 GB) cuts the memory load by 4x. Suddenly that 7ms floor becomes ~1.75ms per token. (Why Your LLM Inference Is Slow does a great breakdown of this.)

But wait — if it’s just memory bandwidth, why don’t we see linear scaling with smaller models? Because there’s a second bottleneck.

The Attention Mechanism: Where Tokens Get Stuck

The transformer architecture’s self-attention mechanism scales quadratically with sequence length. For a 4096-token context, that’s ~16 million attention computations per layer. For a 128K context — which Claude and Gemini now support — it’s 16 billion.

That’s not a typo.

Here’s what happens in practice: you prompt with 10K tokens of context, and the first token takes 20 seconds because the attention computation is enormous. Then the next 1999 tokens take 10ms each because the prefill stage already did most of the heavy lifting.

This is why prompt engineering advice about “short prompts = fast responses” is actually grounded in real physics — shorter prompts mean fewer attention computations during that expensive first-token phase. (What Factors Influence Inference Speed in Machine Learning shows exactly how context length affects time-to-first-token.)

The Memory Wall That Nobody Talks About

Then there’s KV cache management.

During inference, every token’s key and value representations get cached for future attention computations. For a 7B model with 4096 context length, that’s roughly:

  • 32 layers × 4096 positions × 4096 dimensions × 2 (keys + values) × 2 bytes (FP16) = ~2 GB

For a 70B model with 128K context? That’s more like 64 GB. Just for the cache.

And the GPU needs to keep writing to this cache with every new token. If your batch size is 32, you’re now managing 64 GB × 32 = 2 TB of dynamic memory. On an 80 GB GPU. You either run out of memory or spill to CPU, which kills latency.

This is the hidden reason why production systems like vLLM and TensorRT-LLM push paged attention and prefix caching so hard. They’re not optimizing compute — they’re optimizing memory management. (LLM Inference Optimization | Speed, Cost & Scalability has a great section on this.)

Serving Architecture: Where Most Teams Get It Wrong

You can have the fastest model in the world. If your serving stack is bad, it doesn’t matter.

Teams sometimes spend weeks optimizing a model, only to run it through a naive FastAPI server that processes requests one at a time. A model that is fast in isolation then delivers a fraction of that speed in production because the server can’t batch.

Continuous batching is the single biggest performance lever for LLM serving. Instead of waiting for a batch to fill before processing, you add incoming requests to the running batch as slots free up. NVIDIA claims this improves throughput by 10-30x compared to naive batching.

But continuous batching introduces its own problems: variable latency. If you’re serving a mix of short and long requests, the short ones get stuck behind the long ones. That’s why systems split prefill and decode into separate phases — you prefill all pending requests first, then decode them in a coordinated schedule.

Switching from naive batching to continuous batching with split-phase scheduling can cut P95 latency sharply. Same model, same hardware, same traffic. Just better software. (What's New in LLM Inference Optimization covers these recent scheduling advances.)

Model Architecture: The Trade-Offs You Can't Escape

Not all LLMs are created equal when it comes to inference speed. The architecture choices made during training determine your latency ceiling at inference time.

Multi-Query Attention (MQA) and Grouped Query Attention (GQA) are the biggest wins here. Regular multi-head attention computes separate key and value projections for each head. MQA shares them across all heads. GQA groups heads and shares within groups. This dramatically reduces memory bandwidth usage.

LLaMA 2 used MHA. LLaMA 3 switched to GQA. That’s one reason LLaMA 3 8B often outperforms LLaMA 2 13B on latency while matching it on quality.

But there’s a cost: quality degradation. A model with full attention can score higher on quality benchmarks than one with sliding window attention plus GQA (such as Mistral 7B), but it takes longer per token.

The question isn’t “which architecture is faster?” It’s “what quality level can you accept, and what latency do you need?”

Speculative decoding is the other architectural trick that’s gaining traction. You use a cheap, fast draft model to generate candidate tokens, then verify them with the large model. If the draft model’s good, you get 2-3x speedup with zero quality loss.

But it only works well when the draft model aligns with the target model. If the draft model’s accuracy is poor, verification failure rates get so high that the whole thing is slower than just running the big model directly. (What Factors Influence LLM Inference Speed? has benchmarks showing when speculative decoding helps and when it hurts.)

Quantization: The Double-Edged Sword

Quantization is the single biggest knob you can turn for inference speed. But it’s not free.

Going from FP16 to INT8 gives roughly 2x speedup. INT4 gives 3-4x. But at INT4, you start losing quality on generation tasks. On classification or embedding tasks, the loss is negligible. On creative writing or complex reasoning? It shows.

A configuration that works well in practice:

# Using bitsandbytes for quantization - works well for most cases
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,  # Double quantization reduces memory
    bnb_4bit_quant_type="nf4"        # Normalized float 4 is better than int4
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto"
)

The double quantization trick (quantizing the quantization constants) saves another 0.5 GB per 7B model. Not huge, but on an 8 GB card, that extra 500 MB is the difference between running and crashing.

For production serving, AWQ and GPTQ offer better quality at INT4 than naive quantization because they calibrate the quantization based on actual weights. AWQ models typically show only a small accuracy drop vs. FP16 on most benchmarks. (Ultimate Guide to LLM Inference Optimization has a good comparison of these methods.)

Hardware: You're Probably Under-Utilizing Your GPU

Use your GPU better: Flash Attention 2 gives 2–3x on long sequences; H100 adds 67% bandwidth and FP8 tensor cores

A common mistake is buying bigger GPUs when what you need is to use the ones you have better.

Most GPU utilization in LLM inference sits around 20-40%. The rest is stalls — waiting for memory, waiting for attention synchronization, waiting for the next token in the batch.

Flash Attention changed this. By tiling the attention computation and avoiding materializing the full attention matrix in HBM, Flash Attention 2 achieves 2-3x speedup on long sequences. Flash Attention 3 (released early 2026) adds Hopper GPU optimizations that push it further.

# Flash Attention 3 usage - requires CUDA 12.2+ and H100+
import flash_attn

# This replaces the standard attention module
model.model.layers[i].self_attn = flash_attn.flash_attn_func

# For H100s, you can use w/ tensor cores for 50% faster attention
flash_attn.flash_attn_with_tensor_cores(
    q, k, v,
    causal=True,
    softmax_scale=q.shape[-1] ** -0.5
)

The hardware generation also matters more than most people think. A100 to H100 isn’t just a speed bump — it’s a different architecture. H100 has Transformer Engine support, FP8 tensor cores, and 3.35 TB/s memory bandwidth vs. A100’s 2 TB/s. That’s 67% more bandwidth.

But an A100 running Flash Attention + INT4 + continuous batching will beat an H100 running naive FP16 inference. Software first, hardware second. Always.

The Statistical Reality of Decoding

Here’s a counterintuitive truth: the model’s own output distribution determines how fast it can generate.

During autoregressive decoding, the model produces a probability distribution over the vocabulary. To sample the next token, you draw from that distribution. But drawing from a distribution is a synchronous operation — you have to wait for the full distribution to be computed before you can pick the next token.

Speculative decoding works around this by guessing the next few tokens using a cheap model, then verifying them with the expensive model. If the guesses are right, you get multiple tokens per forward pass.

But there’s a subtlety: the speedup depends on the model’s entropy. When the model is confident (low entropy), speculative decoding predicts well. When it’s uncertain (high entropy), the draft model fails more often. This isn’t a failure mode of the technique — it’s a fundamental property of the task.

Track speculative decoding acceptance rates per request and you see this: high-entropy prompts (like “write a creative story about x”) get lower acceptance than low-entropy prompts (like “summarize this financial report”). The difference translates directly to throughput.

Batch Size: The Most Important Knob Nobody Tunes Correctly

Batch size is the easiest thing to change and the hardest to get right.

Small batch (1-4): lowest latency, worst throughput. Each request occupies the full GPU compute for its entire duration.

Large batch (64+): best throughput, worst latency. Requests queue up and wait for the batch to fill.

The optimal batch size depends on your latency SLO. If your service needs <500ms time-to-first-token for 99th percentile, you’re likely stuck with batch sizes of 1-2. If you can tolerate 2-second first tokens, you can batch 32-64 and get 10x the throughput.

Here’s the key insight: latency and throughput are not independent. You trade one for the other. Most teams optimize for the wrong metric — they chase low P50 latency while their P99 service-level objectives (SLOs) bleed.

# vLLM configuration with optimal Batching
from vllm import LLM, SamplingParams

llm = LLM(
    model="mistralai/Mistral-7B-v0.1",
    max_num_batched_tokens=8192,  # Match to your GPU memory
    max_num_seqs=256,             # Max concurrent sequences
    enable_prefix_caching=True,   # Cache common prefixes
    gpu_memory_utilization=0.95,  # Use almost all GPU memory
)

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=500,
    best_of=1,     # No beam search - kills speed
    use_beam_search=False,
)

The gpu_memory_utilization=0.95 setting is aggressive. If you set it too high, the KV cache management fights with the model weights for memory and you get OOM errors. A slightly lower value leaves headroom. Your mileage will vary.

Which Optimization Should You Do First?

If you’re reading this and thinking “where do I start?” — here’s a priority list, starting with the biggest wins:

  1. Measure your actual bottleneck. Is it time-to-first-token or per-token latency? They have different causes. TTFT is about prompt processing; per-token is about memory bandwidth.

  2. Enable continuous batching. Switch to vLLM, TensorRT-LLM, or SGLang. This alone can give you 5-20x throughput improvement. It’s the highest-leverage change.

  3. Use Flash Attention. It’s free speed. Drop-in replacement for standard attention. No quality loss.

  4. Quantize to INT4. AWQ or GPTQ. Test quality on your specific task before declaring it fine. Don’t trust general benchmarks.

  5. Reduce context length. Do you really need 128K context? Most use cases work fine with 4K-8K. Every doubling of context doubles the attention computation.

  6. Speculative decoding. If your model is >30B parameters and you have a good draft model, this is worth it. Below 30B, the overhead of managing two models often outweighs the benefit.

  7. Hardware upgrade. Only after you’ve exhausted the software knobs. An A100 properly tuned beats an H100 out of the box.

The Dirty Secret: Most Inference Sucks Because of Engineering, Not Math

The single biggest reason for slow inference is not the model or the hardware — it’s the software stack around it.

Unoptimized data loading. Wrong batch sizes. Missing caching layers. Synchronous request processing. Poor memory management. No profiling.

When a team complains about slow inference, check the tokenization pipeline first. If the tokenizer runs on CPU while the GPU sits idle, that’s 20-50ms of wasted time per request. Multiply by request volume and it adds up.

The second thing to check: are they running the model with torch.inference_mode()? If not, they’re paying the overhead of gradient tracking on every forward pass. Simple fix, huge impact.

# The absolute minimum for fast inference
import torch

model.eval()  # Set to evaluation mode
model.to("cuda")

# CRITICAL: Disable gradient tracking
@torch.inference_mode()
def generate(prompt_ids):
    return model.generate(
        prompt_ids,
        max_new_tokens=250,
        use_cache=True,           # Must be True for speed
        do_sample=False,          # Greedy decoding is faster
        pad_token_id=tokenizer.eos_token_id
    )

Adding @torch.inference_mode() and use_cache=True to a model that runs on default settings can cut inference time noticeably.

The Future: What’s Coming

As of mid-2026, we’re seeing three trends that will change how we think about LLM inference speed:

Chip-level optimization. Groq’s LPUs and Cerebras’ wafer-scale chips are hitting 500+ tokens/second for models that run at 50 tokens/second on GPUs. These aren’t consumer products yet, but the handwriting is on the wall — memory bandwidth is the bottleneck, and purpose-built hardware solves it.

Multi-modal inference fusion. The new generation of models (Gemini 2.5, GPT-5) process images, audio, and text together. This changes the bottleneck profile. For image-heavy prompts, the vision encoder becomes the constraint, not the language model.

Disaggregated serving. Instead of running everything on one GPU, systems are splitting prefill (compute-heavy) and decode (memory-heavy) across different hardware. Prefill runs on H100s with high compute; decode runs on cheaper hardware with high memory bandwidth. Early results show 40% cost reduction at the same latency.

FAQ: Quick Answers to Common Questions

Q: Why is my LLM inference slow even on an A100?

A: You’re probably not using continuous batching or Flash Attention. Check your utilization with nvidia-smi. If GPU memory is used but compute utilization is under 50%, you’re memory-bandwidth bound. Quantize your model.

Q: Does batch size affect latency or throughput?

A: Both. Larger batches improve throughput (more tokens/second total) but hurt latency (each request waits longer). Find the batch size where you meet your SLO while maximizing throughput. It’s a tuning exercise, not a one-size-fits-all.

Q: Is FP16 or INT4 better for production?

A: Depends on your quality requirements. INT4 is 3-4x faster but has 1-2% accuracy drop on complex tasks. Test on your specific use case. If you need maximum quality, FP16 with Flash Attention. If speed matters more, INT4 with AWQ or GPTQ.

Q: Why does the first token take so much longer than subsequent tokens?

A: The prefill phase processes the entire prompt through the model in one pass. For a 10K-token prompt, that’s processing all those attention pairs. The decode phase processes one token at a time. Prefill is O(n²), decode is O(n). This is normal.

Q: How much does context length affect speed?

A: Dramatically. Doubling context length roughly doubles TTFT. For per-token latency, the effect is smaller but still significant — longer contexts mean more positions to attend to. Every optimization (Flash Attention, KV cache management) focuses on this.

Q: Should I use speculative decoding?

A: Only if your model is >30B parameters and you have a draft model with >70% acceptance rate. For smaller models, the overhead of running two models isn’t worth it. Test with your actual traffic before committing.

Q: Does prompt engineering affect inference speed?

A: Indirectly. Shorter prompts mean faster prefill. Well-structured prompts can reduce generation length (fewer clarification tokens). But prompt engineering optimizes for output quality, not speed. These are separate concerns.

The Bottom Line

Bottom line on slow LLM inference: profile first, continuous batching, Flash Attention and tested INT4, hardware last

LLM inference is slow because it’s fundamentally memory-bandwidth bound. Every token requires loading billions of parameters from memory to compute. No optimization eliminates this — it just works around it.

The good news: most of the low-hanging fruit is in software. Continuous batching, Flash Attention, quantization, and better serving infrastructure can give you 10-100x improvement over naive deployment. You don’t need a new GPU. You need to use the one you have properly.

Start by profiling. Find your actual bottleneck. Then pick the optimization that addresses it. Don’t optimize everything at once — you’ll end up with a system that’s fast but unmaintainable.

And for god’s sake, make sure your tokenizer is running on the GPU.

Want to talk this through?

Sivaro AI →