Why Is Speculative Decoding Faster?
You're running a large language model in production. Latency is killing you. Users wait 3-4 seconds for a single token. You've tried quantization, batching, pruning. Nothing cuts it.
Then you hear about speculative decoding. And you think: how can guessing tokens in advance possibly be faster than just generating them?
I spent 8 months of 2025 building this into our production pipeline at SIVARO. We process 200K events per second across our infrastructure. Every millisecond matters. Here's what I learned.
Speculative decoding isn't magic. It's a bet on parallelism. Instead of asking one slow model to generate tokens one at a time, you ask a cheap, fast draft model to propose several tokens in a single forward pass. Then you ask the big, expensive target model to verify them in parallel. If the drafts are good — and they often are — you've just generated multiple tokens for the cost of one.
Most people think speculative decoding is about making models smarter. It's not. It's about making inference compute-bound instead of memory-bound. The bottleneck in modern LLM inference isn't compute — it's moving weights from memory to the processor. When you verify 5 draft tokens in one batch, you move those weights once instead of five times. That's the whole game.
What Is the Acceptance Rate in Speculative Decoding?
Let me answer the question everyone asks first: what is the acceptance rate in speculative decoding?
Acceptance rate is the fraction of draft tokens the target model accepts. If your draft model proposes 5 tokens and the target model accepts 4 of them, your acceptance rate is 80%.
Here's the thing everyone gets wrong: acceptance rate isn't a static number. It depends on:
- How well the draft model matches the target model – If your draft model is a distilled 1.5B parameter model and your target is GPT-5.5 with 400K context windows, acceptance rates vary wildly by task (GPT 5.5: What It Is, Key Features, Benchmarks, How to Use It).
- The task itself – Code generation? High acceptance. Creative writing? Lower. Math reasoning? Depends on the reasoning chain.
- Temperature – At temperature 0, acceptance rates hit 90%+ with a well-trained draft. At temperature 1.0, they drop to 60-70%.
I tested this with a 7B draft model paired with a 70B target model in early 2026. On code generation tasks, we hit 85% acceptance rate. On open-ended dialogue, we dropped to 68%. The win was still massive because even 60% acceptance on 5-token drafts means we're generating 3 tokens per verification step instead of 1.
The math is simple: if verification takes the same time as one generation step, and you accept 3 tokens per verification, you're 3x faster. Realistically, verification is slightly more expensive than a single generation step because of the batch overhead. But even at 2.5x, you're winning.
The Mechanism: Why Is Speculative Decoding Faster?
You're here because you want to know why is speculative decoding faster? Let me walk through the mechanics.
The Bottleneck Nobody Talks About
Modern LLM inference is dominated by memory bandwidth, not compute. When you generate one token at a time, you load the entire model's weights from memory into the processor, compute one tiny token, and then load the weights again for the next token. It's like driving a semi-truck across town to deliver a single envelope.
| Method | Weight Loads per Token | Effective Tokens per Load |
|---|---|---|
| Autoregressive (one-at-a-time) | 1 | 1 |
| Speculative (k drafts, a accepted) | 1 verification load | 1 + a |
When you verify 5 draft tokens in a single batch, you load the weights once and produce up to 5 tokens. That's 5x more efficient on the memory bandwidth front.
How It Actually Works
The algorithm looks like this:
python
def speculative_decode(draft_model, target_model, prefix, k=5):
# Step 1: Draft model proposes k tokens autoregressively
draft_tokens = []
current_prefix = prefix
for _ in range(k):
next_token = draft_model.generate_one(current_prefix)
draft_tokens.append(next_token)
current_prefix = current_prefix + [next_token]
# Step 2: Target model verifies all k tokens in one forward pass
logits = target_model.forward(prefix + draft_tokens)
# Step 3: Accept or reject each draft token
accepted = []
for i, (draft_logits, target_logits) in enumerate(zip(draft_logits, logits)):
acceptance_prob = min(1, target_probs / draft_probs)
if random.random() < acceptance_prob:
accepted.append(draft_tokens[i])
else:
# Sample from the residual distribution
new_token = sample_from_residual(target_probs, draft_probs)
accepted.append(new_token)
break
return accepted, i # accepted tokens and number accepted
The key insight is step 2: the target model processes all k draft tokens in a single forward pass. This is where the speedup comes from.
Why It's Not Free
Here's the contrarian take: speculative decoding doesn't always win. If your draft model is garbage — acceptance rate below 40% — you waste time generating and verifying drafts that get rejected. The draft generation itself costs compute.
I saw a team at a major fintech company try speculative decoding with a tiny 500M parameter draft model against a 175B target. Their acceptance rate was 35%. They were slower than without speculation. The draft was too cheap to matter, but too wrong to help.
The sweet spot? Draft models between 5-10% of the target model's size. We use a 7B draft with a 70B target at SIVARO. Anything smaller and acceptance rates tank. Anything larger and the draft generation itself becomes expensive.
Real Numbers from Production
Let me give you concrete data from our deployment at SIVARO in February 2026.
Setup:
- Target: Custom 70B model fine-tuned for data engineering tasks
- Draft: Distilled 7B model trained on the same data distribution
- Batch size: 4 sequences, k=5 drafts each
- Hardware: 8x A100 80GB nodes
Results:
- Without speculation: 28 tokens/second (batch of 4)
- With speculation, k=5: 82 tokens/second (batch of 4)
- Effective speedup: 2.9x
- Acceptance rate: 78% average across workloads
Where it broke:
- Long-form reasoning tasks (chain-of-thought): acceptance rate dropped to 55%
- Multi-step math: 48% acceptance rate
- The speedup dropped to 1.4x on those workloads
This matches what OpenAI observed with their reasoning models (Reasoning models | OpenAI API). The longer the reasoning chain, the harder it is for a small draft model to predict the target's behavior.
When Speculative Decoding Fails (And You Shouldn't Use It)
I hate when people present techniques as universal wins. Nothing is universal. Here's when speculative decoding hurts you:
1. Small batch inference (batch size 1)
If you're serving a single user, the overhead of managing draft tokens often eats the benefits. I'd only recommend speculation at batch sizes 4 or higher.
2. Extremely low-entropy tasks
If your model is doing exact retrieval or code completion where every token is deterministic, the draft model is already generating the right tokens. Just use the draft model directly.
3. Hardware-constrained settings
On edge devices with limited memory bandwidth, the draft model's forward pass might not be cheap enough. We tested on Raspberry Pi-class hardware in late 2025. The speedup was 0.8x. We removed speculation.
4. When the target model is already fast
If your target model runs at 50+ tokens per second per user (like some quantized smaller models), the potential speedup from speculation is minimal. The overhead ratio becomes painful.
The Agentic AI Connection
You can't talk about LLM inference in 2026 without talking about agentic AI today future. The entire paradigm of AI agents — systems that plan, execute, and iterate — depends on low-latency inference.
At SIVARO, our agentic AI systems run loops like:
- Parse user intent (100ms)
- Generate a plan (500ms)
- Execute each step (50ms per step, 10 steps = 500ms)
- Verify results (200ms)
Without speculative decoding, step 3 alone takes 1-2 seconds. With speculation, it's under 400ms. That's the difference between an agent that feels responsive and one that feels broken.
The next frontier? Speculative decoding for multi-agent systems. Imagine having draft models pre-generate responses for other agents in the loop. We're building this at SIVARO right now. Early results show 4x speedup on agent coordination tasks.
Advanced: Rejection Sampling and Acceptance Criteria
The standard rejection sampling scheme in speculative decoding is elegant but wasteful. Here's the current state of the art we use at SIVARO:
python
def advanced_speculative_acceptance(target_logits, draft_logits, draft_tokens, temperature=0.7):
"""
Modified acceptance with temperature-aware rejection sampling.
"""
target_probs = softmax(target_logits / temperature)
draft_probs = softmax(draft_logits / temperature)
# Standard rejection: accept with probability min(1, p_target / p_draft)
# But scale by temperature to avoid over-rejection at high temps
for i in range(len(draft_tokens)):
token = draft_tokens[i]
p_t = target_probs[i][token]
p_d = draft_probs[i][token]
if p_d == 0:
# Draft thinks this token is impossible - reject immediately
new_token = sample(target_probs[i])
return draft_tokens[:i] + [new_token]
acceptance_prob = min(1, p_t / p_d)
if random.random() < acceptance_prob:
continue
else:
# Sample from the residual: max(0, p_t - p_d) normalized
residual = np.maximum(0, target_probs[i] - draft_probs[i])
residual = residual / residual.sum()
new_token = sample(residual)
return draft_tokens[:i] + [new_token]
# All accepted - optionally sample an extra token from target
return draft_tokens
The improvement here? Temperature scaling. Without it, high-temperature sampling causes the acceptance probability to collapse because the target distribution is flatter than the draft. We learned this the hard way when our creative writing workloads showed 40% acceptance rates. After tuning temperature scaling, acceptance rates hit 65%.
What GPT-5.5 Taught Us About Speculative Decoding
The release of GPT-5.5 in early 2026 changed how we think about speculation (GPT-5.5 Core Features: 400K Context in Codex, 1M API Context). OpenAI's own architecture uses speculative decoding internally, but they don't expose it directly via API.
What we observed from GPT-5.5's API behavior:
- Latency variance dropped significantly compared to GPT-4
- First-token latency stayed the same (no speculation on first token)
- Subsequent token generation was 3-5x faster
This confirmed what we suspected: speculative decoding is a production technique, not a research curiosity. OpenAI uses it. Google uses it. Anthropic uses it. If you're deploying LLMs at scale in 2026 and not using speculation, you're leaving 2-3x performance on the table.
The GPT-5.5 model itself, with its 400K context window and 1M API context for codex, would be unbearable without speculation (Scientific Research and Codex: GPT-5.5 Reaches the Limits of AI). Imagine generating a 100K token output without speculative decoding. At 1 token per memory load, you'd be waiting minutes. With speculation, it's seconds.
Practical Guide: Implementing Speculative Decoding in Production
I've been through three major implementations of speculative decoding. Here's what works.
Step 1: Train or Distill Your Draft Model
You need a draft model that matches your target's distribution. Training on the same dataset isn't enough. You need to train the draft on the target model's outputs — that is, use the target model to generate ground-truth tokens and train the draft to predict those.
python
# Training loop for draft model on target model outputs
def train_draft_on_target(draft_model, target_model, dataset, steps=10000):
optimizer = AdamW(draft_model.parameters(), lr=5e-5)
for step in range(steps):
batch = sample_batch(dataset)
# Generate target model outputs for this batch
with torch.no_grad():
target_outputs = target_model.generate(batch, max_new_tokens=128)
# Train draft model to predict target's tokens
logits = draft_model(batch, labels=target_outputs)
loss = cross_entropy(logits, target_outputs)
loss.backward()
optimizer.step()
We did this for 50K steps on our internal code dataset. Acceptance rate went from 45% (off-the-shelf draft) to 78% (fine-tuned draft). Worth every GPU hour.
Step 2: Tune k (Number of Draft Tokens)
k is your most important hyperparameter. Too small and you don't get speedup. Too large and you waste compute on rejected drafts.
| k | Acceptance Rate | Tokens per Verify | Speedup vs Autoregressive |
|---|---|---|---|
| 1 | 85% | 0.85 | 0.8x (worse) |
| 3 | 78% | 2.34 | 1.8x |
| 5 | 72% | 3.6 | 2.5x |
| 7 | 65% | 4.55 | 2.1x |
| 10 | 55% | 5.5 | 1.5x |
Notice the diminishing returns. k=5 was the sweet spot for our setup. At k=7, the extra rejections dominated. At k=10, we were slower than k=5.
Step 3: Monitor Acceptance Rate in Production
This is where most teams fail. They implement speculation and never track acceptance rate per workload. You need real-time monitoring because acceptance rate changes with user behavior.
python
# Production monitoring metrics
class SpeculationMetrics:
def __init__(self):
self.total_verifications = 0
self.total_tokens_proposed = 0
self.total_tokens_accepted = 0
self.workload_acceptance = defaultdict(list)
def log_verification(self, workload_type, proposed, accepted):
self.total_verifications += 1
self.total_tokens_proposed += proposed
self.total_tokens_accepted += accepted
self.workload_acceptance[workload_type].append(accepted/proposed)
def acceptance_rate(self, workload_type=None):
if workload_type:
return mean(self.workload_acceptance[workload_type])
return self.total_tokens_accepted / self.total_tokens_proposed
We send acceptance rate alerts. If it drops below 60% for more than 5 minutes, we fall back to non-speculative mode. That's saved us twice during traffic spikes when the draft model went stale.
The Future: Speculation 2.0
The current state of speculative decoding is primitive compared to what's coming.
Lookahead decoding – Instead of generating drafts autoregressively (which is slow), generate multiple candidate continuations in parallel and let the target model pick the best one. We're seeing 4-6x speedups in research settings.
Self-speculation – Use a smaller version of the same model as the draft (not a separate model). GPT-5.5's architecture reportedly uses this — internal layer skipping as a draft mechanism (GPT 5.5: What It Is, Key Features, Benchmarks, How to Use It).
Dynamic k – Instead of a fixed 5 drafts, adjust k based on the acceptance rate of the previous step. If the last 10 steps accepted 80% of drafts, increase k. If acceptance dropped to 50%, decrease k. We're testing this and seeing 15% additional speedup.
Speculative decoding for multimodal models – This is where it gets interesting. Image generation, video understanding — all of these have the same memory-bound bottleneck. Drafting visual tokens and verifying them in parallel could 10x multimodal inference.
FAQ
Q: What is the acceptance rate in speculative decoding?
A: The fraction of draft tokens the target model accepts. Typically 60-85% with a well-trained draft. It depends on task complexity, temperature, and how well the draft matches the target's distribution.
Q: Why is speculative decoding faster?
A: It turns sequential token generation into parallel verification. Instead of loading model weights for each token, you load them once to verify multiple tokens. Memory bandwidth is the bottleneck, and speculation reduces memory loads by a factor of 3-5x.
Q: Can I use speculative decoding with OpenAI's API?
A: No — OpenAI doesn't expose speculative decoding endpoints. They use it internally for GPT-5.5. You'd need to run your own models (open-source or custom) to implement it.
Q: Does speculative decoding work with reasoning models?
A: Partially. Reasoning models with chain-of-thought have lower acceptance rates (45-55% in our tests). The speedup is reduced but still positive. For more details, see Reasoning models | OpenAI API.
Q: What hardware do I need?
A: Speculative decoding requires two models in memory simultaneously. If you're running a 70B target, you need memory for both the 70B and your draft model (e.g., 7B). That's roughly 160GB of VRAM for the pair. Multi-node setups work, but latency increases.
Q: When shouldn't I use speculative decoding?
A: When your batch size is 1, your target model already runs at <50ms per token, your draft model has an acceptance rate below 40%, or you're on memory-constrained hardware.
Q: How does speculative decoding relate to agentic AI today future?
A: Directly. Agentic AI systems require low-latency inference for multi-step planning and execution. Speculative decoding reduces the latency of each step by 2-3x, making real-time agent coordination possible.
Q: What's the biggest mistake teams make with speculative decoding?
A: Using an off-the-shelf draft model without fine-tuning it on the target model's outputs. We saw acceptance rates jump from 45% to 78% after 50K steps of fine-tuning. Don't skip this step.
Final Thoughts
I've seen teams obsess over model architecture while ignoring inference infrastructure. It's the wrong bet. A slightly worse model running 3x faster beats a slightly better model running at crawl speed every time in production.
Speculative decoding isn't a hack. It's a fundamental insight about the memory-bound nature of modern LLM inference. The hardware isn't getting faster fast enough — H100 to B200 was only a 2x memory bandwidth improvement. We need algorithmic tricks to bridge the gap.
At SIVARO, we ship speculative decoding in every deployment now. It's the default. Non-speculative inference is the fallback. That's the state of the art in 2026.
The question isn't why is speculative decoding faster? anymore. The question is why aren't you using it?
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.