SIVARO
High Performance Caching for ML Inference: A Buyer's Guide
I still remember the Tuesday afternoon last March when a client's inference bill hit $47,000 in a single week. Their model was answering the same 200 questions over and over. Same prompts. Same outputs. Same GPU cycles burned 400,000 times for maybe 12,000 unique inputs.
That's not a model problem. That's a caching problem.
If you're running ML inference at any real scale in 2026, high performance caching for ml inference isn't optional infrastructure — it's the difference between a viable unit economics story and a board meeting you don't want to attend. GPU costs haven't dropped nearly as fast as everyone hoped. Llama 4 runs cheaper than GPT-5, sure, but you're still paying for every token.
This guide is what I wish someone had handed me in 2023. I've deployed caching layers at SIVARO for clients processing everything from vector similarity to full LLM completions. Some of it worked beautifully. Some of it was a mess. I'll tell you both.
Why caching ML inference is different from caching a REST API
Most engineers think caching is caching. You put a Redis in front, key by request, done.
That's wrong for ML.
A traditional API cache keys on URL + params. Deterministic. Simple. An ML inference cache has to deal with:
- Semantic equivalence. "What's the capital of France?" and "France's capital city?" are different strings with the same answer. Do you cache them together?
- Probabilistic outputs. Set
temperature=0and you get determinism. Set it to 0.7 and every call is technically a new call. - Embedding-based retrieval. You're caching vectors, not strings. Cosine similarity thresholds matter.
- GPU memory pressure. A cache hit that saves a 3-second generation is worth a lot. A cache hit that saves a 40ms embedding call is worth less.
We ran a benchmark in January 2026 on a client's RAG pipeline. Their retrieval layer was hitting a Pinecone index + a reranker + a generation call. Total latency: 2.8 seconds p50.
After adding semantic caching on the retrieval path alone, p50 dropped to 340ms. Cache hit rate was 61%. Their monthly GPU spend dropped 43%.
But — and this is important — a naive Redis cache on the same workload would have gotten maybe 8% hit rate, because the raw query strings almost never matched.
The four architectural patterns that actually work
After a few years of this, I've seen four patterns that survive contact with production traffic.
Exact-match caching
The simplest layer. Key is a hash of the normalized input. Fast, cheap, useless if your users phrase things differently.
You should still build it. It's your last line of defense and it costs almost nothing. Redis or Valkey works fine here. We use Redis 8 with client-side caching for hot keys.
python
import hashlib
import redis
r = redis.Redis(host='cache.internal', port=6379, decode_responses=False)
def exact_cache_key(model_id: str, prompt: str, params: dict) -> str:
payload = f"{model_id}|{prompt.strip().lower()}|{sorted(params.items())}"
return "xact:" + hashlib.blake2b(payload.encode(), digest_size=16).hexdigest()
def get_exact(cache_key):
return r.get(cache_key)
Hit rate on chat workloads: usually 5-15%. On batch pipelines with retries: 80%+. Depends entirely on your traffic shape.
Semantic caching
This is where high performance caching for ml inference gets interesting. You embed the incoming query, search a vector store for a near-neighbor, and if cosine similarity exceeds a threshold, you return the cached response.
The threshold is the whole game. Too high (0.98), you get low hit rates. Too low (0.85), you start returning wrong answers. I've seen production systems sit at 0.92-0.94 for Q&A workloads and 0.88 for support tickets where users paraphrase heavily.
python
from sentence_transformers import SentenceTransformer
import numpy as np
embedder = SentenceTransformer("BAAI/bge-m3")
SIM_THRESHOLD = 0.93
def semantic_lookup(query: str, cache_index, embedder):
q_vec = embedder.encode(query, normalize_embeddings=True)
matches = cache_index.search(q_vec, k=1)
if matches and matches[0].score >= SIM_THRESHOLD:
return matches[0].response, matches[0].score
return None, None
The trap: you need to invalidate. If your model gets updated, or your knowledge base changes, stale semantic hits are worse than no cache. We version cache namespaces by model hash + prompt template hash.
KV cache reuse (prefix caching)
This is invisible to most application teams and it's where the biggest wins live. If two requests share a long system prompt — say, a 3,000-token instruction block — you can reuse the attention KV cache instead of recomputing it.
vLLM, SGLang, and TensorRT-LLM all support this now. vLLM's automatic prefix caching is on by default in recent versions. SGLang's RadixAttention is arguably more aggressive.
We measured this on a client's customer support bot in February 2026. Same system prompt, 2,800 tokens. Prefix caching cut time-to-first-token from 890ms to 210ms on repeated prompts. That's not a cache hit in the traditional sense — it's compute reuse inside the inference engine.
If you're not using prefix caching, fix that before you buy anything else.
Embedding cache
Embeddings are deterministic functions. Same input text, same model, same vector. Always. So cache them aggressively.
We use a two-tier setup: Redis for hot embeddings (last 24 hours), then a Postgres table with pgvector for cold storage. Hit rate on cold storage was 34% in a recent audit. That's free money.
Comparing the real options
Here's what I'd actually evaluate if I were buying today.
Redis Stack + RediSearch. Full vector search, mature tooling, cheap on managed cloud. Downside: single-threaded per shard on the vector index, and full-text + vector hybrid is clunky. Good enough for up to ~5M cached embeddings.
Valkey 8 with a vector module. The Linux Foundation fork. If you're cost-sensitive and self-hosting, this is what I'd pick in 2026. Redis licensing changes in 2024 pushed a lot of teams here.
Pinecone / Weaviate / Qdrant (managed). Great if you don't want to run vector infra. Expensive at cache scale — a cache isn't supposed to cost more than the thing it's caching. Qdrant self-hosted is my personal favorite for cache workloads.
GPTCache. Open-source semantic cache from Zilliz. Fast to stand up. But the project's pace slowed in 2025, and I've seen scaling issues past 10M entries. Fine for a prototype, risky for a revenue path.
Cloudflare AI Gateway Cache. Newer entrant, late 2024, matured through 2025. Zero infra to run. Useful for edge-deployed apps. Limited control over similarity thresholds.
Custom in-process cache. For very high QPS on a small working set, an LRU in the inference server's memory beats any network hop. vLLM exposes this. So does NVIDIA Triton.
Pick based on scale, not marketing. Under 1M entries, Redis/Valkey. 1M-50M, Qdrant or a self-hosted Weaviate. Above that, you're building custom, and you know it.
What nobody tells you about invalidation
Cache invalidation is famously one of the two hard problems. It's worse for ML.
When do you invalidate a semantic cache entry?
- Model version bump: obvious, nuke the namespace.
- Prompt template change: subtler, easy to miss in CI.
- Knowledge base update in RAG: nightmares. A cached answer references a document that got retracted. Now your bot is confidently wrong.
- Embedding model change: your cached vectors are now in a different space. Everything is garbage.
We handle this with a cache key that includes a "context hash" — a SHA of the model ID, prompt version, and the IDs of any retrieved documents. Any change to that hash gives you a fresh cache line.
Cost: cache hit rate drops by ~15% in our experience. Benefit: you don't ship misinformation to customers.
Most teams skip this. Then they get a nasty support ticket in month six.
Latency math you should actually run
Cache infrastructure costs latency. A Redis round-trip is 0.5-2ms in-region. A vector search on 5M entries through Qdrant is 5-20ms depending on index.
So a semantic cache hit saves you, say, 2,000ms of generation. A miss costs you an extra 15ms. Break-even hit rate is trivial: roughly 1%.
But the real question isn't break-even. It's marginal. If your p95 latency budget is 800ms and your generation is 2,400ms p95, a 15ms miss penalty is nothing. If your generation is 120ms and you're on 3ms Redis hits, every miss hurts.
Do the math for your specific p99, not the average. Tail latency is where caching murders you or saves you.
Where teams get this wrong
Three patterns I've seen burn clients:
One. Caching on temperature > 0. If your app lets users set temperature above 0, the output is supposed to vary. Caching it violates the contract. Either force temp=0 for cacheable paths or use semantic caching only for retrieval (not generation).
Two. Sharing a cache across tenants. We saw a company in 2025 leak one customer's data into another's responses because a semantic cache hit crossed tenant boundaries. Cache key must include tenant ID. Non-negotiable.
Three. Ignoring cache poisoning. If an attacker can craft inputs that match cached entries maliciously, they can steer responses. Semantic caches are especially vulnerable. Validate before you write; namespace by user role where it matters.
I'll be honest: high performance caching for ml inference isn't magic. It's plumbing. Boring, careful plumbing. The teams that succeed are the ones treating it like a data system, not a shortcut.
A concrete stack recommendation
If you want a starting point for a production RAG + LLM app in September 2026:
- Exact cache: Redis 8, 30-minute TTL, tenant-scoped keys.
- Semantic cache: Qdrant, cosine threshold 0.93, 1M-entry HNSW with m=16.
- Embedding cache: Postgres + pgvector, no TTL, dedupe on (model_id, sha256(text)).
- Prefix cache: enable in vLLM or SGLang — free win.
- Invalidation: context hash in every key, hourly sweep of entries whose context hash is stale.
That stack has held up under real traffic. It's what I'd build again tomorrow.
FAQ
Q: What hit rate should I expect from semantic caching on chat traffic?
A: 40-65% for support and Q&A workloads. 15-30% for open-ended creative tasks. If you're getting under 10%, your threshold is too high or your users genuinely don't repeat themselves.
Q: Is Redis still the right choice for ML caching in 2026?
A: For exact-match, yes. For vector search, Redis Stack works but Qdrant or Weaviate win past a few million entries. Valkey is a fine Redis alternative if you're cost-sensitive.
Q: How much can high performance caching for ml inference actually save?
A: On our clients' workloads, 30-60% GPU cost reduction is typical. The best case we've seen was 71% on a Q&A bot with heavy repetition. Worst case was 8% on a coding assistant where every prompt was unique.
Q: What's the right similarity threshold for semantic cache?
A: 0.92 for general Q&A. 0.95+ for anything where a wrong answer is expensive (legal, medical, financial). Tune empirically — collect a labeled set of "should match" and "shouldn't match" pairs and sweep the threshold.
Q: Should I cache LLM completions with temperature > 0?
A: No. Use semantic caching on the retrieval or prompt-construction path instead. Caching non-deterministic outputs breaks user expectations.
Q: Do I need a separate cache for embeddings?
A: Yes, and it's basically free. Embeddings are deterministic. Same input, same vector. Cache them forever. We've seen 30-40% cold-cache hit rates on embedding lookups.
Q: What about prefix caching inside vLLM?
A: Turn it on. It's a 2-4x TTFT improvement on workloads with shared system prompts, and it costs nothing to enable. There's no reason not to.
Q: How do I handle model updates without nuking my cache?
A: Version your cache namespace by model hash + prompt hash. Keep the old namespace alive for a rolling window (we use 24 hours) so in-flight requests don't miss. Then delete.
The bottom line
You don't need the fanciest cache. You need the right cache for your traffic shape, honest thresholds, and invalidation you didn't skip.
Every dollar you spend on high performance caching for ml inference returns somewhere between 10 and 100 dollars in avoided GPU time, depending on your repetition rate. That's the best ROI in the ML infra stack right now, and it's not close.
Start with prefix caching. Then exact-match. Then semantic. Measure everything. Kill what doesn't earn its keep.
And for the love of everything, put the tenant ID in the cache key.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.