Cache Coherence in Large Scale Serving

Cache Coherence in Large Scale Serving

You've got 400 GPUs serving a model that's supposed to respond in under 100 milliseconds. The model weights are cached, the KV cache is warm, and your telemetry looks perfect — until a new deployment rolls out at 2 PM and everything falls apart.

The cache went cold. Not because you cleared it. Because you changed the model_version tag in your config and every node decided to invalidate its local copy of the weights simultaneously. The stampede hit your object store, the inference latency spiked from 90ms to 4 seconds, and your SLO was dead before you finished your coffee.

This is cache coherence in large scale serving. It's the difference between a system that behaves like a distributed database and one that behaves like a well-oiled machine. Most people think it's a computer architecture problem from 1995. They're wrong. It's the thing that determines whether your real-time inference system survives a deployment, a traffic spike, or a regional failover.

What Cache Coherence Actually Means Here

Cache coherence in large scale serving is the discipline of keeping multiple distributed caches consistent with each other and with the source of truth — without killing your throughput or latency. It's not about hardware cache lines in a CPU. It's about model weights, feature stores, embedding vectors, and prompt templates that live in memory across hundreds of nodes.

The problem is fundamental. You have N nodes serving traffic. Each node keeps a local copy of data it needs to serve requests quickly. That data changes — new model version, updated embeddings, refreshed user features. If even 1% of nodes serve stale data, you've got inconsistent predictions. If you synchronize aggressively, you take a latency hit on every single request.

Most systems I've audited in the last two years are doing versioned key-value lookups with TTLs and hoping for the best. It works until it doesn't. And when it doesn't, it's catastrophic.

The Three Coherence Models You'll Actually Use

There isn't a single solution. There are three patterns, and you'll use all of them depending on what you're caching.

Strong coherence — every read returns the most recent write. This is what you want for configuration, feature flags, and anything security-related. The cost is high: you're doing a round-trip to a consensus layer or a strongly consistent store on every cache miss. We used etcd for this at a fintech client in 2025, and it added 2-3ms to cold reads. Fine for config. Terrible for model weights.

Eventual coherence — every node eventually gets the update, but there's a window where stale data is served. This is what you want for model weights and embeddings. The trick isn't eliminating the window — it's making it tiny (milliseconds, not minutes) and handling the boundary gracefully.

Versioned coherence — you serve both old and new data simultaneously, marking requests with a version ID. This is the workhorse for inference systems. You load the new model, you drain the old one, and you never have a moment where a request gets served by a half-updated system.

Here's the thing most people miss: cache coherence in large scale serving is not a binary state. It's a spectrum, and you need to decide where each piece of your data sits on that spectrum based on how wrong it is to serve stale data.

Cache Warming Strategies for Inference

Let's talk about the part that actually keeps you up at night: cache warming strategies for inference. You can have perfect coherence logic, but if you don't warm your caches correctly, you're still going to fall over.

I tested this at SIVARO with a client running a large language model on 128 A100s. We had a robust invalidation system — versioned keys, propagate updates, done. But on a new deployment, we watched 4 out of 128 nodes return cold cache misses and take the whole fleet down with request timeouts and retry storms.

The problem was obvious in hindsight. We warmed caches by letting them fill naturally. Which meant different nodes warmed at different rates, and the clients (gRPC load balancers) kept sending traffic to nodes that weren't ready. The fix was a three-phase warming protocol that changed everything:

# Phase 1: Registration
# New node starts, fetches the model manifest, and registers with the control plane
GET /api/v1/models/llama-3.1-70b/versions
{
  "version": "2026.08.14-b",
  "weight_hash": "sha256:9f86d081...",
  "kv_cache_config": {"size": "48GB", "warmup": "synthetic"}
}

# Phase 2: Warmup (no traffic)
# Node loads weights, runs synthetic warmup queries, builds KV cache
while warmup_accuracy < 0.99:
    batch = generate_synthetic_traffic(pattern="production_mix")
    run_inference_passes(batch)
    report_warmup_progress(node_id, pct_complete)

# Phase 3: Gradual traffic ramp
# Control plane tells load balancer to send 5% traffic to this node, then 10%, etc.
POST /api/v1/load-balancer/set-capacity
{
  "node": "inference-08",
  "traffic_percent": 5,
  "ramp_rate": "5_percent_per_30_seconds"
}

The synthetic warmup is the trick. You don't wait for real traffic. You replay a sample of your production traffic — with the tokens and attention patterns that match what your users actually send. This builds the KV cache before a single real request hits the node.

The numbers were stark. With natural warming, a new node took 6-8 minutes to reach peak throughput. With synthetic warmup and a 5% ramp every 30 seconds, we got there in 90 seconds. Total deployment time went from 15 minutes to 4. And we didn't see a single timeout during rollout.

Caching for Real-Time Inference Systems

Caching for real time inference systems is where things get interesting. It's not just model weights — it's everything that feeds the model.

In a typical production setup, you're caching:

1. Model weights — immutable between deployments, large (70GB for a 70B model in FP16), shared across nodes
2. KV cache — the attention cache for ongoing conversations, node-local, ephemeral
3. Feature vectors — user embeddings, context embeddings, item embeddings
4. Computed results — "If user X makes request Y, the answer is Z" (semantic caching)
5. Context templates — system prompts, chain-of-thought starters, RAG context blocks

Each of these needs a different coherence strategy. Model weights are nearly immutable — you fetch them once per deployment. KV cache must stay in local memory with 100% hit rate. Feature vectors are the real challenge — they update continuously and you need them to be current without paying a network round trip on every request.

Here's what I'm seeing in production systems in 2026:

python
class CoherentFeatureCache:
    """Feature cache with versioned reads and async invalidation."""
    
    def __init__(self, store, coherence_ms=50):
        self.local = {}
        self.version_map = {}
        self.store = store
        self.coherence_ms = coherence_ms
        self._warmer = BackgroundTask(self._warm_continuously)
    
    def read(self, key):
        """Read the highest version of a feature, falling back to local if fresh enough."""
        local_version, local_data = self.local.get(key, (None, None))
        
        # If the local copy is within the coherence window, serve it directly.
        # This is the key data point: use `coherence_ms` to define "fresh."
        if self._version_in_window(key, local_version):
            return local_data
        
        # Otherwise, check the store for the latest version.
        latest_version = self.store.get_version(key)
        if latest_version != local_version:
            data = self.store.get(key)
            self.local[key] = (latest_version, data)
            return data
        
        return local_data
    
    def _version_in_window(self, key, local_version):
        """Check that the local version was updated within the coherence window."""
        # Track timestamps, not just versions, for realtime systems.
        if key not in self._last_checked:
            return False
        elapsed_ms = (time.now() - self._last_checked[key]) * 1000
        return elapsed_ms < self.coherence_ms

The key number is coherence_ms. I've found that 50 milliseconds is a sweet spot for most serving systems. If your feature can be 50ms stale, you serve it from local cache and you get 99.9% hit rates. If it absolutely can't be stale (fraud detection, safety filters), you pay the round trip.

A trading firm we worked with in early 2026 had a 5ms coherence window for their market data features. It worked, but they had to build a custom shared memory layer across nodes using RDMA. It cost them a quarter of a million dollars in engineering time. The business requirement justified it. Most requirements don't.

The Invalidation Problem You're Ignoring

Let me be direct about something. Most of you are doing cache invalidation by TTL. You set a 60-second TTL on a feature, and you accept that users might see a 60-second-old value.

Here's what happens in practice. An A/B test starts. The control group sees the new feature, the treatment group sees the old feature. Your analytics say the treatment group is underperforming by 15%, so you kill the test. But users in the treatment group have cached the old value, and for the next 60 seconds they're still seeing it. That's fine. The real problem is when your cascade of TTL expiration hits the database all at once.

I saw a client in the ad-tech space (anonymous for obvious reasons) bring down their Postgres replica with a TTL stampede. They had 60-second TTLs on 4 million ad embeddings, with all of them expiring within the same second. The replication lag spiked to 40 seconds, which meant every node was serving stale bids. Revenue impact: measurable in six figures for a single hour.

The fix is jitter. Add randomness to your TTLs. Instead of expiring at exactly 60 seconds, expire at 60 + random(-10, +10). This smooths the load curve. Simple, boring, effective.

But jitter only fixes the stampede. It doesn't fix the deeper issue: if you rely on TTLs for coherence, you're accepting a window of inconsistency you haven't measured. You don't know if that window is 10ms or 10 seconds. You haven't measured it because you can't — the TTL is a random variable from your perspective.

Version-Based Coherence: The Way You Should Be Doing It

Stop invalidating. Start versioning.

This is the biggest shift I've seen in production systems over the past 18 months. Instead of deleting a cache entry when data changes, you keep the old version alive for a short period and serve it alongside the new version. Every request gets a version tag. The load balancer knows which version is "current." The model knows which version it's using.

Here's a concrete example from a recommendation system we built:

go
type CacheVersion struct {
    VersionID string
    Features  map[string]FeatureValue
    CreatedAt time.Time
}

// Store keeps the last two versions alive.
// Version 1 is still finishing requests. Version 2 is the new hot version.
// No invalidation. No TTL. Just a pointer update.
func (c *CoherentCache) GetLatestVersion() CacheVersion {
    c.mu.RLock()
    defer c.mu.RUnlock()
    
    // The "current" pointer is atomic. Readers pick up the new version
    // within nanoseconds of the writer updating it.
    return c.versions[c.current]
}

// The key insight: we don't wait for requests in flight. We let them finish.
// If a request starts with version 1, it gets version 1's data. 
// New requests get version 2. This is the AB-test-safe approach.
func (c *CoherentCache) Serve(request Request) Response {
    version := c.GetLatestVersion()
    // ... run inference with version.Features ...
}

Version-based coherence eliminates the stampede problem entirely. You never have a moment where a key is missing. You never have a thundering herd requesting the same data from the source of truth. The old version is still cached, and you're draining it gracefully while the new version warms up.

We tested this at SIVARO in a multi-region deployment. The coherence-critical data (model configuration, embedding tables) was versioned. The result: zero invalidation-related incidents across 90 days of production traffic, including 23 deployments.

The cost is memory. You're holding at least two versions of some data. For a 700MB embedding table, that's an extra 700MB per node. For a 70GB model, you're holding 140GB. That's why versioning works better for small-to-medium datasets and you should still use shared model caching with a different pattern for the weights.

The Control Plane: What Holds It Together

You can't solve cache coherence with clever client code alone. You need a control plane that orchestrates version transitions across the fleet. This layer is what separates coherent serving systems from chaotic ones.

The control plane should handle:

1. Version registry — the authoritative list of what's current, what's old, what's deprecated
2. Rollout orchestration — ramping traffic to new versions node by node
3. Health monitoring — detecting nodes that serve stale data beyond the defined window
4. Automated rollback — reverting to a previous version when the new one causes errors

I've built this with Kubernetes Custom Resource Definitions and a controller loop. It's the right abstraction. The controller watches for a CacheVersion resource, then orchestrates the rollout across all replicas. The operative concept is to make the rollout itself the orchestrated process, not just the data.

yaml
apiVersion: sivar.io/v1
kind: CacheVersion
metadata:
  name: model-2026-08-14b
spec:
  weightsRef: "s3://model-bucket/llama-3.1-70b/2026-08-14b"
  featuresRef: "s3://feature-store/embeddings/2026-08-14b"
  priority: high
  rollout:
    type: ramped
    initialPercent: 5
    incrementPercent: 5
    incrementInterval: "30s"
  rollback:
    auto: true
    maxErrorRate: 0.02
    minHealthyNodes: 6

The control plane approach has a learning curve. But the alternative — every node independently deciding when to fetch new data — is chaos. I've seen systems where nodes were serving a model version from two weeks ago because they never successfully fetched the new one. The control plane eliminates that class of failure.

When Cache Coherence Breaks Down

Let me give you the failure modes. Not theoretical ones. Ones I've seen in production.

Failure 1: The cold-node stampede. You deploy a new model version. The control plane marks all nodes as "ready for update." All 200 nodes fetch the 70GB model from S3 simultaneously. Your ingress to S3 is now 14 terabytes of traffic. S3 throttles. Nodes start timing out. The deployment hangs.

Fix: Stagger the fetches. Have the control plane assign deployment slots. Node group A updates at T+0. Node group B updates at T+30 seconds. And so on. Implementation is simple. The problem is that most of you don't even know you're all fetching at once because you don't have coordination.

Failure 2: The partial-update nightmare. Your feature store has 5,000 features. A batch job updates 200 of them every hour. If you're treating the store as a single cacheable unit, you're invalidating 5,000 features to update 200. That kills your hit rate.

Fix: Shard your feature cache. Cache per-feature, not per-store. This is more granular bookkeeping, but the hit rate improvement is worth it. We saw a 40% reduction in feature fetching load when we moved to per-feature caching.

Failure 3: The stale-model-serving. A node fails to fetch the new model. The deployment completes. Now you have one node serving the old model. Requests to that node get different answers than requests to other nodes. In an A/B test, you've now got the control and treatment groups mixed in ways that corrupt your experiment.

Fix: The control plane should check the model version on every health probe. If the version doesn't match, the node gets marked unhealthy and sent to quarantine. It doesn't serve traffic until it fetches the correct version.

Failure 4: The memory leak of versions. You version everything and never clean up. After 3 months of daily deployments, you've got 90 versions of the model cached across your fleet. Memory is exhausted. People start fighting about who allocated what.

Fix: Keep exactly two versions alive. The current one and the previous one. Everything else gets garbage collected. This is non-negotiable. I've built eviction into the control plane so it's not a manual process.

How to Diagnose Coherence Issues Fast

You can't fix what you can't observe. Most coherence issues are invisible because teams measure the wrong things — average latency, average hit rate. What you need is the tail.

Monitor these:

• P95 and P99 read latency per node. A single node with 5-second latency for feature lookups is your cold node.
• Version skew across nodes. Track which model version and feature store version each node is serving. Any node with a different version is a problem.
• Stampede detection: Track the number of concurrent external fetches per data unit. If 50 nodes are fetching the same key, you're in stampede territory.
• Coherence staleness: Measure the actual observed staleness — the time between a write to the source and a read of that write across all nodes. Your target is 10-50ms.

FAQ: Cache Coherence in Large Scale Serving

Q: Is cache coherence in large scale serving the same as CPU cache coherence?

No, but it shares the same fundamental problem: multiple actors with local copies of shared data needing consistency. The difference is scale (hundreds of nodes instead of 8 cores), the cost of communication (network round trips instead of bus messages), and the tolerance for staleness (you can serve data that's a few milliseconds old, but not data that's minutes old).

Q: Should I use Redis for my cache coherence layer?

Redis works as a high-throughput shared cache, but it doesn't solve coherence by itself. You still need the versioning and validation logic on the client side. The server side is the easy part. The question is whether your clients know how to serve stale data when Redis is slow or unavailable. In 2025 we started seeing teams move coherence-critical logic from Redis to embedded solutions with versioned snapshots.

Q: What's the right coherence window for my inference system?

Start with 50 milliseconds. Most features can tolerate 50ms of staleness without issue. If you need tighter, measure the actual correctness impact before you invest in infrastructure. A lot of teams think they need 1ms coherence but are actually fine with 500ms. The cost difference is significant, and it's usually not worth it to optimize for a requirement that doesn't exist.

Q: How do I test cache coherence with load?

You test the same way you test everything else. Inject stale data. Inject missing data. Inject a failed fetch. Run a model deployment while serving traffic and verify that your telemetry shows a smooth transition between versions. Use chaos engineering: kill a node that's serving the old version and see if the system recovers on its own. At SIVARO, we run chaos drills as part of every release pipeline, and coherence issues are the first thing we catch.

Q: Is caching for real time inference systems the same as cache coherence?

Caching for real time inference systems is a broader category. It includes warm caches, request deduplication, semantic caching, and KV cache management. Cache coherence is the specific problem of keeping those caches consistent when data changes. The two work together: you need caching for speed, and coherence for correctness.

Q: What's the cheapest way to improve cache hit rates in inference?

You start recording every cache miss, and then you query the miss log (easily answerable with a streaming query engine like ClickHouse or DuckDB) to find the top 10 miss patterns, and you warm those specific patterns into the cache proactively. The cheap wins are in your own logs.

Q: Should I cache computed results (semantic caching)?

Yes, if your requests repeat. For interactive inference systems, users often send nearly identical prompts. Semantic caching based on embedding similarity can serve 30-40% of requests from cache without running the model. The coherence problem is more complex — you need to invalidate semantic caches when the model version changes, not just when the data changes. Approach with care, but it's worth the effort.

Coherence Is a Product Decision, Not a Technical One

At the end of the day, cache coherence in large scale serving isn't solved by a specific technology. It's solved by making a deliberate decision: how wrong is it okay to be, for how long, and in which circumstances?

I've worked with a ride-hailing company in 2025. Their pricing features had to be coherent within 10 milliseconds — a wrong price on surge could be a regulatory issue. I've also worked with a content recommendation system where 5 seconds of staleness was perfectly acceptable. The two systems looked completely different under the hood. The first had RDMA shared memory. The second had a Redis cluster with TTLs and simple versioning. Both were correct for their requirements.

The mistake is treating coherence as an all-or-nothing property. It's not. It's a continuum. You pick which level of coherence you need for each type of data, and then you build the right mechanism for it.

When you get it right, the system feels boring. You deploy new models and nothing breaks. You scale up and the cache warms predictably. You do a regional failover and requests continue smoothly with old data while caches warm. That's the goal. Not fancy invalidation schemes, but boring, reliable serving where nobody's surprised.

Get your control plane right. Version your data. Warm caches deliberately. And measure the actual staleness you're serving. That's the whole game.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.