When a Cache Hit Isn't a Win: The Load Balancer's Hidden Role

The Glow of a Cache Hit

There's a moment when you glance at your metrics and see a 95% cache hit rate. The dashboard glows green. The load balancer hums along. You think, 'We've nailed it.'

But here's the thing: a cache hit only means the system found something in the cache. It doesn't mean that something is correct, current, or even relevant. It could be stale, the wrong variant, or a response that violates a policy. And when that happens, the load balancer is just doing its job—routing traffic to a service that thinks it's fine.

In a recent project, we saw a service return a 200 OK with a cached response that was hours old. The cache had a TTL of 5 minutes, but a clock skew bug made it think 5 minutes hadn't passed. The load balancer saw a healthy backend. Users saw outdated prices. Nobody noticed until someone complained.

That's the gap between 'the cache worked' and 'the system worked.'

Why a Hit Can Be a Miss

Caches are great at reducing latency and offloading backends. But they also hide the truth. A cache hit doesn't tell you whether the underlying data has changed, whether the user is authorized to see it, or whether the source of truth is even reachable.

Consider a CDN edge that caches a JSON response. A load balancer routes a request to the edge, and the edge returns a cached copy. The user gets a fast response, but the copy is from before a critical update. The system just served a lie at lightning speed.

The same logic applies to application-level caches. When a service layer caches a query result, it might not invalidate when a dependent record changes. The load balancer keeps sending traffic, the cache keeps hitting, and the truth gets buried.

Load Balancing: More Than Just Spreading Traffic

Load balancers are often seen as traffic cops—send requests to the least busy server, retry on failure, and move on. But they also play a role in cache consistency and correctness.

Sticky sessions, for example, can pin a user to a specific backend. If that backend has a local cache, it might serve stale data that other backends have already invalidated. The load balancer isn't aware of the cache state, so it keeps routing to the same node, amplifying the error.

Health checks are another blind spot. A backend that returns a cached 200 might pass a health check even if its upstream connections are broken or its data is stale. The load balancer thinks the node is healthy, so it keeps sending traffic. The node keeps serving cached responses that are increasingly out of sync.

The Cost of Silent Success

Silent success is worse than a loud failure. A loud failure triggers alerts, pages, and incident reviews. A silent success just slowly erodes trust.

Think about an e-commerce checkout that returns a cached 'order placed' response after the actual order failed. The user thinks they've bought something. The business thinks the order exists. The cache just saved a few milliseconds, but it cost a customer.

In another case, a CDN cached a redirect for a URL that had changed. Users were sent to an old page, the load balancer saw no errors, and the SEO ranking took a hit. Again, no alarm bells.

Observability: The Missing Piece

To tell a good hit from a bad hit, you need observability that goes beyond hit rates and latency percentiles.

Start by tracking cache age. How old is the data being served? If the cache is serving data that's older than the freshness window, that's a problem, even if the hit rate is high.

Second, correlate cache events with source-of-truth changes. When a backend updates a record, does the cache get invalidated? If not, you're building a time bomb.

Third, monitor what the load balancer sees. If it's routing to a node that's serving stale data, that's a load balancing decision that should be flagged. Health checks should include a data freshness probe, not just a TCP ping.

Designing for Correctness, Not Just Speed

Caching and load balancing should be designed with correctness as a first-class citizen, not an afterthought.

Use cache keys that include version or timestamps when possible. This way, a new version creates a new key, and old keys expire naturally. It's a simple pattern that avoids a whole class of stale-data bugs.

Consider write-through or write-behind caching for critical data. This ensures the cache is updated when the source changes, reducing the window for stale reads.

For load balancing, use health checks that verify the backend can actually serve correct data. For example, a health check could hit a synthetic endpoint that returns a timestamp. If the timestamp is too old, the backend is marked unhealthy.

When to Bypass the Cache

Sometimes the cache should be bypassed entirely. For high-risk operations—like a payment or a permission change—always read from the source of truth.

In one system, we added a header to force a cache bypass for requests that involved user account changes. The load balancer was configured to route these requests directly to the origin, skipping the CDN. It was a small change, but it prevented a whole category of consistency bugs.

The same goes for writes. Never cache a write operation. If a user submits a form, the response should reflect the actual write result, not a pre-written placeholder.

Conclusion: The Real Metric Is Correctness

Cache hit rate is a vanity metric if it doesn't align with correctness. Load balancing is more than traffic distribution; it's about ensuring every request gets the right response, fast.

So next time you see a green dashboard, ask yourself: Are we serving the right data, or just serving something? The answer might be hiding in plain sight.