Cache Invalidation: The Load Balancer Blind Spot You're Ignoring

The Load Balancer's Dirty Secret About Caches

When you put a cache behind a load balancer, you assume the balancer will always send a request to a node with the freshest data. That assumption is wrong more often than you think. Load balancers distribute traffic by health, weight, or session stickiness — none of which account for whether a node's cache is stale. The result: one user gets a fresh response, the next gets a cached copy from five minutes ago, and your application's behavior becomes inconsistent.

We've seen this pattern in production across thousands of deployments. A team adds Redis or Memcached to reduce database load, configures an NGINX or HAProxy load balancer, and calls it done. Then a support ticket arrives: 'Why does the dashboard show old data on refresh?' The cache is working. The load balancer is working. They just aren't talking to each other.

Why Cache Invalidation Is a Distributed Systems Problem

Cache invalidation is famously one of the two hard things in computer science, but in a single-node setup it's manageable: you delete a key, and the next request misses and fetches fresh data. Behind a load balancer, every node has its own cache (or a shared cache, but then you still have per-node in-memory caches like an L1). Invalidation has to reach every node, and the load balancer has no idea which node might serve the next request.

Consider a typical architecture: three application servers, each with a local Redis, fronted by an L7 load balancer using round-robin. When you invalidate a key on node A, nodes B and C still hold the stale value. If the balancer routes the next request to B, you've served stale data. The window is not milliseconds — it's the entire time it takes for your invalidation message to propagate, if it propagates at all.

Invalidation Propagation: The Missing Piece

You need a mechanism that tells every node to drop or update a key. The standard solution is a pub/sub channel. Redis has built-in pub/sub, and you can use it to publish an invalidation event. Each node subscribes and deletes the key from its local cache. This works, but it has a catch: if a node is down or disconnected during the event, it misses the invalidation. When it comes back up, it still has the stale entry.

A more robust approach is to version your cache keys. Instead of storing user:123:profile, store user:123:profile:v42. When data changes, increment the version. The application requests the latest version, so even if a node has an old version cached, it will miss and fetch the new one. This turns invalidation into a read-time check, not a write-time push.

How Load Balancer Stickiness Affects Cache Consistency

Session stickiness, also called sticky sessions, routes a client to the same backend node for the duration of a session. This is great for cache hit rates — the same user keeps hitting the same node's cache. But it amplifies inconsistency: if that node's cache is stale, that user is stuck with stale data until the session expires. Load balancer health checks don't help because they only check if the node is alive, not if its cache is fresh.

If you use sticky sessions and a write-through cache, you must invalidate on the node that owns the session. That's easy if the invalidation happens on that node, but if a write can come from any node (e.g., an admin dashboard), you still need broadcast invalidation. Some load balancers, like HAProxy, let you define a hash-based stickiness (e.g., on the user ID), which at least makes the node predictable.

A Step-by-Step Invalidation Workflow for Load-Balanced Caches

Here's a concrete workflow that has worked in production for us. It assumes you have a shared cache (like Redis) and an L7 load balancer.

1. Use a shared cache for the canonical copy. Store all cacheable data in a single Redis cluster. This avoids per-node divergence entirely. If you need in-memory speed, use the shared cache as the source of truth and local caches as a best-effort L1.
2. Publish invalidation events to a Redis pub/sub channel. On every write, publish a message with the key pattern (e.g., user:123:*).
3. Each node subscribes to that channel. On receiving an event, it deletes the matching keys from its L1 cache. Use a short TTL (e.g., 60 seconds) on L1 entries as a safety net.
4. Version your keys for long-lived data. For data that changes rarely but is read often (like configuration), append a version number to the key. When the data changes, increment the version and publish an invalidation event for the old version.
5. Configure your load balancer to prefer recently-healthy nodes. If a node has been down for more than a few seconds, drain it before re-enabling. This prevents a node with a stale cache from serving traffic while it re-subscribes to the pub/sub channel.
6. Have a fallback for cache misses. When a node gets a cache miss, it should fetch from the database and write to the shared cache. This ensures that even if an invalidation is missed, the next read refreshes the data.

Comparing Invalidation Strategies: A Practical Table

As the table shows, there's no free lunch. TTL is simplest but can serve stale data for minutes. Versioning gives you strong consistency without pub/sub, but you must generate and track versions. Our recommendation: combine versioning for immutable-ish data and pub/sub for dynamic data.

Real-World Example: An E-Commerce Product Catalog

Let's make this concrete. An e-commerce site has a product catalog that changes when an admin updates pricing. The catalog is cached in a Redis cluster, and the load balancer uses round-robin across three nodes. Without invalidation, a price change takes effect only when the cache TTL expires (say, 10 minutes). During that time, some users see the old price, others see the new one — a compliance nightmare.

We implemented key versioning for product data. The key became product:12345:price:v42. When the admin updates the price, the application increments the version to 43 and writes a new key. The old key expires via TTL (e.g., 1 hour). Any node that has v42 cached will miss on v43 and fetch from the database. The load balancer doesn't need to know anything. The inconsistency window dropped from 10 minutes to under 1 second, and the database load stayed low because only the first request for each product after a change hits the database.

When to Use a Load Balancer That Understands Caching

Some load balancers and API gateways offer cache-aware features, like caching responses at the edge (e.g., Varnish, Cloudflare, or NGINX's proxy_cache). In that case, the load balancer itself becomes a cache, and invalidation happens via cache purge requests. This is simpler for the backend nodes — they just serve fresh data — but you now have a single point of failure for cache consistency. If the edge cache misses an invalidation, it might serve a stale response to many users.

For most applications, we recommend keeping caching inside the application tier and using the load balancer purely for traffic distribution. If you do use an edge cache, ensure it supports key-based invalidation and test your failure scenario: what happens when a purge fails?

Conclusion: Invalidation Is a Feature, Not an Afterthought

You can't ignore cache invalidation when you put a load balancer in front of your cache. The balancer doesn't know what's in your cache, and it won't help you keep it fresh. Choose a strategy that fits your consistency requirements, implement it deliberately, and test it under load. A few hours spent on invalidation will save you from a production incident where users see stale prices, old comments, or incorrect balances.