Stop Blaming the Load Balancer: Fix Your Cache Key First

There's a pervasive myth in our industry that the load balancer is the root of all cache hit ratio problems. Engineers tweak round robin versus least connections, obsess over sticky sessions, and blame the algorithm for every dip in performance. But here's the hard truth: your load balancer is innocent. The real culprit is your cache key. I've seen it time and again: teams spend weeks tuning load balancing algorithms, only to discover that the cache key was mangling their hit rates all along.

The Cache Key Is Your Real Load Balancer

Think about it. A cache key is what determines whether a request hits the same cached response or falls through to the origin. If your cache key is too broad, you're serving stale data to users who should get fresh content. If it's too narrow, you're fragmenting your cache into thousands of tiny, useless shards. The load balancer, whether it's nginx or HAProxy, just routes traffic; it doesn't care about your cache key. But the cache key determines what gets stored and what gets reused.

Look at the Vary header, for example. According to MDN, Vary lists the parts of the request (aside from method and URL) that influenced the response, so responses are cached separately per listed header. This is most often used for content negotiation. If you're not careful with Vary, you can accidentally create a separate cache entry for every User-Agent, every Accept-Language, every random header. I've seen caches with millions of entries for what should be a handful of resources, all because someone threw Vary: * on there, which MDN tells us implies uncacheable. That's a cache key disaster.

So before you even look at your load balancer configuration, audit your cache key. Are you using the full URL? Are you including the query string? Are you respecting Vary? If your cache key is a mess, no load balancing algorithm can save you.

Why Your Load Balancer Is Not the Problem

Let me preempt the obvious counter-argument: "But what about consistent hashing?" Yes, nginx offers a consistent hash directive that, per the nginx upstream module documentation, maps clients to servers by a hashed key and, with the consistent parameter, uses ketama consistent-hashing so adding or removing a server remaps only a few keys, helping achieve a higher cache hit ratio. That's a real technique, and I'm not saying it's useless. But here's the thing: consistent hashing only helps if your cache is distributed across multiple servers and you're trying to keep related keys on the same node. If you're using a shared cache like Redis (which you probably should be), the load balancer's role in cache locality is minimal.

Moreover, the load balancer doesn't even see the cache key in most cases. At Layer 7, nginx and HAProxy can inspect HTTP headers, but they don't parse the cache key; that's the cache's job. So blaming the load balancer for a low hit ratio is like blaming the post office for a poorly addressed letter.

The Real Fix: A Sane Cache Key

Here's what I actually recommend: start with a cache key that's as narrow as possible while still being correct. That means including the method, the URL, and any headers that genuinely affect the response (as indicated by Vary). Don't include headers that don't matter. And for heaven's sake, don't use Vary: * unless you want to disable caching entirely.

But a cache key is only half the story. You also need to think about cache invalidation. If you're using a CDN like Cloudflare, you can purge by single-file, which instantly removes a cached resource across all data centers, as documented by Cloudflare. That's a powerful tool, but it's manual. If you're building your own cache, you need to handle invalidation on writes. RFC 9111 tells us that unsafe methods like PUT, POST, or DELETE must invalidate the stored target URI when they get a non-error response. That's a non-negotiable rule. If you're not invalidating on writes, your cache key is moot.

What I'd Actually Do

Here's my concrete recommendation: stop fiddling with load balancer algorithms and fix your cache key strategy. First, audit your current cache key. Are you using Vary correctly? Are you accidentally including volatile headers? Second, implement proper invalidation. If you're using a CDN, set up purge-by-URL or purge-by-tag. If you're using Redis, use the cache-aside pattern (check cache, on miss fetch from DB, write back) and set a TTL via SETEX, as AWS recommends. Third, use a distributed cache like Redis for your application layer, and don't rely on the load balancer for cache affinity. Redis is an in-memory key-value store with typical hit rates of 90-95%, and adding it often reduces database CPU by 70-90%, per AWS. That's the kind of improvement that matters, and it comes from cache design, not load balancing.

Finally, if you're still convinced you need consistent hashing, use it at the cache layer, not the load balancer. Redis Cluster shards data using hash slots (CRC16(key) modulo 16384), per Redis documentation. That's where consistent hashing belongs. Your load balancer should just round-robin or least-connections based on connection characteristics, as the system-design-primer suggests. But don't expect it to fix your cache hit ratio.

In short: fix your cache key, implement proper invalidation, and use a real cache. The load balancer was never the problem.

Sources

• MDN (Vary) - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary
• Nginx (upstream module) - https://nginx.org/en/docs/http/ngx_http_upstream_module.html
• Cloudflare (purge by single-file) - https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-single-file/
• RFC 9111 (HTTP Caching) - https://httpwg.org/specs/rfc9111.html
• AWS (caching overview) - https://aws.amazon.com/caching/
• Redis (Cluster scaling) - https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/