Stop Chasing Load Balancer Algorithms — Fix Your Cache Key First

Here's the contrarian take: the load balancing algorithm barely matters for cache hit rates. Most teams obsess over round robin versus least connections, but the real lever is the cache key. Get that wrong, and no algorithm saves you.

The Algorithm Trap

Round robin is the default in nginx and HAProxy for a reason: it's simple and fair for short, stateless connections (system-design-primer). But when you're trying to improve cache hit ratios, the algorithm is a side show. Least connections? HAProxy's own docs say it's recommended for long sessions like LDAP or SQL, not short HTTP requests (HAProxy). IP hash? It maps clients to servers, but it doesn't fix a cache key that varies by user-agent or Accept-Encoding.

The Cache Key Is the Real Deal

The Vary header is where the magic happens. It tells caches which request headers to include in the cache key. If your origin sends Vary: User-Agent, you're splitting your cache by every browser version. That's a hit ratio killer. The right move is to normalize the request. Don't vary on User-Agent unless you absolutely must; vary on a normalized header like Vary: Accept-Encoding only. That way, a single cached response serves all clients that accept gzip, and you don't fragment your cache.

But What About Sticky Sessions?

You might argue: "I use IP hash to keep a user on the same server, so their session stays warm." Fair point. But that's session persistence, not cache efficiency. And it comes at a cost. If a server dies, IP hash remaps clients, and your cache on that server is gone. A shared cache like Redis or a CDN edge doesn't care which origin server handled the request. The cache key is the same. So the algorithm only matters for origin-local caches, and even then, the cache key dominates.

The Practical Fix

Stop tuning least_conn versus roundrobin. Instead, audit your Cache-Control and Vary headers. Make sure you're not caching dynamic content with no-store unless you mean it. Use s-maxage for shared caches. And for CDNs, remember Cloudflare caches specific status codes by default—200/206/301 for 120 minutes (Cloudflare). If you're serving a 404 for a missing image, it's cached for 3 minutes. That's fine. But if your cache key varies on every request, you're flushing money down the drain.

Here's a concrete example: an e-commerce site with 1,000 products. Each product page has a canonical URL. If you set Cache-Control: max-age=604800 (7 days) and don't vary on anything, the CDN can serve 95% of requests from the edge (AWS). But if you add Vary: Cookie, every user with a different session cookie gets a separate cache entry. Hit ratio tanks. The fix is to move personalization to an API call and keep the HTML cacheable.

One more thing: stale-while-revalidate is your friend. RFC 5861 lets you serve stale content while revalidating in the background (RFC 5861). That's a hit ratio booster without changing your load balancer.

So, the next time someone pitches a new load balancing algorithm, ask them about the cache key. The algorithm is a rounding error. The cache key is the whole ballgame.

Bottom line

Fix your cache key. Normalize Vary, set sane Cache-Control, and let the CDN do its job. The load balancing algorithm is a distraction.

Sources

• HAProxy configuration manual - https://docs.haproxy.org/3.4/configuration.html
• MDN Vary - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary
• RFC 5861 - https://www.rfc-editor.org/rfc/rfc5861.txt
• Cloudflare default cache behavior - https://developers.cloudflare.com/cache/concepts/default-cache-behavior/
• AWS caching overview - https://aws.amazon.com/caching/
• system-design-primer - https://github.com/donnemartin/system-design-primer