Myth-Busting: The Case for Putting Your Cache Key Before Your Load Balancer

There's a myth that keeps circulating in our industry: that your load balancer's algorithm is the main lever for cache hit rates. I've seen teams burn weeks tweaking least_conn versus round robin, only to watch their CDN hit ratio stay stuck at 60%. That's backwards. The real lever is your cache key, and I'm going to show you why. If you're not thinking about how your responses are keyed, no load balancer algorithm will save you.

Isn't the Load Balancer the First Thing That Affects Caching?

No, and that's the misconception. The load balancer sits at Layer 4 or Layer 7 (system-design-primer), but it doesn't decide what gets cached or how. It decides which origin server gets a request. The cache key is decided by your HTTP headers and URL. When you have a CDN in front, the CDN's edge cache is what serves most requests. Cloudflare's network spans 348 cities (Cloudflare (network)), and a CDN can absorb 70% or more of origin requests (AWS (caching overview)). If your cache key varies wildly by user or session, you'll miss that absorption. The load balancer is downstream of that.

Does Round Robin Kill My Cache Hit Rate?

Only if you're using it for session persistence and your cache is per-server. But in a modern setup with a shared distributed cache or a CDN, round robin is fine. The problem isn't the algorithm; it's that your cache key includes something that shouldn't be there. For example, if you include a session ID in the URL or as a cookie that's part of the cache key, you're fragmenting your cache. The solution is to fix the key, not the algorithm. As the system-design-primer puts it, use round robin for short, stateless connections (system-design-primer). If your requests are stateless, round robin won't hurt.

What Does "Cache Key" Even Mean, and Why Should I Care?

Your cache key is the set of request attributes that determine whether a response is a hit or a miss. The Vary header is the HTTP mechanism for this: it lists the request headers that influenced the response, so responses are cached separately per listed header (MDN (Vary)). If you set Vary: User-Agent, you get separate cache entries per user agent. That's sometimes necessary, but it multiplies your cache entries. The more entries, the lower your hit rate. So before you touch your load balancer, ask: are you varying unnecessarily? Could you strip Vary or reduce its scope?

Can I Really Get 90%+ Hit Rates with Just Headers?

Yes. Typical Redis hit rates are 90% to 95% (AWS (caching overview)), and CDN hit ratios commonly reach 80% to 95% (AWS (caching overview)). But those numbers assume you're not fragmenting your key. Let me give you a concrete example. Say you serve a product page, and you set Cache-Control: max-age=604800, which keeps it fresh for 7 days (MDN (Cache-Control)). That's great. But if you also send Vary: Cookie and every user has a unique cookie, you've just made that page effectively uncacheable for shared caches. Instead, you should only vary on the specific cookie that changes the response, like a currency or A/B test cookie. That's a real fix.

What About Stale-While-Revalidate? Isn't That a Band-Aid?

No, it's a legitimate strategy, and it's defined in RFC 5861. It allows a cache to serve a stale response for up to the indicated seconds while revalidating in the background (RFC 5861 (stale-while-revalidate)). This is not a band-aid; it's a deliberate trade-off: you accept serving slightly stale data in exchange for lower latency and higher availability. I've seen this work wonders for news sites or product pages that don't change every second. But it doesn't fix a bad cache key. If your key is fragmented, stale-while-revalidate won't help because there's nothing to serve stale. So fix the key first, then consider stale-while-revalidate as a bonus.

So What's the Real Case Study: What Should I Do First?

Here's my recommendation, and I'll own it: audit your cache key before you even look at your load balancer configuration. Start with your CDN's default caching behavior. For instance, Cloudflare caches certain status codes by default: 200/206/301 for 120 minutes, 302/303 for 20 minutes, and 404/410 for 3 minutes (Cloudflare (default cache behavior)). If you're not sending explicit headers, you're getting those defaults, which may or may not be right. But the bigger issue is whether you're sending Vary headers that fragment your cache. Also, check if you're using no-store when you don't need to — that forbids any cache from storing the response (MDN (Cache-Control)). If you have a public page that changes rarely, no-store is a killer. So: fix your cache key, set appropriate Cache-Control, and then, if you still have issues, consider whether your load balancer algorithm matters for your specific traffic pattern. For short, stateless connections, round robin is fine (system-design-primer). For long sessions, least connections is better (system-design-primer). But don't blame the algorithm for a cache key problem.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• AWS (caching overview) - https://aws.amazon.com/caching/
• MDN (Vary) - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary
• RFC 5861 (stale-while-revalidate) - https://www.rfc-editor.org/rfc/rfc5861.txt
• Cloudflare (default cache behavior) - https://developers.cloudflare.com/cache/concepts/default-cache-behavior/