Stop Blaming Round Robin: Make Your Load Balancer Cache-Aware

Who This Is For

You're running a web service behind a load balancer, and you've got a Redis cache or a CDN in front of your origin. You've probably read that round robin is a trap, or that you should switch to least connections. But here's the thing: if your cache hit ratio is lousy, the algorithm is rarely the root cause. Most teams blame the balancer when they should be looking at the cache key. This guide is for engineers who want to stop guessing and start tuning the two together.

The Misconception: It's the Algorithm

Everyone thinks the load balancing algorithm is the magic dial. Switch from round robin to least connections and your cache hits will soar. Wrong. The algorithm distributes requests, but it doesn't decide whether a request can reuse a cached response. That's decided by the cache key and the headers that shape it. If your cache key is broken, no algorithm can save you. If your cache key is right, even a simple round robin works fine for most stateless traffic.

Step 1: Audit Your Cache Key Before You Touch the Balancer

Before you change a single balancing directive, look at what makes a response unique. The cache key is built from the request method, URL, and any headers listed in the Vary response header (MDN). If your origin sends Vary: Accept-Encoding, that's fine. But if it sends Vary: User-Agent or Vary: Cookie, you're fragmenting the cache into thousands of tiny pieces. Each fragment means a miss. The fix is to reduce the number of headers that go into the cache key. Do you really need to vary on the whole cookie? Probably not. Strip it down.

Step 2: Fix Your Cache-Control Headers

Your cache headers are the traffic lights for every cache in the path. If you're not sending them, or sending them wrong, your cache is guessing. Set an explicit max-age for static assets, and use s-maxage for shared caches like your CDN. Remember, per RFC 9111, when max-age is present, a recipient must ignore Expires. And if s-maxage is present, shared caches must ignore Expires too. So stop relying on Expires—it's a legacy fallback. Also, don't forget stale-while-revalidate from RFC 5861; it lets a cache serve a stale response while it revalidates in the background, which smooths out origin spikes.

Step 3: Choose the Algorithm That Matches Your Traffic

Now, the algorithm. For short, stateless connections—the kind you get with typical HTTP API calls—round robin is perfectly fine (system-design-primer). But if you're serving long-lived connections like WebSockets or SQL, least connections is better (HAProxy). For session persistence, you need a hash based on the client IP or a cookie (system-design-primer). The real trick for caching is consistent hashing. nginx's hash directive with the consistent parameter uses ketama, so adding or removing a server only remaps a few keys, which helps keep cache hits high (nginx upstream module). That's the one to use if you have an application cache that's tied to a specific node.

Step 4: Don't Ignore the CDN Layer

Your CDN is the first cache your users hit. If you're behind Cloudflare, know its default behavior: it caches certain status codes for a fixed time when no cache headers are present—200/206/301 for 120 minutes, 302/303 for 20 minutes, 404/410 for 3 minutes (Cloudflare default cache behavior). That's a lot of caching based on nothing. If you're not sending explicit headers, you're letting the CDN decide. That might be fine, or it might be terrible. Take control. And remember, the Vary header affects the CDN cache key too. If you over-vary, you'll kill your CDN hit ratio, which should be 80% to 95% (AWS caching overview).

Step 5: Test with Real Traffic

Don't change anything in production without measuring. Run a split test: keep one pool on round robin, another on least connections, and watch your cache hit ratio and origin load. Use a tool like curl to compare response headers, or your CDN's analytics. If you're using Redis, you can watch the hit rate—typical hit rates are 90% to 95% (AWS caching overview). If your hit rate is below that, your cache key or headers are the problem, not the algorithm.

What I'd Actually Do

Here's my blunt advice: For most web services, keep round robin. It's simple, it's predictable, and it's fine for short stateless requests. Spend your energy on the cache key and headers. If you have a sticky session requirement, use IP hash or a cookie-based hash. If you have a distributed cache that's per-node, use consistent hashing with nginx's consistent parameter. But don't chase least connections unless you have long-lived connections. And above all, measure. Your cache hit ratio is the metric that matters. Get it above 90% and your origin will thank you.

Sources

• MDN (Cache-Control) - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• RFC 9111 (HTTP Caching) - https://httpwg.org/specs/rfc9111.html
• Nginx (upstream module) - https://nginx.org/en/docs/http/ngx_http_upstream_module.html
• Cloudflare (default cache behavior) - https://developers.cloudflare.com/cache/concepts/default-cache-behavior/
• AWS (caching overview) - https://aws.amazon.com/caching/
• HAProxy (configuration manual) - https://docs.haproxy.org/3.4/configuration.html