Why Round Robin Is a Trap: Load Balancing for Cache-Friendly Architectures

Let’s settle this right now: round robin is the default load-balancing algorithm everywhere—Nginx, HAProxy, your cloud’s managed LB—and it’s quietly sabotaging your cache hit ratio. I get why it’s the default: it’s simple, it’s fair, and for short, stateless connections it’s genuinely fine (system-design-primer). But if you’re running a cache anywhere in front of your origin—a CDN, a reverse proxy, even a Redis layer—round robin is the wrong tool. It’s not about fairness; it’s about reuse.

Here’s the thesis: for any workload where you’re caching responses or data, you should be using hash-based load balancing (or at minimum least connections) between your load balancer and your cache tier. Round robin spreads requests evenly, but it also spreads them thin, so each cache node sees a tiny fraction of the traffic—and that’s exactly how you end up with a 40% hit rate instead of 90%.

Round Robin Is a Cache-Killer

Think about how a cache works. The whole point is that the same key gets served from memory, not from disk or the database. But a cache only works if the same key keeps hitting the same node. Round robin sends request #1 to server A, request #2 to server B, request #3 to server C. If you have three cache nodes, the same key will land on a different node every third request—and each node will miss, because the key isn’t there. The result? Your cache hit ratio tanks, and your database or origin gets hammered.

I’ve seen this play out in production more times than I can count. A team sets up Nginx as a reverse proxy with a shared cache (proxy_cache_path) and uses the default round robin. They wonder why the cache hit ratio is stuck at 30%. The answer is right there in the network path: the cache key is the URL, but the load balancer is sending each request to a different cache instance. It’s a recipe for cache thrash.

What Hash-Based Balancing Actually Does

Hash-based balancing—like Nginx’s hash key [consistent] directive or HAProxy’s source algorithm—fixes the fundamental mismatch. Instead of rotating through servers, it computes a hash of some part of the request (the URL, the client IP, a cookie) and maps that hash to a specific server. The same key always lands on the same node, which is exactly what a cache needs. Nginx has supported consistent hashing since version 1.7.2, using the ketama method, so adding or removing a server only remaps a few keys, which helps keep your cache hit ratio stable during scaling (Nginx upstream module).

For a cache layer, you want that stickiness. It’s the same reason AWS Route 53 offers weighted and multivalue routing, but those are DNS-level tricks—they don’t give you per-key affinity (AWS Route 53). Hash-based balancing is the load-balancer-level answer.

But Wait, What About Least Connections?

You might argue: “Least connections is better for long sessions—HAProxy’s docs recommend it for LDAP or SQL, where connections stay open (HAProxy configuration manual).” True. But those are long-lived, stateful sessions, not short HTTP requests. For caching, we’re usually talking about HTTP GETs that are short and stateless. Least connections balances by current load, not by key affinity, so it has the same cache-thrash problem as round robin. The only time I’d pick least connections over hash is if your cache nodes have wildly different capacities or if you’re serving mostly uncacheable dynamic content. Otherwise, hash wins.

And yes, round robin is fine for stateless services—if you’re not caching anything, spread the load evenly and move on (system-design-primer). But the moment you add a cache, you’ve changed the game.

Make It Work: Consistent Hashing and Cache Keys

Here’s the concrete pattern I recommend:

• Put your cache nodes behind a load balancer that supports consistent hashing (Nginx with hash $request_uri consistent; or HAProxy with balance source).
• Use a cache key that matches the hash input. If you hash the URL, make sure your cache key is the URL (plus Vary headers, if any).
• Set max_fails and fail_timeout on Nginx to avoid sending requests to dead nodes—the default is 1 failure in a time window, but tune it to your tolerance (Nginx HTTP load balancing).

For example, say you have three Nginx cache nodes behind a load balancer. With round robin, a single URL might hit node A, then B, then C, so each node misses and fetches from origin—three origin hits for one URL. With consistent hashing, that URL always hits node A, so the first request is a miss, but every subsequent request is a hit. That’s the difference between a 33% hit rate and a 99% hit rate for that key.

One caveat: consistent hashing only helps if your cache nodes are independent—i.e., each node has its own local cache. If you’re using a shared distributed cache like Redis Cluster, the load balancer’s job is different; you’re balancing to the Redis cluster, and the cluster itself shards keys across nodes using hash slots (16384 of them) (Redis Cluster scaling). In that case, round robin is fine—the cache is already key-aware. But for reverse-proxy caches or CDN edge caches, each node is a separate cache, and you need the LB to keep keys on the same node.

What I’d Actually Do

If you’re running Nginx as a reverse proxy with a file-based cache (proxy_cache_path), switch your upstream from round robin to hash $request_uri consistent;. If you’re on HAProxy, use balance source for TCP or balance hdr(host) for HTTP. Test it in staging first—the hit ratio should jump noticeably. And if you’re using a CDN, you don’t control the LB, but you can still benefit from URL-based caching at the edge; just make sure your origin returns proper Cache-Control headers so the CDN caches aggressively (MDN Cache-Control).

Don’t just accept round robin because it’s the default. The algorithm you choose at the load balancer is a cache-affinity decision, not a traffic-spreading decision. Choose hash, and your cache will finally do its job.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• Nginx (HTTP load balancing) - https://nginx.org/en/docs/http/load_balancing.html
• Nginx (upstream module) - https://nginx.org/en/docs/http/ngx_http_upstream_module.html
• HAProxy (configuration manual) - https://docs.haproxy.org/3.4/configuration.html
• AWS (Route 53 routing policies) - https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
• Redis (Cluster scaling) - https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/