Least Connections vs. Round Robin for Cache Hit Rates

The 80% Problem: Why Your Load Balancer May Be Sabotaging Your Cache

Here's a number that should make you pause: CDN cache hit ratios commonly reach 80% to 95% (AWS). That's the industry baseline. But if you're running a homegrown cache or relying on an upstream load balancer that ignores cache affinity, you're likely nowhere near that. I've seen teams celebrate a 40% hit rate, not realizing they're leaving performance on the table. The culprit? The load-balancing algorithm itself.

We often think of load balancers as traffic cops—they just distribute requests. But they also shape how your cache behaves. If every request for a given resource lands on a different backend, your cache becomes a game of whack-a-mole. Each server stores its own copy, and you end up with duplicate entries and constant misses. So which algorithm actually plays nice with caching?

In this head-to-head, we're pitting two heavyweights: least connections and round robin. We'll evaluate them on three concrete criteria: cache affinity, connection distribution, and operational simplicity. By the end, you'll know exactly which one to use—and when to break the rules.

Round Robin: The Predictable Workhorse

Round robin is the default in both Nginx and HAProxy (Nginx, HAProxy). It's simple: each new request goes to the next server in line. For short, stateless connections—think API calls that don't hold a connection open—it's a solid choice (system-design-primer). It's also fair in the long run, ensuring no single server gets hammered.

But here's the rub: round robin has zero concept of cache affinity. If you have three backend servers, and a user requests the same product page three times, each request might hit a different server. That means three separate cache entries for the same resource, tripling your memory footprint and cutting your effective hit rate. This is especially painful for edge caches or application caches that live behind your load balancer.

Nginx's round robin also doesn't account for server load. If one server is slow due to a noisy neighbor, round robin keeps sending requests its way, leading to uneven response times and potential timeouts. That's why least connections exists.

Least Connections: The Adaptive Ally

Least connections sends each new request to the server with the fewest active connections. HAProxy's docs recommend it for long sessions like LDAP or SQL, but note it's not well suited to short HTTP sessions (HAProxy). Why? Because with short requests, connection counts fluctuate wildly, and you can end up with a server that just finished a burst of requests getting hit again, while others sit idle.

For caching, least connections doesn't inherently improve affinity either. It's still per-request routing, so the same resource can land on different servers. However, it does a better job of keeping load balanced, which can prevent any single cache from becoming a hotspot. If one server's cache is hot, it might have more active connections, so least connections might route traffic away from it—potentially lowering hit rate if that cache has the data.

But there's a nuance: some load balancers, like HAProxy, offer a uri algorithm that hashes the URL to ensure the same resource always goes to the same server. That's the real cache winner, but it's not in this head-to-head. Still, if you must choose between round robin and least connections for a cache-heavy workload, which is better?

Criterion 1: Cache Affinity

Cache affinity is the name of the game. Round robin gives you zero affinity—every request is a fresh roll of the dice. Least connections is marginally better: if a server has a cached resource, it might be handling fewer connections if that resource is popular? No, actually, popular resources cause more connections, so least connections would route away. It's a wash.

Neither algorithm is designed for cache affinity. If you want affinity, you need IP hash or URL hash. Nginx's ip_hash ensures a client always hits the same server, which helps session persistence but not necessarily cache hit rates for anonymous traffic (Nginx). HAProxy's uri does hash the URL, which is ideal for caching (HAProxy). So if you're choosing between these two solely for cache affinity, you're choosing between two wrong answers.

That said, if you're using a CDN in front, the CDN itself handles caching, and your origin load balancer's algorithm matters less for hit rate. But for internal caches or when you're the CDN, you need something better.

Criterion 2: Connection Distribution

Round robin distributes connections evenly by count, but not by weight. If you have heterogeneous servers (one with 2x the RAM), round robin will send equal traffic to both, potentially overloading the weaker one. Least connections adapts to actual load, sending more traffic to the underutilized server. In a caching scenario, this might mean the server with more memory (and thus a larger cache) gets more requests, which is actually good for hit rate.

HAProxy's leastconn is recommended for long sessions because it keeps connections balanced over time (HAProxy). For short HTTP requests, the connection count doesn't reflect CPU load, so least connections might not be as effective. Nginx's least_conn has similar caveats (Nginx).

If your cache is in-memory on the backend, you want to maximize the chance that a request for a cached object hits the server that has it. Neither algorithm does that. But least connections at least adapts to load, which can help if one server's cache is hot and thus busier—though it might route away from it, hurting hit rate.

Criterion 3: Operational Simplicity

Round robin is dead simple. It's the default, requires no configuration, and is predictable. Least connections is equally easy to enable—just one line in HAProxy or Nginx. But the operational simplicity of round robin masks a hidden cost: lower cache hit rates mean more origin requests, which can increase database load and latency.

Consider this: a 1% increase in cache hit rate can reduce origin traffic by 1%. If you're serving 1 million requests a day, that's 10,000 fewer requests hitting your database. That's a big deal for performance tuning.

But there's another angle: if you use a CDN, the CDN does the heavy lifting. Cloudflare's network spans 348 cities (Cloudflare), and it caches static content by default with specific TTLs (Cloudflare). Your origin load balancer's algorithm is irrelevant for edge traffic. So if you're using a CDN, you might not care about cache affinity at the origin—your CDN is the cache.

What I'd Actually Do

Here's my take: if you're running a cache behind your load balancer, neither least connections nor round robin is ideal. Instead, use a hashing algorithm like hash $request_uri consistent in Nginx (Nginx) or balance uri in HAProxy (HAProxy). This ensures the same URL always hits the same backend, maximizing cache affinity. But if you're forced to choose between the two, I'd go with least connections—it adapts to load and prevents any single server from becoming a bottleneck, which is better for overall latency even if hit rates are similar.

For most teams, though, the real win is to push caching to the edge with a CDN. CDNs can absorb 70% or more of origin requests (AWS), and they handle cache invalidation via single-file purge (Cloudflare). Your origin load balancer then only sees the misses, and round robin is perfectly fine for that.

So my concrete recommendation: if you have a CDN, use round robin for simplicity. If you don't, and you're stuck with these two, choose least connections, but also implement URL hashing if you can. And always monitor your cache hit rate—if it's below 80%, something's wrong.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• AWS - https://aws.amazon.com/caching/
• Nginx - https://nginx.org/en/docs/http/load_balancing.html
• HAProxy - https://docs.haproxy.org/3.4/configuration.html
• Cloudflare - https://developers.cloudflare.com/cache/concepts/default-cache-behavior/