Stop Round-Robin: Hash Your Way to a Smarter Load Balancer

Who This Is For

You're running a web service that has outgrown a single server. You've got a load balancer in front of two, five, or twenty instances, and you've noticed something annoying: your cache hit ratio is in the toilet, or users keep getting logged out, or your database is doing way more work than it should. You've read the default advice—"just use round-robin, it's fine"—and you suspect it's not fine. You're right. This is for you.

The Contrarian Take: Round-Robin Is a Crutch

Most load balancers default to round-robin, and most tutorials tell you to leave it that way. They'll say it's simple, fair, and works for stateless services. That's true if your servers are truly stateless and your cache is external. But the moment you add a caching layer—and you should—round-robin becomes an active liability. It scatters requests for the same resource across every backend, so each backend's cache is a tiny, useless fragment of the whole. You end up with a cache hit ratio that hovers around 30% instead of 90%.

Here's the blunt truth: if you're using round-robin with any local cache, you're burning money. The fix is consistent hashing. It sends the same key—a user ID, a URL, a session cookie—to the same backend every time, turning each server's cache into a genuinely useful partition. The nginx documentation has called this out since version 1.7.2, when it added the consistent parameter to its hash directive specifically to improve cache hit ratios (Nginx upstream module).

Step 1: Admit You Have a State Problem

Before you touch your config, inventory your state. Is there any reason a request for a given user must always hit the same server? Sessions in memory? A local cache? A WebSocket connection? If yes, you need sticky sessions or a hashing algorithm. If your app is truly stateless and you're using a shared cache like Redis, you can skip ahead—but even then, consistent hashing can reduce origin load by keeping repetitive requests on one box.

The general guidance from the system-design primer is clear: use round-robin for short, stateless connections; use hashing or cookies when you need session persistence (system-design-primer). Don't pretend you're stateless when you're not.

Step 2: Pick Your Hash Flavor

There are two ways to hash: plain old IP hash, and consistent hashing. IP hash is the default in many load balancers, but it's brittle—when you add or remove a server, almost every key gets remapped, and your cache goes cold. Consistent hashing (ketama in nginx) only remaps a few keys, so your cache survives scaling events (Nginx upstream module).

If you're using nginx, the directive is hash $request_uri consistent;. If you're on HAProxy, you'd use balance source or balance uri—but be aware that HAProxy's source hash is not consistent by default; you need the consistent option if you want that property (HAProxy configuration manual).

Step 3: Configure the Hash Key

Your hash key determines what gets pinned. For a content site, hash on the URL: hash $request_uri consistent;. For an API, hash on the user ID from a header or cookie. For WebSockets, hash on the source IP. The key must be present in every request, or you'll get a hash of an empty string, which is worse than round-robin.

In nginx, you can use any variable that's available in the HTTP request context (Nginx HTTP load balancing). In HAProxy, you'd use balance hdr(User-ID) or balance url_param if you have a query parameter (HAProxy configuration manual). Test with a few representative keys to make sure they're not all landing on the same server.

Step 4: Add a Cache Layer

Hashing alone won't save you if you don't have anything to cache. The most common pattern is cache-aside: check the cache, on a miss fetch from the database, then write back (AWS caching overview). Redis is the usual choice—it's in-memory, with typical hit rates of 90–95% and 1–5 ms latency (AWS caching overview). But if you're already hashing to a backend, you can also use nginx's built-in proxy cache, which stores responses on disk and uses the cache key to serve them.

Nginx's proxy cache is easy to set up: proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=one:10m; and then proxy_cache_valid 200 302 10m; (Nginx proxy module). One megabyte of keys_zone stores about 4000 keys, so size it accordingly (Nginx proxy module). This isn't a replacement for Redis, but for static assets or API responses, it can offload a lot of origin traffic.

Step 5: Test, Then Test Again

After you flip the switch, watch your cache hit ratio. If you're using a CDN, you can expect hit ratios of 80–95% (AWS caching overview). If you're using nginx's proxy cache, you can check the X-Cache-Status header to see hits and misses. If you're using Redis, monitor the hit rate—it should jump dramatically.

Also test what happens when a backend goes down. With consistent hashing, requests for that server's keys will be remapped to the next server in the ring, which is fine, but you'll see a temporary cache cold start. Make sure your health checks are aggressive enough. In nginx, max_fails=1 and fail_timeout=10s is a reasonable starting point (Nginx HTTP load balancing). In HAProxy, inter 2000ms fall 3 rise 2 is the default and works well (HAProxy configuration manual).

What Can Go Wrong

The classic mistake is hashing on the wrong key. If you hash on the URL but your CDN strips query parameters, you'll get cache misses. If you hash on the IP, users behind a NAT or a mobile network will flip between servers as their IP changes—your cache will be useless. And if you use plain IP hash without consistent hashing, scaling your fleet will cause a stampede of cache misses that can take down your origin.

Another gotcha: don't mix hashing with least connections. They're opposites. Hashes are for locality; least connections are for even load. If you need both, you'll have to implement a two-tier scheme, but that's rare. Most of the time, hashing is the right call.

Sources

• Nginx HTTP Load Balancing - https://nginx.org/en/docs/http/load_balancing.html
• Nginx Upstream Module - https://nginx.org/en/docs/http/ngx_http_upstream_module.html
• HAProxy Configuration Manual - https://docs.haproxy.org/3.4/configuration.html
• AWS Caching Overview - https://aws.amazon.com/caching/
• System Design Primer - https://github.com/donnemartin/system-design-primer