Load Balancing Algorithms: A Field Guide for Your First Production Outage

Imagine you are a backend engineer at a mid-sized e-commerce company. It's Black Friday. Your marketing team just blasted a 40% off code to two million subscribers. Your origin servers are behind a load balancer. You chose round robin because it's the default and it works fine in staging. Now, three minutes into the sale, your checkout service is returning 503s for one out of every three users. The other two are fine. You have no idea why.

Here's what's happening: round robin distributes requests evenly across your servers, but it doesn't care whether a server is already drowning. If one of your three checkout servers is slower—maybe it has a cold JVM, maybe it's on a noisier neighbor—round robin keeps sending it the same share of traffic. That server's queue grows, latency spikes, and eventually it starts timing out. The load balancer marks it unhealthy and removes it. Now two servers handle the load. One of them is the slow one. The cycle repeats until you're down to one server, then zero. Your flash sale becomes a flash crash.

This is the single most common load balancing mistake I see in production: choosing an algorithm based on what's easy to configure rather than what matches your traffic shape. You can do better. Here's how.

Match the algorithm to the connection, not the server count

The system-design-primer lays out the classic options: round robin, weighted round robin, least connections, IP/source hash, and least response time. That's a good starting menu, but it doesn't tell you when to use which. The guidance from the same source is blunt and correct: use round robin for short, stateless connections; least connections or least response time for long connections; and hashing or cookies for session persistence.

Your checkout service is not stateless. It holds a cart session. If you use round robin, a user's second request might land on a different server that doesn't have their cart in memory. You'll get empty carts, duplicate orders, and a lot of support tickets. For that service, you need either a shared session store (Redis, for example) or a load balancer that supports session persistence. IP hash is the cheap fix: nginx has ip_hash, and HAProxy has the source algorithm. But be careful—IP hash breaks when users are behind a corporate NAT or a mobile carrier gateway. Thousands of users share one IP, and they all get pinned to the same server. That server becomes a hotspot while the others idle.

Better: use a shared session store and keep your load balancing stateless. Redis is an in-memory key-value cache with typical hit rates of 90% to 95% and 1 to 5 ms latency (AWS caching overview). That's fast enough to make session lookups invisible to your users, and it frees you to use the algorithm that actually fits your traffic.

Least connections is not a silver bullet—but it's close for long sessions

For your checkout service, least connections is usually the right answer. HAProxy's documentation recommends it for long sessions such as LDAP or SQL, and warns it's not well suited to short HTTP sessions. Your checkout requests take 200–800 ms. That's long enough that connection counts matter. With least connections, the load balancer sends each new request to the server with the fewest active connections. If one server is slow, its connection count climbs, and it naturally receives less new traffic. The system self-corrects.

nginx supports least_conn, and it also offers least-time, which is even more aggressive: it factors in response time, not just connection count. If you're on nginx, least_time is worth the extra configuration. But there's a catch: least_time requires nginx Plus. If you're on open-source nginx, use least_conn.

What about weighted round robin? That's for when your servers are not identical. Maybe you have two beefy instances and one older, smaller one. You can assign weights—say, 5, 5, and 1—so the small server gets a proportional share. But weights are static. They don't adapt to real-time load. If the small server gets slow, weighted round robin keeps sending it the same fraction of traffic. It's a blunt instrument. Use it only when you have a known, stable capacity difference and you can't use least connections.

The health check is part of the algorithm

You can pick the perfect algorithm and still get burned by bad health checks. nginx's passive health checks use max_fails (default 1) and fail_timeout. That means after one failed attempt, nginx marks the server as failed for the duration of fail_timeout. With the default max_fails of 1, a single transient error—a dropped packet, a slow query—can eject a healthy server from the pool. That's too aggressive for most production systems. Set max_fails to 3 or 5. Setting it to 0 disables health checks entirely, which is almost never what you want.

HAProxy gives you finer control. The inter parameter sets the interval between checks (default 2000 ms). The fall parameter marks a server dead after that many consecutive failures (default 3). The rise parameter marks it operational again after that many consecutive successes (default 2). Those defaults are reasonable, but you should tune them to your service. If your checkout service has a 5-second timeout on a bad day, a 2-second health check interval will mark it dead before it even has a chance to respond. Set inter to 5000 ms or more for slow services.

Quick tip: never set max_fails to 1 in production. One bad request should not remove a server from your pool. Set it to 3 or higher and watch your error rate drop.

When you need consistent hashing, use it deliberately

If you're caching at the load balancer layer—nginx's proxy_cache, for example—you care about cache hit ratio. If you use plain round robin or least connections, the same request will hit different cache nodes, and your hit ratio will tank. That's where consistent hashing comes in. nginx's hash directive with the consistent parameter uses the ketama method, which maps clients to servers by a hashed key and remaps only a few keys when you add or remove a server. That keeps your cache warm and your origin load low.

But consistent hashing has a dark side: it can create hotspots. If one key is extremely popular—say, a viral product page—it will always map to the same cache node. That node gets hammered while the others sit idle. You can mitigate this with a tiered cache, like Cloudflare's Tiered Cache, which divides data centers into lower and upper tiers. On a miss, the lower-tier data center asks an upper-tier, and only the upper-tier can contact the origin. That reduces origin load and spreads the hot-key problem across the hierarchy.

My recommendation: for most HTTP services, start with least connections. It's simple, it adapts to real-time load, and it handles both short and long sessions reasonably well. Add consistent hashing only when you're caching at the load balancer and you've measured a hit ratio problem. And whatever you do, configure your health checks before you go live. The algorithm is only half the story; the other half is knowing when to stop sending traffic to a server.

The single most important thing to remember: your load balancer is not a traffic cop—it's a feedback loop. Pick an algorithm that responds to server health, not just server count, and tune your health checks so they don't panic at the first sign of trouble.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• HAProxy (configuration manual) - https://docs.haproxy.org/3.4/configuration.html
• Nginx (HTTP load balancing) - https://nginx.org/en/docs/http/load_balancing.html
• AWS (caching overview) - https://aws.amazon.com/caching/
• Cloudflare (tiered cache) - https://developers.cloudflare.com/cache/how-to/tiered-cache/