Stop Defaulting to Round Robin: Pick a Load Balancer That Fits

The Misconception

Many engineers think round robin is the only load balancing algorithm they need. They set up a simple nginx config with the default method, and they're done. But that's a mistake. Round robin is a blunt instrument, and it can cause serious problems for applications with long-lived connections or uneven request sizes. I've seen it happen too many times: a service starts with a few servers and round robin, then traffic grows, and suddenly some servers are overloaded while others sit idle. The fix isn't just adding more servers; it's choosing the right algorithm.

Who This Is For

This guide is for developers and ops engineers who are responsible for a web service that's outgrown a single server. You probably have nginx or HAProxy in front of a few application servers, and you're starting to notice that some servers work harder than others. You're not building a global CDN; you just need to balance traffic across a handful of instances. I'm going to walk you through the four algorithms that matter most, when to use each one, and what can go wrong if you pick wrong.

Step 1: Know Your Traffic Profile

Before you touch any config file, think about what your traffic actually looks like. Are your requests short and stateless, like a typical REST API call that returns JSON in milliseconds? Or are they long-lived, like WebSocket connections or streaming responses? The answer points you to a different algorithm. For short, stateless requests, round robin is hard to beat—it's simple, fair, and works well when every request takes roughly the same time (system-design-primer). But if your requests are long or vary wildly in duration, round robin can send a new request to a server that's already busy holding a dozen WebSocket connections, while another server sits idle. That's a recipe for timeouts and poor user experience.

Step 2: Match the Algorithm to the Workload

Here's where I make a concrete recommendation. For short, stateless HTTP requests, stick with round robin—it's the default in nginx and HAProxy for a reason (HAProxy configuration manual). It's easy to reason about and gives each server an equal share of requests. But if you have long-lived sessions, like WebSocket or SQL connections, switch to least connections. HAProxy's leastconn algorithm sends new requests to the server with the fewest active connections, which is exactly what you want for long sessions (HAProxy configuration manual). For HTTP APIs, least connections can still work, but it's not ideal because a server might have a few slow requests and a bunch of fast ones, and connection count doesn't reflect that. For that, you might consider least response time, but that's more complex to implement and not always necessary.

Step 3: When to Use Hashing

Hashing, specifically IP hash or URL hash, is your friend when you need session persistence. If your application stores session state on a particular server, you need to ensure that all requests from a given user go to the same server. nginx's ip_hash does exactly that—it maps a client's IP to a server (Nginx HTTP load balancing). But there's a catch: if a server goes down, the hash table changes, and users may get rerouted to a different server, losing their session. That's a trade-off you have to accept. For caching, hash-based routing can dramatically improve cache hit rates because the same key always goes to the same cache. nginx's hash directive with the consistent parameter uses ketama consistent hashing, which minimizes remapping when servers are added or removed (Nginx upstream module). That's a smart choice for a cache layer, but for general load balancing, it's overkill.

Step 4: Don't Forget Weighted Round Robin

If your servers aren't identical—maybe one has twice the CPU and memory—use weighted round robin. Both nginx and HAProxy support it. You assign a weight to each server, and the algorithm sends requests proportionally. This is a simple way to get better utilization without changing the algorithm. For example, if you have two servers, one with 4 cores and one with 8, you could give them weights 1 and 2, and the 8-core server will get twice as many requests. It's not as precise as least connections, but it's a good middle ground.

What Can Go Wrong

Here's the warning: don't just pick an algorithm and forget about it. The worst thing you can do is set up least connections for a service that has short requests. I've seen this cause a skewed distribution because the connection count is a lagging indicator—by the time a request finishes, the count updates, but during the request, the server might be overloaded. Also, beware of the default health check settings in nginx. The max_fails directive defaults to 1, which means after a single failed attempt, a server is marked down for the fail_timeout period (Nginx HTTP load balancing). That can cause flapping if a server has a tiny blip. Set max_fails to a higher value, like 3, to avoid false positives. And remember, health checks are passive by default—they only probe a server when a client request is sent to it. That's fine for most cases, but if you need proactive checks, you'll have to configure them separately.

The Takeaway

Stop defaulting to round robin without thinking. Match the algorithm to your workload: round robin for short, stateless requests; least connections for long sessions; hashing for session persistence or caching; and weighted round robin for heterogeneous servers. Test your choice under realistic load and monitor connection counts and response times. A few minutes of planning can save you from a production incident later.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• HAProxy (configuration manual) - https://docs.haproxy.org/3.4/configuration.html
• Nginx (HTTP load balancing) - https://nginx.org/en/docs/http/load_balancing.html
• Nginx (upstream module) - https://nginx.org/en/docs/http/ngx_http_upstream_module.html