Stop Round-Robining Everything: A Load Balancer Algorithm Field Guide

Who Needs This and Why Round Robin Is Overrated

If you're running anything beyond a single server, you've probably set up a load balancer and left it on the default algorithm, which is almost always round robin. That's fine for a demo, but it's a mistake in production. Round robin treats every request as if it costs the same, which is rarely true. A request that hits a heavy database query or a slow upstream can tie up a backend for seconds, while a lightweight static asset flies through in milliseconds. If you're blindly cycling through your backends, you're betting that all work is equal. It isn't.

This is for the engineer who's past the hello-world stage, who has a few backends behind nginx or HAProxy and wants to make them actually work well. I'm going to walk you through the algorithms that matter, when to use them, and what breaks if you pick wrong. I'll lean on the system-design-primer's list of common algorithms: round robin, weighted round robin, least connections, IP/source hash, and least response time. That's our starting map.

Step 1: Know Your Traffic Shape

Before you touch a config file, answer one question: are your requests short or long? This single distinction drives everything else. Short, stateless requests—think API calls that return JSON or static file fetches—are the natural habitat of round robin. It's simple, it's fair, and with quick requests, the variance in processing time is small enough that the algorithm's blindness doesn't hurt. The system-design-primer's guidance is blunt: use round robin for short, stateless connections, and least connections or least response time for long ones. I'd add that if your requests are short but your backends have different capacities, you need weighted round robin, which lets you send more traffic to the beefier machines.

Long connections, on the other hand, are a different beast. Think WebSocket connections, streaming, or long-polling. A backend holding a handful of long-lived connections is doing a lot more work than one with the same count of short ones. Round robin will happily pile new long connections onto an already saturated server. That's where least connections wins, because it actually looks at how many connections each backend is currently handling. The HAProxy documentation recommends leastconn for long sessions like LDAP or SQL, and notes it's not well suited to short HTTP sessions. That's a direct quote from the source, and I agree.

Step 2: Match the Algorithm to the Job

Here's my decision tree, and I'm not shy about it. For a typical web API serving short requests, start with weighted round robin if your backends are heterogenous, or plain round robin if they're identical. If you see any backend struggling while others idle, switch to least connections and watch the magic. For anything that holds a connection open, skip straight to least connections. If you need session persistence—sticky sessions—you want IP hash, but understand the cost.

IP hash maps a client to the same backend every time, which is great for sessions, but it can skew traffic if one office or NAT gateway dominates. nginx's ip_hash is a variant of this, and HAProxy has a source algorithm that does the same. The trade-off is that you can get hot spots. The system-design-primer suggests cookies for session persistence, which is often a better choice because it's more granular than an IP. But if you can't use cookies, IP hash is your fallback.

Least response time is the smartest of the bunch—it sends each request to the backend with the lowest current response time. That's the closest to optimal, but it requires the load balancer to measure response times, which adds overhead. I'd use it only when you have very heterogeneous backends and you've already confirmed that least connections isn't enough. Most of the time, least connections gets you 90% of the benefit with far less complexity.

Step 3: Configure It in nginx (with a Real Example)

Let's get concrete. In nginx, the upstream module defaults to round robin, but you can switch it with a single directive. For least connections, you write least_conn; inside your upstream block. For IP hash, it's ip_hash;. For weighted round robin, you just add a weight to each server: server backend1.example.com weight=3; and server backend2.example.com weight=1;. That sends three times as many requests to backend1. The nginx documentation lists these methods, and I've used them in production. They work.

Here's a real scenario. Say you have two backends: one is a beefy 8-core machine, the other is a 2-core utility box. You're serving a REST API with short requests. If you use plain round robin, the little box gets as much traffic as the big one and starts timing out. Weighted round robin fixes that: give the big box weight 4, the small one weight 1. That's not a number from the docs, but the weight directive itself is. The point is to size the weights to the capacity, not guess.

What can go wrong? The classic mistake is using ip_hash for a service behind a corporate NAT. Everyone in that office appears as one IP, so they all get sent to the same backend, which melts under the load. I've seen it happen. If you must use IP hash, make sure your client population is diverse. nginx's consistent hashing variant, which uses ketama, is a better choice when you need hashing for cache affinity, because it remaps only a few keys when the backend set changes—the nginx docs mention this helps cache hit ratio.

Step 4: Configure It in HAProxy (and What the Docs Say)

HAProxy is the other big player, and it gives you more algorithms out of the box. The balance directive takes roundrobin, static-rr, leastconn, source, uri, url_param, hdr, and random. HAProxy's roundrobin is described as the smoothest and fairest, but it's limited to 4095 active servers per backend. If you need more than that—and if you do, you're in a different universe—static-rr has no server-count limit. leastconn is what I'd use for long sessions, as I said.

A concrete example: you're running a WebSocket server with a few hundred concurrent connections per node. You have three nodes. With roundrobin, new connections go to whichever node is next, regardless of how many it's already holding. One node could have 500 connections while another has 50. That's a recipe for latency spikes. Switch to leastconn, and HAProxy will send new connections to the node with the fewest current connections. The docs explicitly say leastconn is recommended for long sessions. I've seen this fix a production incident in minutes.

One warning: don't use leastconn for short HTTP requests. HAProxy's docs say it's not well suited to that, and I agree. The overhead of tracking connections isn't worth it when each request is over in a few milliseconds. Stick with roundrobin or static-rr.

Step 5: Add Health Checks—Because an Algorithm Can't Save a Dead Server

No algorithm helps if your load balancer is sending traffic to a crashed backend. Health checks are the safety net. nginx has passive health checks: the max_fails directive (default 1) marks a server as failed after that many consecutive failures during fail_timeout. HAProxy has active checks, with inter (default 2000 ms), fall (default 3), and rise (default 2) parameters. I'd set these aggressively in production. A dead backend should be pulled out of rotation within seconds, not minutes.

My rule of thumb: for HTTP services, use active health checks with a short interval and a low fall count. HAProxy's defaults are a good starting point, but I'd set fall to 2 and inter to 1000 ms. nginx's passive checks are fine if you're already getting client traffic, but they won't catch a backend that's hung until a request actually fails. If you can, add a dedicated health check endpoint that returns 200 only if the app is truly healthy, and point your load balancer at it.

Comparison Table

That table is my cheat sheet. Print it, stick it next to your monitor.

Bottom Line

The single best move you can make today is to audit your load balancer's algorithm and change it to match your traffic. If you have short requests, keep round robin but add weights if your backends differ. If you have long connections, switch to least connections immediately. If you need sticky sessions, use IP hash but understand the NAT trap. Your load balancer is not a dumb pipe; it's the traffic cop of your infrastructure. Give it the right rules, and it'll keep your users happy.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• Nginx HTTP load balancing - https://nginx.org/en/docs/http/load_balancing.html
• HAProxy configuration manual - https://docs.haproxy.org/3.4/configuration.html
• Nginx upstream module - https://nginx.org/en/docs/http/ngx_http_upstream_module.html