Why Round Robin Fails Your API and What to Use Instead

Everyone starts with round robin. It's the default in nginx and HAProxy, it's easy to explain, and for a toy service it works fine. But in production, round robin is frequently the wrong answer. It treats every request as identical, and every server as equally capable. That's rarely true. If you're running anything beyond a trivial API, you need to think harder.

The Myth of the Uniform Backend

The misconception is that load balancing is about spreading requests evenly. It's not. It's about keeping every user happy while your backend behaves unpredictably. Requests aren't uniform: some are cheap, some are expensive. Servers aren't uniform: some are newer, some are slower, some are already overloaded. Round robin ignores all of that. It's a fair scheduler for a world that doesn't exist. (system-design-primer)

Imagine You're Running a Checkout Service

Let's make this concrete. You're the lead engineer for an e-commerce site. You have three application servers behind an nginx load balancer, and you've just launched a flash sale. Traffic spikes, and suddenly some users are seeing timeouts. You check the dashboards: one server is at 95% CPU, the other two are at 30%. Round robin sent every third request to the hot server, regardless of whether that request was a lightweight product page or a heavy checkout with payment processing. That's the problem.

What the Algorithms Actually Do

Round robin is the default because it's simple and stateless. But the moment you have variable request costs or variable server capacity, you need something smarter. The two most useful alternatives are least connections and IP hash.

• Least connections (HAProxy calls it leastconn): sends each new request to the server with the fewest active connections. It's ideal for long-lived sessions like database connections or WebSockets, but for short HTTP requests the connection count can be a poor proxy for load. (HAProxy)
• IP hash (nginx ip_hash): hashes the client's IP address to pick a server, so the same client always hits the same server. That gives you session persistence without cookies, but it can lead to uneven distribution if a few heavy users dominate. (system-design-primer)

For your checkout service, least connections is a good first step. It will route new checkout requests away from the server that's already juggling many. But it's not perfect: a server with one expensive request might look idle while a server with ten cheap requests is actually busier. In that case, you want least response time, which measures actual latency and sends traffic to the fastest server. (system-design-primer)

Health Checks and the Danger of Blind Round Robin

Whatever algorithm you choose, you must pair it with health checks. Round robin doesn't know a server is down; it will keep sending requests to a dead server until a health check marks it failed. nginx's passive health checks work by counting failed attempts: after max_fails consecutive failures (default 1) within fail_timeout, the server is marked down. That's a blunt instrument. HAProxy's active checks are more tunable: you can set an interval (inter, default 2000 ms), a failure threshold (fall, default 3), and a recovery threshold (rise, default 2). (nginx, HAProxy)

For our checkout service, I'd configure active health checks to hit a lightweight endpoint like /healthz every 2 seconds, fail after 3 misses, and recover after 2 successes. That gives you a 6-second window to detect a downed server without cutting off traffic too eagerly.

When to Break the Rules: Sticky Sessions and Caching

Sometimes you don't want even distribution at all. If you're using an in-memory cache on each application server (a common pattern), you want the same user to hit the same server to maximize cache hits. That's where IP hash shines. But IP hash can cause a "hot server" problem: if one office or one NAT gateway generates a disproportionate share of traffic, that server gets overloaded. A better approach is to use a cookie-based sticky session, but that's a Layer 7 feature and requires careful handling of cookie expiration.

Alternatively, you can use a distributed cache like Redis and avoid stickiness altogether. Redis is an in-memory key-value store with typical hit rates of 90-95% and 1-5 ms latency, and it can reduce database CPU by 70-90%. (AWS) With Redis, any server can serve any user, so you can use least connections without worrying about cache locality. That's the architecture I'd recommend for a growing service: stateless app servers, a Redis cache layer, and a load balancer that picks the least-loaded server.

Quick tip: If you're using nginx, try least_conn instead of the default round robin. For most web APIs, it's a strict improvement and costs almost nothing to configure.

Making the Call

There's no one-size-fits-all algorithm. For short, stateless requests, round robin is fine. For long-lived connections, use least connections. For session persistence, use IP hash or cookies. For variable request costs, use least response time. (system-design-primer) The key is to measure. Look at your request latency, your server CPU, and your connection counts. If you see one server pegged while others idle, you're using the wrong algorithm.

In our checkout scenario, I'd start with least connections, add active health checks, and move session state to Redis. That combination will handle a flash sale without melting one server. Round robin might be the default, but defaults are for people who haven't watched their backend fall over yet.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• HAProxy - https://docs.haproxy.org/3.4/configuration.html
• nginx - https://nginx.org/en/docs/http/load_balancing.html
• AWS - https://aws.amazon.com/caching/