Stop Round-Robin Shrugging: Match Load Balancing to Your Traffic

You've got a fleet of servers, and you want to spread the load without breaking a sweat. Which load-balancing algorithm should you pick? If you're like most engineers, you default to round robin because it's simple—and that's exactly the trap. I'll say it plainly: round robin is the wrong choice for most production traffic, and you're hurting your latency and wasting capacity by using it blindly. The right algorithm depends on your traffic pattern, and I'm going to tell you how to think about it.

The Default Is a Crutch

Look, I get it. Round robin is the default in Nginx, and it's the first thing you learn. It just cycles through your backend servers in order, giving each an equal share. That's perfect for short, stateless connections—think simple API calls that finish in milliseconds. The system-design-primer explicitly recommends round robin for that scenario. But here's the rub: most of us aren't running a homogenous fleet of identical, short-lived requests. You have some endpoints that are heavy, some clients that hog connections, and some servers that are slower than others. Round robin ignores all of that. It's like a traffic cop who lets every car through one at a time, regardless of whether one lane is backed up. You're going to end up with a queue.

Match the Algorithm to Your Reality

Here's my blunt advice: stop using round robin for anything that isn't short and stateless. If your connections are long-lived—think WebSockets, streaming, or database connections—you need least connections or least response time. The system-design-primer is clear: use those for long connections. Why? Because round robin will happily send a new WebSocket to a server that's already juggling a thousand, while another server sits idle. Least connections actively looks at the current load and sends new traffic to the least busy server. That's just common sense. And if you're dealing with heterogeneous servers—some beefy, some puny—you need weighted round robin, which lets you give more traffic to the big guys. But even that's static. The real winner for variable workloads is least response time, which sends traffic to the server that's actually responding fastest. That's not just a nice-to-have; it's a direct proxy for user experience.

What About Sessions and Sticky Bits?

Now, you might be thinking, "But I need session persistence!" And you're right—if your app stores state in memory, you need the same server to handle a user's requests. That's where hashing or cookies come in. The system-design-primer suggests using IP/source hash or cookies for session persistence. So, in that case, round robin is out, and you'd use a hash based on the client's IP to stick them to one server. That's fine, but it has a tradeoff: if that server dies, you lose the session (unless you've got a shared cache). Which brings me to the elephant in the room: caching. A lot of people think load balancing is the whole story, but the real performance win is often in the cache layer. The AWS caching overview points out that a typical architecture layers caches from the browser out to a CDN, then a load balancer cache, an app cache, and a distributed cache like Redis. If you're relying on sticky sessions to hold state, you're missing the point. You should be using a distributed cache like Redis, which can handle 90-95% hit rates with 1-5 ms latency, and it can cut your database CPU by 70-90% (AWS). That's a massive win. So, yes, use hashing for sticky sessions if you must, but design your app so that state lives in the cache, not in the server's memory.

The Counterargument: Simplicity Wins

I can already hear the counterargument: "But round robin is simple, and it works fine for my scale." I've been there. And it's true that if you have a handful of servers and low traffic, round robin is perfectly fine. But here's the thing: simple isn't the same as correct. The cost of moving to least connections or least response time is tiny—it's just a config change in HAProxy or Nginx. HAProxy supports leastconn and source algorithms, and Nginx has least_conn and ip_hash. You're not rewriting your app. So why not use the right tool? The only time I'd stick with round robin is if your requests are truly short and stateless, and your servers are identical. And even then, I'd ask: are you sure? Because a CDN in front of your origin can already absorb 70% or more of requests (AWS), so the load balancer is only seeing a fraction of the traffic. In that case, round robin might be fine. But if you're seeing uneven load or latency spikes, don't blame the servers—blame your algorithm.

My Recommendation

Here's what I'd do, and I'm not shy about it: use least response time as your default for HTTP traffic. It's the most adaptive and gives users the fastest server. If your load balancer doesn't support it, use least connections. For anything that needs session persistence, use a hash—but only if you've got a shared cache. And for the love of all that is holy, don't forget the cache layer. Redis is not a luxury; it's a necessity at any real scale. The bottom line: the algorithm is a means to an end—the end is low latency and high availability. Don't let round robin's simplicity lull you into mediocrity.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• AWS Caching Overview - https://aws.amazon.com/caching/