Choosing the Right Load Balancing Algorithm: A Practitioner's Guide

Who This Is For

If you're running a web service that's outgrown a single server, you've probably stared at your load balancer's configuration screen wondering which algorithm to pick. This guide is for the working engineer who needs to make a practical choice today, not a survey of every academic option. We'll walk through the decisions in the order you'll actually face them.

Start With Your Traffic Pattern

The first question isn't "which algorithm?"—it's "what do my requests look like?" If your requests are short and stateless—think API calls that return JSON in milliseconds—round robin is a solid default. It's the default in both nginx and HAProxy for good reason: it spreads load evenly with minimal overhead (Nginx, HAProxy). But if your requests hold connections open for a long time—think WebSocket connections or streaming video—round robin will pile requests onto the same servers while others sit idle. In that case, least connections is your friend. HAProxy's documentation explicitly recommends leastconn for long sessions like LDAP or SQL, and notes it's not well-suited to short HTTP sessions (HAProxy).

Consider Your Servers' Capacities

Not all servers are created equal. If you've got a mix of instance sizes, you need weighted round robin. This lets you send twice as much traffic to the bigger box. Both nginx and HAProxy support this. But remember, weights are static—if your traffic pattern shifts, you'll need to adjust them manually. For most of us, it's better to keep servers homogeneous and use round robin, or move to a dynamic algorithm like least response time if your latency varies. Least response time routes to the server with the fastest current response, which adapts automatically. It's not in every balancer's toolbox, but it's worth checking if your latency matters more than simplicity.

When You Need Sticky Sessions

If your application stores session state in memory, you need the same user to hit the same server every time. That's where hashing comes in. IP hash (or source hash in HAProxy) maps a client's IP to a server deterministically. But beware: if a server goes down, all the clients hashed to it get remapped, which can cause a thundering herd. A better approach is to use a cookie-based sticky session if your load balancer supports it—nginx and HAProxy both do—so you can tie a session to a specific server without relying on IP stability. For caching layers, consistent hashing is the magic word. The nginx `hash ... consistent` directive uses ketama consistent hashing, which means adding or removing a server remaps only a few keys, preserving your cache hit ratio (Nginx).

What Can Go Wrong: The Health Check Trap

No algorithm saves you from a dead server. If your load balancer keeps sending requests to a failed backend, users get 502s no matter how clever your routing is. Passive health checks—where the balancer marks a server down after a few failed responses—are better than nothing, but they only catch failures after they happen. For example, nginx's `max_fails` defaults to 1, meaning one failure during `fail_timeout` marks the server failed (Nginx). That might be too aggressive for a momentary blip. HAProxy's active health checks are more robust: you can configure `inter` (default 2000 ms), `fall` (default 3), and `rise` (default 2) to precisely control how quickly a server is marked down and up (HAProxy). My advice: use active health checks whenever possible, and tune the thresholds to match your application's startup time. A server that takes 30 seconds to boot will flap if your `rise` is set to 2.

Layer 4 vs Layer 7: Where the Algorithm Lives

Your choice of algorithm also depends on which layer your load balancer operates at. Layer 4 balancers route on IP and port, making them fast and protocol-agnostic. Layer 7 balancers can inspect HTTP headers, URLs, and cookies, enabling smarter routing like path-based or header-based rules. Nginx works at Layer 7, while HAProxy can do both (system-design-primer). For most modern web services, Layer 7 is worth the slight overhead because it lets you do things like route `/api` to one backend and `/static` to another. But if you're moving massive volumes of traffic and need raw throughput, Layer 4 with a simple round robin might be all you need.

What I'd Actually Do

For 90% of web services, here's my recommendation: start with round robin if your requests are short and your servers are homogeneous. If you have long-lived connections, switch to least connections. If you need session persistence, use a cookie-based sticky session (not IP hash, unless you have a good reason). And for any caching layer—like Redis or a CDN—use consistent hashing to keep your hit ratio high. Don't overthink it. The best algorithm is the one you understand and can debug at 3 AM. As your traffic grows, revisit your choice, but don't let analysis paralysis stop you from shipping.

One quick tip: always set a health check. Even a simple ping on the TCP port is better than none.

Sources

• system-design-primer - https://github.com/donnemartin/system-design-primer
• Nginx (HTTP load balancing) - https://nginx.org/en/docs/http/load_balancing.html
• HAProxy (configuration manual) - https://docs.haproxy.org/3.4/configuration.html
• Nginx (upstream module) - https://nginx.org/en/docs/http/ngx_http_upstream_module.html