Cache or Load Balancer: Which Knob to Turn When Things Slow Down?

Cache or load balancer? If you're staring at a slow stack, you probably want to know which knob to turn first. I've been in this situation more times than I can count, and I've got a strong opinion: tune the cache first. But hear me out—it's not a one-size-fits-all answer.

Why This Question Even Comes Up

Think about it. Every request that hits your origin server burns CPU, runs database queries, and takes time. A cache's job is to stop those requests before they ever reach your origin. A load balancer, on the other hand, spreads the requests that do get through across multiple servers. They do different things, but they're often lumped together when people talk about scaling. That's a mistake.

The Contenders: HTTP Cache vs. Load Balancer

I'm comparing the two most common ways to speed up a web service: an HTTP cache (like a CDN or nginx's proxy cache) and a load balancer (like HAProxy or nginx's upstream). I'm not pitting specific products against each other; I'm looking at the architectural layers. But to keep it real, I'll talk about them as they exist in practice.

Here are the criteria I'll use:

• Where it sits: How close is it to the user vs. the origin?
• Impact on origin load: Does it actually reduce the number of requests your origin sees?
• What a miss costs: When it doesn't work as planned, what's the damage?
• Risk of messing it up: How easy is it to configure wrong and cause a disaster?

Where Each One Lives

In a typical setup, you have layers: browser cache, CDN edge, load balancer, app cache, distributed cache (like Redis), then the database. So the CDN cache is the first thing users hit. The load balancer sits behind that, usually. This placement alone tells you a lot: the cache is the first line of defense. If it does its job, the load balancer might not even see the request.

How Much They Reduce Origin Load

CDNs can absorb 70% or more of origin requests, and hit ratios often hit 80-95%. That's massive. A load balancer doesn't reduce the total number of requests—it just spreads them across servers. So if you're trying to lower the absolute load on your origin, the cache wins hands down.

When Things Go Wrong: The Cost of a Miss

What happens when a cache misses? The request goes to the origin. If you have a load balancer, it routes to one server. The cost of a cache miss is one extra trip to the origin. If you've set up your cache right, that miss rate should be low. But a load balancer misrouting can be nastier: it might send a request to an overloaded server, or if you've botched sticky sessions, you break user state. A cache miss is just a wasted trip; a load balancer error can cause actual downtime or data corruption.

Here's a concrete example. I once had a client whose API responses were cached for an hour. An admin updated a product, and users kept seeing the old version for up to an hour. We fixed it by purging the cache by URL, but that required knowing the exact URL. If we'd set Cache-Control: no-cache instead, every request would revalidate, which defeats the purpose. So you need to find that sweet spot.

The Danger Zone: Misconfiguration Risk

Let me get on my soapbox for a second. Caching is easy to screw up. You can accidentally cache dynamic content, serve stale data, or break your cache keys. Load balancer config is also tricky—you've got to pick the right algorithm and health checks. But in my experience, cache misconfigurations cause more spectacular failures. Why? Because the cache sits closer to the user. If you cache a page that includes a user's personal info, you're serving that to everyone. That's a privacy nightmare.

For instance, I remember a project where someone set Cache-Control: public on a URL that returned user-specific data. Suddenly, everyone was seeing the first person's account details. That's the kind of bug that makes you double-check every header before you ship.

So Which One Wins? It Depends, But Here's My Take

If you're a small to medium service with a single origin server, tuning your cache first gives you the most benefit. A CDN or an nginx reverse proxy can cut origin load by 70% or more, and you can often get it working with just a few HTTP headers. A load balancer won't help if you only have one server. So start with the cache.

But if you've got multiple origin servers and you're already serving real traffic, then the load balancer is critical. You need to pick the right algorithm: round robin for short, stateless connections; least connections for long sessions; and IP hash for session persistence. And don't forget health checks—otherwise you'll route traffic to a dead server and wonder why everything's failing.

Real-World Numbers and a Quick Tip

Let me give you a quick tip that's saved my bacon more than once. If you're using nginx as a reverse proxy cache, remember that proxy_cache_path needs a keys_zone to store cache keys in memory. One megabyte of keys_zone holds about 4000 keys. That might sound like a lot, but if you have thousands of products, that fills up fast. I once set it too small and saw cache hit rates tank because keys were being evicted constantly. Bump it up to 16MB or so, and you'll see a huge difference.

Another thing: if you're using Cloudflare, you can purge a single file by URL. That's handy when you need to invalidate a specific cached resource without nuking the whole cache. Just remember that purge takes a few seconds to propagate across the edge.

Wrapping Up

So here's my advice, plain and simple: tune your cache first. It's your first line of defense and reduces load at the source. But don't ignore the load balancer—it's essential for distributing traffic and keeping things available when you have multiple origins. The right order is: set up your CDN or HTTP cache with proper headers and revalidation, then configure your load balancer with the right algorithm and health checks. If you're on a single server, skip the load balancer for now. Start with the cache. You'll see the biggest improvement with the least effort.

Sources

• AWS - https://aws.amazon.com/caching/
• Cloudflare - https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-single-file/
• Nginx - https://nginx.org/en/docs/http/ngx_http_proxy_module.html
• RFC 5861 - https://www.rfc-editor.org/rfc/rfc5861.txt
• system-design-primer - https://github.com/donnemartin/system-design-primer