Caching Case Study: Fix Your Hit Rate Before Adding Load Balancers

Who This Is For And Why Your Cache Is Probably Broken

You think your problem is load. It isn’t. It’s cache misses. I’ve seen too many teams buy another load balancer or spin up more app servers when their origin is drowning in requests that a cache could have absorbed. A CDN can take 70% or more of your origin requests off your plate (AWS). If yours isn’t doing that, you don’t need more hardware. You need to fix your caching.

This walkthrough is for engineers running a web service that’s getting slow under traffic. You’ve probably got a load balancer already, or you’re about to add one. Here’s the thing: a load balancer spreads load across servers, but it does nothing to reduce the total number of requests hitting your stack. Caching does. So before you touch your nginx upstream block, let’s make sure your cache is actually working.

Step 1: Look At Your Cache Layers And Find The Leak

Start by mapping where your requests are being served from. The layers, in order, are: browser cache, CDN edge, load balancer/proxy cache, application cache, distributed cache, then database (AWS). Most teams only think about the CDN or Redis, but they ignore the browser and the proxy cache sitting right in front of their origin. That’s a leak.

Check your CDN analytics. If your hit ratio is below 80%, you’re leaving requests on the table — good CDNs run 80-95% (AWS). I’d want to see north of 90% for static assets. If you’re not there, move to step 2. But before you do, one warning: don’t just crank up cache TTLs. That’s how you end up serving stale content and then doing emergency purges at 2 AM. Slow down. Fix the headers properly.

Step 2: Set Cache-Control Headers Like You Mean It

The single biggest mistake I see is not setting Cache-Control on responses. Your origin is probably sending no cache headers, so CDNs and browsers guess — and they guess badly. The default Cloudflare behavior, when you send no cache-control or expires header, is to cache a 200 response for 120 minutes and a 404 for 3 minutes (Cloudflare). That’s fine for some things, but you have no control.

Here’s what I recommend: For static assets like images, CSS, and JavaScript, set Cache-Control: max-age=604800, immutable (MDN). That’s seven days. For HTML pages, set Cache-Control: no-cache (MDN). That tells caches they can store the response but must revalidate with the origin before reusing it. That sounds like a round-trip, but it’s cheap if you also send a validator. And for personalized content, set Cache-Control: private or no-store (MDN). Private means only the browser can cache it; no-store means don’t cache it at all.

And don’t forget the Vary header. If you serve different content based on Accept-Encoding or cookies, you must send Vary: Accept-Encoding, or your CDN will mix gzipped and plain responses (MDN). I’ve seen that cause weird bugs that are hard to trace.

Step 3: Add Validators And Revalidate, Don’t Just Expire

Once you’ve got Cache-Control set, make sure you’re sending validators. ETag is the gold standard — it’s a token that changes when the content changes (MDN). Last-Modified is a fallback, but it’s less accurate (MDN). When a cache has a stale response, it sends a conditional request with If-None-Match. If the ETag matches, the origin sends a 304 Not Modified and saves bandwidth.

But what if you want to serve stale content while you revalidate in the background? That’s what stale-while-revalidate is for. RFC 5861 defines it as a Cache-Control extension that allows a cache to serve a response that’s stale for up to X seconds while it revalidates in the background (RFC 5861). This is a killer for slow origin responses. Set Cache-Control: max-age=60, stale-while-revalidate=600 on your HTML pages. Users get instant responses, and your origin gets a trickle of revalidation requests instead of a stampede.

I’d also add stale-if-error, which lets a cache serve a stale response if the origin returns a 500 or has a DNS failure (RFC 5861). That’s a safety net that keeps your site up during origin hiccups. It’s not a substitute for monitoring, but it’s a cheap insurance policy.

Step 4: Use A Distributed Cache For Dynamic Data — And Pick The Right Eviction Policy

If you’re still hammering your database for things like user sessions or product details, you need a distributed cache in front of it. Redis is the obvious choice. It has typical hit rates of 90-95% and 1-5 ms latency, and it often reduces database CPU by 70-90% (AWS). That’s the kind of number that makes you look good in a status meeting.

But Redis is not a database. It’s a cache. The default eviction policy is noeviction, which means Redis returns an error when memory is full and you try to set a new key (Redis). That’s a recipe for crashed writes. You want allkeys-lru, which evicts the least recently used keys across the whole keyspace (AWS). Redis also offers allkeys-lfu, which uses a least frequently used strategy and can give better hit rates in some workloads (Redis). I’d start with allkeys-lru and switch to lfu only if your access pattern is skewed toward a hot set of keys that you want to keep.

Now, a concrete example. Say you’re caching product pages for an e-commerce site. You set keys with SETEX, giving each product a TTL of 3600 seconds (Redis SETEX). Your access pattern is that a few popular products get hammered, while long-tail products get accessed once a day. With allkeys-lru, those popular products will stay in cache because they’re accessed frequently, and the long-tail ones will get evicted. That’s exactly what you want. But if you’re doing a flash sale where every product gets equal attention for a short window, LRU might evict a product that’s about to get a burst. In that case, consider volatile-ttl, which evicts keys with the shortest remaining TTL (Redis). That way, you evict the ones that are about to expire anyway.

Step 5: Put A Load Balancer In Front, But Use It For The Right Reasons

Now you can add a load balancer. But choose the algorithm based on your traffic pattern. Round robin is fine for short, stateless connections (system-design-primer). If you have long-lived connections like WebSockets or SQL, use least connections. And if you need session persistence, use IP hash or a cookie-based method (system-design-primer).

Nginx is a solid Layer 7 load balancer that supports round robin, least_conn, ip_hash, and more (Nginx). If you’re using a cache like Redis behind nginx, you want to use the hash directive with the consistent parameter. That uses ketama consistent hashing, so when you add or remove a backend, only a few keys are remapped (Nginx upstream module). That keeps your cache hit ratio high. Without consistent hashing, adding a server would remap a large fraction of your keys, and your Redis would suddenly miss on everything.

But don’t ignore your proxy cache. Nginx can cache responses with proxy_cache_path and proxy_cache_valid. Set up a cache zone with keys_zone=one:10m, which can hold about 40,000 keys (Nginx proxy module). And use proxy_cache_lock to prevent a thundering herd when a popular resource expires — only one request will populate the cache while others wait (Nginx proxy module). This is the kind of thing that saves you from a cache stampede that takes down your origin.

Step 6: What Can Go Wrong — Cache Invalidation And Purge

Here’s the warning. Caching is easy until you need to change something. If you update a product image and your CDN still serves the old one for 7 days, users see broken images. You need a plan for invalidation.

HTTP gives you one mechanism: when you send an unsafe request like PUT, POST, or DELETE, caches must invalidate the stored response for that URI (RFC 9111). But that only works if your origin sends the right headers and caches honor them. In practice, you’ll also need a manual purge button. Cloudflare lets you purge by single-file across all its data centers instantly (Cloudflare). Use that sparingly — purging a CDN cache causes a thundering herd back to your origin.

I’ve seen teams get burned because they cached HTML for an hour and then a typo went live. They had to purge, but they didn’t have a process. So, here’s my recommendation: cache static assets aggressively with long TTLs, but cache HTML for a short time or use no-cache with revalidation. And always send a Cache-Control header that matches the content’s sensitivity to change.

The One Thing To Remember

Your cache is a system, not a switch. Set headers deliberately, choose eviction policies that match your access pattern, and have a purge plan. Do that before you buy another load balancer, and your origin will thank you.

Sources

• AWS caching overview - https://aws.amazon.com/caching/
• MDN Cache-Control - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• MDN ETag - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
• RFC 5861 (stale-while-revalidate) - https://www.rfc-editor.org/rfc/rfc5861.txt
• Redis eviction - https://redis.io/docs/latest/develop/reference/eviction/
• Nginx upstream module - https://nginx.org/en/docs/http/ngx_http_upstream_module.html