Why Your Cache Hit Rate Is Lying to You (and How to Fix It)

Conventional wisdom says a cache hit rate of 90% to 95% is the mark of a well-tuned system (AWS). But that number, by itself, can be a trap. A high hit rate tells you that your cache is doing its job, but it doesn't tell you whether your cache is doing the right job. The real goal isn't a high hit rate; it's reducing the load on your origin and improving user-perceived latency. Sometimes, the best way to achieve that is to deliberately lower your hit rate.

Let's ask a specific question: When should you ignore the siren song of a 95% hit rate and instead tune your caching to serve slightly stale data? The answer is more nuanced than 'maximize cache hits.' It's about understanding the trade-offs between freshness, origin load, and user experience.

The Hit Rate Fallacy

A 95% hit rate sounds impressive, but if the 5% of misses are for your most popular, dynamic content, those misses can hammer your origin. A CDN can absorb 70% or more of origin requests, but that still leaves 30% hitting your servers (AWS). If you're serving a personalized dashboard, a product page with real-time inventory, or a news feed that updates constantly, a single miss for that content can be expensive. A better metric is origin request rate—how many requests per second actually reach your application. If that number is high, your hit rate is irrelevant.

Moreover, a high hit rate can mask poor cache key design. If you're caching by URL alone but your responses vary by user or by device, you're either serving wrong content or you're not caching at all. The Vary header exists to solve this: it tells caches to store a separate response for each value of a listed request header (MDN). If you set Vary: Accept-Encoding, you get separate cache entries for gzipped and plain responses. If you forget Vary on a response that depends on cookies, you'll either serve one user's data to another (a security disaster) or you'll miss on every request because the cookie changes. The hit rate looks fine, but your origin is still doing all the work.

stale-while-revalidate: The Underused Hero

The most powerful tool in your HTTP caching arsenal is a humble two-word directive: stale-while-revalidate. It's defined in RFC 5861 and lets a cache serve a stale response for a specified number of seconds while it revalidates in the background (MDN). This is a game-changer for dynamic content that changes less often than users refresh.

Consider a news article that gets updated every few minutes. With a standard max-age=60, a user who requests the article 61 seconds after the last update gets a cache miss, and the origin generates the page again. With stale-while-revalidate=300, that same user gets the cached version (up to 5 minutes stale) while the cache fetches the fresh version in the background. The user sees a response instantly, and the origin only handles the revalidation request—which can be a cheap conditional GET if you use validators like ETag or Last-Modified (MDN).

This pattern can slash origin load without sacrificing perceived freshness. In fact, you can often serve content that is 10 minutes stale if your users don't notice. But how do you know what's acceptable? Look at your user behavior: if a page gets most of its views within the first 5 minutes after publication, a 5-minute stale window is fine. If it's a live scoreboard, you need a much shorter window. The point is, you can trade hit rate for freshness, and often you can have both.

Cache Invalidation: The Other Side of the Coin

High hit rates also make cache invalidation harder. When you update a resource, you need to purge the cached copy. If you're using a CDN like Cloudflare, you can purge by single-file, which instantly removes the asset across all data centers (Cloudflare). But if you're serving content that is generated dynamically, you can't just purge a URL—you need to coordinate with your application. That's where the must-revalidate directive becomes crucial. RFC 9111 says that once a response is stale, a cache MUST NOT reuse it until it has been validated (RFC 9111). This is the strictest setting, and it's appropriate for content that can never be served stale, like a user's account balance.

But here's the contrarian take: don't use must-revalidate for everything. It forces a revalidation request on every miss, which can be as expensive as a full response if you don't have validators. Instead, use stale-while-revalidate for content that can tolerate staleness and reserve must-revalidate for the truly critical stuff. This way, you get the benefit of high hit rates for stable content and freshness for dynamic content, without sacrificing either.

A Practical Tuning Checklist

So how do you actually tune your caching for performance? Start by measuring your origin load. If your origin CPU is at 80% and your cache hit rate is 95%, you're still in trouble. Add stale-while-revalidate to your dynamic endpoints and watch your origin load drop. Next, audit your Cache-Control headers. Are you setting max-age on static assets? Are you using s-maxage for shared caches? Remember that s-maxage takes precedence for shared caches (RFC 9111). If you're using a CDN, you can also rely on its default caching behavior: Cloudflare caches 200 responses for 120 minutes by default (Cloudflare). But don't let defaults be your strategy—set explicit headers.

Here's a quick tip: Use Vary: Accept-Encoding on all compressible responses. It's a simple way to ensure your cache doesn't serve a gzipped response to a client that doesn't support it, and it's a common mistake that kills hit rates.

• Check your origin request rate, not just hit rate.
• Apply stale-while-revalidate to dynamic content that can tolerate staleness.
• Use must-revalidate only for content that must always be fresh.
• Audit your Vary headers to ensure proper cache keys.

Finally, don't forget the load balancer. If you're using nginx as a reverse proxy, you can enable its built-in cache with proxy_cache_path and proxy_cache_valid. This sits between the CDN and your application, catching any misses before they hit your origin. It's a second layer that can further reduce load.

The Takeaway

Stop chasing a 95% hit rate as an end in itself. Instead, define a target for origin load and user latency, and tune your caching to meet those goals. Use stale-while-revalidate to serve stale content while revalidating in the background, and use must-revalidate for the few things that truly need it. The result is a system that is both fast and fresh, and your origin will thank you.

Sources

• AWS Caching Overview - https://aws.amazon.com/caching/
• MDN Cache-Control - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• RFC 5861 (stale-while-revalidate) - https://www.rfc-editor.org/rfc/rfc5861.txt
• RFC 9111 (HTTP Caching) - https://httpwg.org/specs/rfc9111.html
• Cloudflare Default Cache Behavior - https://developers.cloudflare.com/cache/concepts/default-cache-behavior/
• Nginx proxy module - https://nginx.org/en/docs/http/ngx_http_proxy_module.html