Why Your Cache Hit Rate Is a Lie (and How to Fix It)

I've lost count of the times a developer has told me, "Our cache hit rate is 90%, we're good." And I've lost count of the times that 90% was hiding a serious problem. The truth is, a high hit rate can be a vanity metric, especially if you're not tracking the right things or if you're letting your cache behave like a stubborn mule. So, let's answer the question I get more than any other: "Why is my cache hit rate high but my origin server is still getting hammered?"

Is a High Cache Hit Rate Always a Good Thing?

No. A high hit rate on a CDN, for instance, often means your cache is serving stale content or that you've configured it to cache things you shouldn't. The real goal isn't a high hit rate; it's a healthy origin server. CDNs can absorb 70% or more of origin requests (AWS), but if your origin is still struggling, that 90% hit rate might be masking a problem with your cache key or your invalidation strategy. I've seen teams celebrate a 95% hit rate while their database is on fire because they're caching the wrong layer or because their cache is serving stale data that's causing errors downstream. Don't chase the number; chase the user experience.

What's the Difference Between a CDN and a Load Balancer?

People often treat them as interchangeable, but they're not. A load balancer distributes traffic across your servers; a CDN caches content closer to users. However, they're not mutually exclusive. In fact, a CDN acts like a massive, distributed load balancer. When a CDN cache misses, it has to fetch from the origin, but a good CDN will use tiered caching to reduce the load on your origin. With Cloudflare's Tiered Cache, only an upper-tier data center can contact your origin, so you're not getting requests from every edge node (Cloudflare). That's a load-balancing win. So, the next time you're scaling, think about whether a CDN could offload some of the work your load balancer is doing.

Why Does My Cache Keep Serving Stale Content?

This is the classic cache invalidation headache. It's not just about setting a TTL; it's about actively telling your cache when content changes. HTTP has a mechanism for this: when you make an unsafe request (like POST, PUT, or DELETE), caches must invalidate the stored URI (RFC 9111). But if you're not using those methods, or if you're relying on a long max-age, you're asking for trouble. I recommend using a short max-age combined with stale-while-revalidate. That way, users get a fast response (even if slightly stale) while your cache fetches the fresh version in the background (RFC 5861). It's the best of both worlds.

Is It True That I Should Never Use no-store?

That's a myth. no-store is not evil; it's a tool. It tells caches to not store the response at all (MDN). You should use it for anything sensitive or personalized, like a user's private dashboard or a shopping cart. But for public, static content, it's a disaster. If you're using no-store on your images, you're forcing your CDN to fetch from your origin every time, which defeats the purpose of having a CDN. So, don't ban no-store; use it where it belongs, and use public with a sensible max-age for everything else.

How Do I Choose Between Redis and Memcached?

This is a classic architecture debate. For years, I leaned on Memcached because it's multithreaded and simple. But Redis has won me over. Redis supports advanced data structures, pub/sub, Lua scripting, and persistence (AWS). But the real game-changer for me is the eviction policies. Redis gives you allkeys-lru or allkeys-lfu, and the LFU mode can provide a better hit ratio than LRU in certain cases (Redis). Memcached uses a slab allocator that can waste memory, and it lacks the flexibility. For a cache, I'd pick Redis every time. Just remember: a cache is not a storage, so use allkeys-lru or allkeys-lfu and let it evict.

What's the Best Load Balancing Algorithm for My API?

There's no single best; it depends on your traffic pattern. Round robin is fine for short, stateless connections (system-design-primer). But if you have long-lived connections or sticky sessions, you need something smarter. For HTTP APIs, I'd start with round robin and monitor. If you see uneven load, switch to least connections or least response time. HAProxy's leastconn is great for long sessions like SQL or LDAP, but not for short HTTP (HAProxy). And if you need session persistence, use IP hash or a cookie-based method. Don't overthink it; start simple and iterate.

Bottom Line

The single best move you can make is to stop obsessing over the hit rate and start measuring what matters: origin load, latency, and user experience. Then, set sane cache headers (short max-age + stale-while-revalidate) and use a CDN with tiered caching to offload your origin. That combination will keep your users happy and your servers breathing.

Sources

• AWS Caching Overview - https://aws.amazon.com/caching/
• Cloudflare Tiered Cache - https://developers.cloudflare.com/cache/how-to/tiered-cache/
• MDN Cache-Control - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• RFC 5861 - https://www.rfc-editor.org/rfc/rfc5861.txt
• RFC 9111 - https://httpwg.org/specs/rfc9111.html
• Redis Eviction - https://redis.io/docs/latest/develop/reference/eviction/