Forget Cache Hit Ratios: What Actually Worked When I Fixed a Product Catalog API

Stop optimizing your cache hit ratio. That's the contrarian advice I give every backend engineer who shows me a Grafana dashboard full of hit rates. The number that matters is origin offload. A 99% hit rate on a cache serving 10 requests per second saves you nothing. A 70% hit rate on a cache serving 50,000 requests per second saves your database from melting. I learned this the hard way on a product catalog API, and this is the walkthrough I wish I'd had.

This is for you if you run a read-heavy service behind a load balancer and your database is the bottleneck. You've probably already added a cache somewhere, and it's probably not doing what you think.

Map your layers before you touch a config file

The first mistake is treating caching as one thing. It's not. A typical architecture is layered from the user outward: browser cache, CDN edge cache, load balancer or proxy cache, application cache, distributed cache, then the database. Each layer has different invalidation semantics, different costs, and different failure modes. If you can't draw this stack on a whiteboard from memory, you're not ready to tune anything.

In my case study, the stack was browser, Cloudflare, nginx, a Node app with an in-process LRU, Redis, and Postgres. The in-process cache was the problem. It was fast, yes, but it had no shared invalidation, so every one of eight app instances had a different view of the catalog. I ripped it out.

Put the CDN in front and let it eat the static traffic

Static content is any file that's identical every time it's delivered, and CDNs are built to serve it. Your product images, your CSS bundles, your JS chunks — those should never touch your origin after the first request. Cloudflare's default behavior caches 200, 206, and 301 responses for 120 minutes, 302 and 303 for 20 minutes, and 404 and 410 for 3 minutes when you haven't set cache headers. That's a reasonable starting point, but you should be explicit, not default.

Set Cache-Control: public, max-age=604800, immutable on hashed assets. That's seven days of freshness. If your build pipeline changes the filename on every deploy, you can go longer. If it doesn't, fix your build pipeline first.

Get your HTTP caching semantics right, or nothing else matters

Here's where most teams quietly fail. They set max-age and assume they're done. But RFC 9111 defines a precedence order: s-maxage wins for shared caches, then max-age, then Expires minus Date, otherwise a heuristic is used. And if you set both max-age and Expires, compliant caches must ignore Expires entirely. I've seen teams set Expires to a year in the future and max-age to 60 seconds, then wonder why their CDN is revalidating constantly. The max-age wins.

Use validators. An ETag is an identifier for a specific version of a resource, and when content hasn't changed, the server doesn't need to resend the full response. Last-Modified is a fallback and it's less accurate, so use it only when you can't generate ETags. If you're serving JSON from an ORM, you can usually hash the serialized body cheaply enough.

And watch Vary. It lists the request headers that influenced the response, and caches store separate copies per listed header. If you set Vary: User-Agent on a catalog endpoint, you just destroyed your hit rate. Every browser version gets its own cached copy. Don't do it.

Configure Redis as a cache, not a database

This is the step where you have to make an actual decision. Redis is an in-memory key-value store with typical hit rates of 90% to 95% and 1 to 5 ms latency, and adding it often cuts database CPU by 70% to 90%. Those are the numbers that justify the operational cost. But only if you configure it as a cache.

Set maxmemory and set maxmemory-policy allkeys-lru. The default eviction policy is LRU, and allkeys-lru is recommended because a cache is not storage. If you're running noeviction, you've built a time bomb: Redis will start returning errors for commands that try to cache new data once memory is full. I've watched a team spend a weekend debugging 500s that were just OOM command not allowed errors.

For writes, use cache-aside. Check the cache first, fetch from the database on a miss, then write the result back. It's the most common pattern for a reason. Don't try to be clever with write-through on a catalog that changes once a day.

Use stale-while-revalidate so your users never wait

RFC 5861 defines two Cache-Control extensions that most teams ignore: stale-while-revalidate lets a cache serve a stale response for up to the indicated seconds while it revalidates in the background, and stale-if-error lets a cache return a stale response when it hits an error like a 500 or a DNS failure.

Apply this to your catalog API. Set Cache-Control: max-age=60, stale-while-revalidate=300, stale-if-error=86400. The first request after freshness expires gets a stale response immediately, and the origin revalidation happens behind it. Your p99 latency stops spiking every 60 seconds. The stale-if-error window means a brief origin outage doesn't become a user-visible outage. This is the single highest-leverage header change I've made.

Load balance with the algorithm that matches your traffic

Round robin is the default for a reason, but it's not always right. Round robin works for short, stateless connections; least connections or least response time is better for long connections; and hashing or cookies are for session persistence.

For a catalog API with uniform request costs, round robin behind nginx is fine. nginx's default is round-robin, and it also supports least-connected, ip-hash, least-time, and weighted methods. If your requests have wildly different costs — say, a search endpoint that's 100x more expensive than a product lookup — switch to least_conn. If you're load balancing in front of your own cache tier and you want to maximize hit ratios, use consistent hashing: nginx's hash directive with the consistent parameter uses ketama hashing so adding or removing a server remaps only a few keys.

One warning here: don't put a load balancer in front of a cache tier without thinking about what happens when a node dies. With consistent hashing, only the keys on that node get remapped, but that's still a burst of misses hitting your origin all at once. The ketama approach keeps it manageable, but you should pre-warm or accept the spike.

Speaking of spikes: when we added a new Redis node to handle growth, the consistent hashing remapped about 5% of keys. That tiny fraction translated to roughly 2,500 extra origin requests per minute for about 90 seconds. Our Postgres CPU jumped from 40% to 85%. We survived because we'd pre-warmed the new node with the top 1,000 product keys the night before. Without that, we'd have had a rough morning.

What can go wrong: invalidation will bite you

Here's the failure I actually hit. We cached a product response for 10 minutes. A price change went out. The origin updated. The cache didn't. For 10 minutes, customers saw the old price. That's a business problem, not a technical one.

RFC 9111 says that unsafe methods like PUT, POST, or DELETE can change origin state, so caches must invalidate the stored target URI when they see a non-error response to an unsafe request. But that only helps if your writes go through the same cache layer. If you're writing directly to Postgres and reading through Cloudflare, nothing invalidates. You need to either purge explicitly or shorten your TTL to something you can tolerate.

Cloudflare's purge by single-file instantly removes a cached asset across all data centers, so the next request gets the latest version. That's your escape hatch. Build the purge call into your write path from day one. Don't wait until a pricing incident forces you to.

Also: if you use Vary: *, the response becomes uncacheable. I've seen this set accidentally by a framework middleware. Check your actual response headers with curl before you trust your config.

What I'd actually do

If I were rebuilding this catalog API tomorrow, I'd do exactly four things. First, put Cloudflare in front with explicit Cache-Control headers on every route — no defaults. Second, run Redis with maxmemory-policy allkeys-lru and a TTL on every key via SETEX, so nothing lives forever by accident. Third, set stale-while-revalidate and stale-if-error on every cacheable API response, because the latency win is free. Fourth, wire cache purge into the write path before I write the first read endpoint.

Skip the in-process cache. Skip the hit-rate dashboard. Measure origin offload, measure p99 latency, and measure database CPU. Those are the numbers that tell you whether your caching strategy is working.

Sources

• AWS (caching overview) - https://aws.amazon.com/caching/
• MDN (Cache-Control) - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• RFC 9111 (HTTP Caching) - https://httpwg.org/specs/rfc9111.html
• RFC 5861 (stale-while-revalidate) - https://www.rfc-editor.org/rfc/rfc5861.txt
• Nginx (HTTP load balancing) - https://nginx.org/en/docs/http/load_balancing.html
• Redis (eviction) - https://redis.io/docs/latest/develop/reference/eviction/