Cache Stampede, Hot Keys and Stale Reads

Caching is where a system's performance problems go to become availability problems. Each of the three failures here appears only under load, only when the cache is doing its job, and each has a well-known fix that is easy to omit.

The stampede#

public Featured get() {
    Featured cached = cache.getIfPresent(KEY);
    if (cached != null) return cached;
    // Every thread that reaches here runs the query. Nothing coordinates them.
    Featured fresh = repository.loadFeatured();
    cache.put(KEY, fresh);
    return fresh;
}

LoadingCache<String, Featured> cache = Caffeine.newBuilder()
        .maximumSize(10_000)
        .expireAfterWrite(Duration.ofMinutes(5))
        .build(key -> repository.loadFeatured());   // LOADER
 
public Featured get() {
    return cache.get(KEY);      // 1,000 concurrent misses -> ONE query
}

LoadingCache.get guarantees that the mapping function runs at most once per key at a time. The first thread to miss loads; the rest block on that computation and receive its result. One query instead of a thousand, with no coordination code of your own.

This is the single most valuable line in this topic, and it is why Caffeine.build(loader) should be the default over hand-rolled get-check-put.

Refresh instead of expire#

LoadingCache<String, Featured> cache = Caffeine.newBuilder()
        .refreshAfterWrite(Duration.ofMinutes(5))     // refresh in the BACKGROUND
        .expireAfterWrite(Duration.ofMinutes(30))     // hard ceiling on staleness
        .build(key -> repository.loadFeatured());

The distinction matters. With expireAfterWrite alone, a request arriving after the TTL blocks while the value is reloaded. With refreshAfterWrite, a request arriving after five minutes gets the slightly stale value immediately while a background thread reloads. The user never waits, and the expireAfterWrite ceiling guarantees staleness is bounded even if refreshes fail.

For a value that is expensive to compute and tolerant of being a few minutes old — which describes most caches — this pairing is the right default.

Probabilistic early expiry#

Refresh-ahead solves the hot key. For thousands of keys that were all populated at the same moment — after a deploy, or a cache flush — they will all expire at the same moment too.

// Instead of a uniform 5 minutes, spread expiry over 5-6 minutes.
Caffeine.newBuilder()
        .expireAfter(new Expiry<String, Featured>() {
            @Override public long expireAfterCreate(String k, Featured v, long now) {
                long base = Duration.ofMinutes(5).toNanos();
                long jitter = ThreadLocalRandom.current().nextLong(Duration.ofMinutes(1).toNanos());
                return base + jitter;
            }
            @Override public long expireAfterUpdate(String k, Featured v, long now, long d) { return d; }
            @Override public long expireAfterRead(String k, Featured v, long now, long d)   { return d; }
        })
        .build(loader);

The formal version is XFetch: on each read, refresh early with a probability that rises as the entry approaches expiry, weighted by how expensive it was to compute. Expensive, popular entries get refreshed well before they expire; cheap ones wait. A random jitter on the TTL captures most of the benefit with a fraction of the complexity.

Hot keys#

Redis cluster, 6 nodes, 100,000 keys.
Key 'config:feature-flags' is read on EVERY request: 50,000 req/s.
 
That key hashes to one slot on one node. That node is at 100% CPU
while the other five are at 8%. Adding nodes does not help.

Two fixes, often used together.

A local cache in front. Feature flags, configuration and reference data change rarely and are read constantly. A one-second in-process Caffeine cache in front of Redis reduces 50,000 requests per second to one per second per instance, and the one-second staleness is almost always acceptable.

public Flags flags() {
    return local.get("flags", k -> redis.get("config:feature-flags"));
}
// local: Caffeine, expireAfterWrite(1s)   -> absorbs the read volume
// redis: shared, authoritative            -> one read per instance per second

Shard the key. When the value genuinely must be per-request fresh, write it under N suffixed keys and have each client read a random one:

private static final int REPLICAS = 16;
 
public String read() {
    int shard = ThreadLocalRandom.current().nextInt(REPLICAS);
    return redis.get("counter:" + shard);
}
 
public void write(String value) {
    for (int i = 0; i < REPLICAS; i++) redis.set("counter:" + i, value);
}

Reads spread across sixteen slots and therefore across nodes; writes cost sixteen operations, which is fine when reads outnumber writes by orders of magnitude.

Invalidation#

@Transactional
public void updateProduct(Product p) {
    repository.save(p);
    events.publish(new ProductChanged(p.id()));
}
 
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void evict(ProductChanged event) {
    cache.invalidate(event.id());
}

Invalidate rather than update. Writing the new value into the cache looks more efficient and introduces a race: two concurrent writers can insert their values in the opposite order to which they committed, leaving the cache permanently disagreeing with the database. Deleting is idempotent and order-independent; the next reader repopulates from the source of truth.

Cache the misses too#

public Optional<Product> find(String id) {
    Product cached = cache.getIfPresent(id);
    if (cached != null) return Optional.of(cached);
 
    Optional<Product> found = repository.findById(id);
    found.ifPresent(p -> cache.put(id, p));      // absent results are NOT cached
    return found;
}

Every request for a nonexistent id goes straight to the database, every time. A crawler or a scanner walking random identifiers has an unlimited, uncached path to your primary store — this is cache penetration, and it is exploitable.

Fixes: cache a null sentinel with a short TTL (30–60 seconds), or put a Bloom filter of known identifiers in front so lookups for ids that certainly do not exist never reach the database at all.

The incident, end to end#

1. Symptom. Every day at 09:00 exactly, database CPU hits 100% for two minutes and the site is unusable.
2. The clock is the clue. A precise time means something scheduled, not organic load.
3. Find it. A nightly job flushes the product cache at 08:55. At 09:00 the morning traffic peak arrives to a completely cold cache.
4. Root cause. Every entry expires simultaneously and there is no single-flight loading, so each popular product produces hundreds of identical queries.
5. Mitigate. Stop the flush; rely on TTL.
6. Fix. Move to Caffeine.build(loader) for single-flight, add refreshAfterWrite, and jitter the TTL so entries never expire in lockstep.
7. Guardrail. Alert on cache hit ratio dropping below 80%, and on database query rate rising more than 3× over its five-minute baseline — both would have fired minutes before users noticed.

What gets asked#

"What is a cache stampede and how do you prevent it?" is the direct question. Answer with single-flight loading first, then TTL jitter, then refresh-ahead. The hot-key question — "one Redis node is at 100% and the others are idle" — is the follow-up, and the two-tier local cache is the answer that shows you have run one.