Caching · How it works
11 min readHow it works
Where caches live
There are more layers than people usually name, and each removes work from everything behind it.
| Layer | Holds | Invalidation difficulty |
|---|---|---|
| Browser | Assets, API responses | Hard — you cannot reach it; only TTL and URL change |
| CDN / edge | Static assets, cacheable GETs | Moderate — purge API, usually seconds |
| Reverse proxy | Rendered pages, API responses | Easy — you control it |
| Application memory | Hot objects, config | Easy per instance, but each instance differs |
| Distributed cache (Redis) | Shared hot data, sessions, computed results | Easy and consistent across instances |
| Database buffer pool | Recently read pages | Automatic |
The layer with the best performance is the one you can least control. A browser cache costs zero network round trips, which no server-side cache can match — and if you cache an asset for a year and then need to change it, you cannot. This is why content-hashed filenames exist: app.4f3a9b.js can be cached forever because a change produces a different URL, so the cache never needs invalidating at all. The HTML entry point that references it must then be no-cache, because that is the one file whose URL cannot change.
A: Because the hash makes the URL a function of the content, so any change to the file produces a different URL. A cached copy of app.4f3a9b.js is therefore never stale — it is a permanently correct answer for that exact content — and the new build simply requests a URL nobody has cached. That is what makes an immutable one-year cache safe and gives repeat visitors near-zero asset downloads. The HTML entry point is the exception because its URL is fixed: /index.html must stay /index.html for people to reach the site, and it is the file containing the references to the hashed assets. If it were cached long-term, a returning visitor would get the old HTML pointing at the old asset URLs and would never discover the new deploy, no matter how many times they reloaded. So the pattern is a single no-cache document that is always revalidated and is cheap because it is small, referencing immutable assets that are never revalidated at all.
Write strategies
The naming is standard and interviewers use it precisely, so it is worth having straight.
Cache-aside (lazy loading) is the default. The application checks the cache; on a miss it reads the database, populates the cache, and returns. Writes go to the database and delete the cache entry. Only requested data is cached, and a cache failure degrades to slow rather than broken.
Write-through writes to the cache and the database synchronously. Reads after a write are always warm and the cache never holds stale data, at the cost of higher write latency and of caching data nobody reads.
Write-behind (write-back) writes to the cache and acknowledges immediately, flushing to the database asynchronously. This gives the fastest writes and can coalesce many updates to the same key into one database write, but a cache node dying loses acknowledged writes — so it is only acceptable where that loss is tolerable, such as view counters.
Read-through is cache-aside with the loading logic inside the cache library rather than the application.
A: With cache-aside the application owns the logic: on a read it checks the cache, and on a miss it loads from the database, populates the cache, and returns; on a write it updates the database and invalidates the entry. Only data that is actually requested occupies memory, and if the cache is unavailable the system still works, just slower — which makes it the sensible default. Its weakness is that every first read of an item is a miss, so a cold cache after a restart or deploy means a burst of database load. Write-through updates cache and database together on every write, so the cache is never stale and a read immediately after a write is guaranteed warm — valuable when the write-then-read pattern is common, such as a user editing their profile and being shown it. Its costs are that write latency now includes both stores, that a cache failure can block writes unless you handle it, and that you cache everything written regardless of whether it is ever read, which wastes memory when writes are spread over a large key space. I would default to cache-aside, and choose write-through where reads reliably follow writes closely and the write volume is low enough that the added latency does not matter.
The cache stampede
This is the highest-value failure mode to be able to describe. A popular key expires. Between its expiry and the moment it is repopulated, every request for it misses, and all of them go to the database simultaneously. A key serving 10,000 requests per second produces 10,000 concurrent identical database queries, which can be enough to take the database down — and then nothing repopulates the cache, so the failure persists.
There are three standard mitigations and they compose.
// Mitigation 1: single-flight. Only one caller recomputes; the rest await it.
private static readonly ConcurrentDictionary<string, Lazy<Task<Product>>> _inFlight = new();
public Task<Product> GetProductAsync(string id)
{
if (_cache.TryGet(id, out Product cached)) return Task.FromResult(cached);
// Lazy ensures the factory runs at most once even under concurrent access.
var lazy = _inFlight.GetOrAdd(id, key => new Lazy<Task<Product>>(async () =>
{
try
{
var fresh = await _database.LoadProductAsync(key);
// Mitigation 2: jitter the TTL so keys do not all expire together.
var ttl = TimeSpan.FromMinutes(10) + TimeSpan.FromSeconds(Random.Shared.Next(0, 120));
_cache.Set(key, fresh, ttl);
return fresh;
}
finally
{
_inFlight.TryRemove(key, out _);
}
}));
return lazy.Value;
}The third mitigation is to serve stale data while refreshing. Rather than deleting an expired entry, mark it stale and return it immediately while a background task refreshes it. Users get an instant, slightly old answer and the database sees exactly one query. This is what HTTP's stale-while-revalidate directive expresses, and it is usually the best option when a few seconds of staleness is acceptable.
A: It is what happens when a heavily requested key expires and every concurrent request for it misses at once, so all of them hit the database simultaneously with the identical query. A key serving ten thousand requests per second generates ten thousand concurrent identical queries the instant it expires, which can saturate the database — and because the database is now overwhelmed, nothing succeeds in repopulating the cache, so the situation sustains itself rather than recovering. There are three composable fixes. Single-flight, sometimes called request coalescing, ensures only the first caller recomputes while the others wait on that same in-flight result, so the database sees one query. TTL jitter adds a random offset to each expiry so that keys populated together — which is exactly what happens after a deploy or cache flush — do not expire together and create a synchronised wave. And serving stale while revalidating means an expired entry is returned immediately and refreshed in the background, so no request ever waits on a recomputation. The first is essential for hot keys; the second is nearly free and should be routine; the third is best wherever a few seconds of staleness is acceptable.
Hot keys
A hot key is a single item whose traffic exceeds what one cache node can serve. Because a distributed cache shards by key hash, every request for that key lands on the same node no matter how many nodes you add — so scaling the cluster does nothing. This is the celebrity problem: one user with fifty million followers, or one product in a flash sale.
The fixes all reduce to spreading the key or moving it closer.
Adding a small in-process cache in front of the distributed cache is usually the most effective single change, because the hottest keys are then served from local memory and never reach the network. A few seconds of local TTL removes the overwhelming majority of traffic for that key.
Alternatively, split the key into N variants — post:123:v0 through post:123:v9 — each holding the same value, and have readers pick one at random. Traffic divides across ten nodes, at the cost of ten copies to invalidate.
A: Because a distributed cache decides placement by hashing the key, so a given key deterministically maps to exactly one node regardless of how large the cluster is. All requests for that key therefore converge on that single node, and adding nodes only redistributes the other keys — the hot one stays exactly where it was, with exactly the same load. The node saturates on network bandwidth or CPU while the rest of the cluster is idle, which also makes the problem easy to misdiagnose from cluster-average metrics. The fixes have to either move the key off the network or stop it being one key. A short-lived in-process cache in each application instance serves the hot key from local memory, so the distributed cache sees one request per instance per TTL instead of thousands per second, and this is usually both the simplest and the most effective option. Key splitting — storing the same value under several suffixed keys and having readers choose randomly — spreads the load across nodes at the cost of multiple copies to invalidate. Dedicated handling, such as pinning known-hot entities to their own replicated tier, is the heavier option for cases like celebrity accounts that are hot permanently rather than briefly.
Eviction and TTL
When memory fills, something must go. LRU evicts the least recently used entry and is the sensible default because recency predicts reuse well. LFU evicts the least frequently used, which is better when a stable set of items is popular over time and you do not want a burst of one-off requests flushing them out. Redis also offers TTL-based and random eviction, and — importantly — a noeviction mode that returns errors on write when full, which is what you want when the data is not reconstructible.
TTL is a correctness control, not just a memory control. It bounds how wrong the cache can be, which is why a TTL should be chosen from the staleness the business tolerates rather than from a round number. Short TTLs increase database load; long ones increase the window of incorrectness.
A: Because the TTL is the maximum time the system may serve an incorrect answer, and only the business can say what that is worth. Technically you can set any value, and the technical trade is simple and monotonic — shorter means fresher and more database load, longer means staler and less load. But the question of whether a price can be five minutes out of date, whether a permission revocation may take an hour to take effect, or whether a follower count can lag a day is a question about user expectation, regulatory obligation, and revenue risk, not about infrastructure. A product price served stale after a change can mean selling below cost or a consumer-protection issue; a stale permission means a removed employee retains access for the TTL. So the correct approach is to ask what staleness is acceptable for this specific data, set the TTL from that answer, and where the tolerable staleness is effectively zero, recognise that TTL is the wrong tool and you need explicit invalidation on write, or you should not be caching that read at all.
Invalidation
Time-based expiry is the simplest and the one that always works, because it requires no coordination. Explicit invalidation on write is more precise but must reach every layer that holds a copy — and the layers you do not control, notably browsers, cannot be reached at all.
The subtle choice on write is between deleting the entry and updating it. Deleting is safer: the next reader repopulates from the database, so the cache cannot contain a value that never existed. Updating is faster but risks writing a stale value if two concurrent writes complete in a different order in the cache than in the database. For anything where correctness matters, delete.
Quick recall
Short-answer versions of the same material, for spaced repetition.
A: A hot key expires and every concurrent request for it misses at once, sending thousands of identical queries to the database simultaneously.
A: Single-flight coalescing so only one caller recomputes, TTL jitter so keys do not expire together, and serving stale data while refreshing in the background.
A: Placement is decided by hashing the key, so that key maps to exactly one node no matter how large the cluster is. Adding nodes only moves other keys.
A: Delete. Two writers can reach the database and the cache in opposite orders, so updating can leave the older value cached permanently.
A: The maximum time the system may serve an incorrect answer, which is why it is chosen from business staleness tolerance rather than a round number.
A: When a scanning workload shares the cache with an interactive one, because a one-off sweep of many items would evict the hot working set under LRU.