Caching, Async & Throughput
10 min readCaching, Async & Throughput
TL;DR
Caching is the highest-leverage optimisation available and the one most likely to introduce correctness bugs, because it trades freshness for speed and the bug shows up as "a user saw stale data", not as an exception. Async is the highest-leverage throughput optimisation in .NET and does nothing for latency — it frees threads while waiting rather than making the wait shorter. Both are graded in interviews on whether you know the failure modes: stampede, invalidation, and per-instance divergence for caching; deadlock, thread-pool starvation, and sync-over-async for concurrency.
How it works
Cache decisions, in order
- What are you caching? Immutable reference data is trivially safe. User-specific mutable data is where the bugs live.
- How stale may it be? This is a product question, not a technical one, and asking it is the mark of a senior answer.
- In-process or distributed? In-process is faster (no serialisation, no hop) but each instance caches independently — so with three instances a user can see three different values, and an invalidation reaches one of them. Distributed caching (Redis) gives one shared, coherently-invalidatable copy at the cost of a network round trip.
- How does it get evicted? A cache without an eviction policy is a memory leak with better branding.
Invalidation strategies
| Strategy | How | Trade |
|---|---|---|
| TTL / absolute expiry | Entry lives N seconds | Simple, predictable; stale for up to N |
| Sliding expiry | Resets on access | Keeps hot data; cold data ages out; hot data can be stale indefinitely |
| Explicit invalidation on write | Delete the key when the source changes | Freshest; needs every write path to remember, and misses out-of-band changes |
| Write-through | Update cache and store together | Consistent; couples the write path to the cache |
Explicit invalidation is the most correct and the most fragile, because it fails silently the moment someone adds a new write path — including a database migration, an admin tool, or another service. A TTL as a backstop under explicit invalidation gives you correctness with a bounded worst case, which is usually the right combination.
Cache stampede
When a popular key expires, every concurrent request misses simultaneously and all of them hit the database at once — the moment the cache is most needed is the moment it provides no protection. Mitigations: a lock or semaphore so only one caller recomputes while the others wait, serving the stale value while refreshing in the background, or jittering expiry times so keys do not expire in lockstep.
// Single-flight: only one caller recomputes a given key.
private static readonly SemaphoreSlim Gate = new(1, 1);
public async Task<Report> GetAsync(string key, CancellationToken ct)
{
if (_cache.TryGetValue(key, out Report? cached)) return cached!;
await Gate.WaitAsync(ct);
try
{
// Re-check inside the lock: another caller may have populated it.
if (_cache.TryGetValue(key, out cached)) return cached!;
Report fresh = await BuildExpensiveReportAsync(ct);
_cache.Set(key, fresh, TimeSpan.FromMinutes(5));
return fresh;
}
finally { Gate.Release(); }
}The re-check inside the lock is not optional — without it, every queued caller recomputes anyway once it acquires the gate.
Async is about throughput, not speed
await on genuine I/O returns the thread to the pool while waiting. The operation takes exactly as long as before; what changes is that the thread can serve another request meanwhile. That is why async raises throughput and leaves single-request latency unchanged — and why making CPU-bound work async achieves nothing but overhead.
The three async failure modes
Sync-over-async — .Result or .Wait() on a task. In ASP.NET Core this does not deadlock (there is no synchronisation context) but it does block a pool thread for the duration of the I/O, which is precisely what async existed to prevent. Under load, blocked threads accumulate, the pool injects replacements slowly by design, and the queue grows — latency rises with load while CPU sits idle. In UI or legacy ASP.NET contexts it deadlocks outright.
Thread-pool starvation — the systemic version of the above. The signature is a growing thread-pool queue, latency that scales with load, and low CPU. The fix is to make the blocking path genuinely async, not to raise MinThreads, which masks the symptom and burns memory on stacks.
Missing ConfigureAwait(false) in library code — matters where a synchronisation context exists; harmless but still recommended in ASP.NET Core for libraries that may be consumed elsewhere.
Batching, backpressure and bounded concurrency
Batching amortises fixed per-operation costs (round trips, locks, flushes) and is the main throughput lever after async — at the cost of latency for the items that wait for the batch.
Unbounded concurrency is a failure mode, not an optimisation: fanning out 10,000 concurrent calls exhausts connection pools, triggers downstream rate limits, and can take out a dependency. Bound it with Parallel.ForEachAsync's MaxDegreeOfParallelism, a SemaphoreSlim, or a Channel<T> with a bounded capacity, which additionally gives backpressure — the producer slows when the consumer cannot keep up, instead of buffering until memory runs out.
A: It is when a popular cache key expires and every concurrent request misses at once, so all of them recompute the same expensive value simultaneously and hammer the backing store — the cache fails precisely when it is most needed, and the resulting load spike can be worse than having no cache at all. The standard preventions are single-flight, where a lock or semaphore lets one caller recompute while the others wait for its result; serving the stale value while refreshing in the background so no request ever waits; and jittering expiry times so keys created together do not expire together. The detail people miss in the lock version is re-checking the cache after acquiring it, since another caller has usually populated it while you waited.
A: In-process is faster because there is no serialisation and no network hop, and it needs no extra infrastructure, but each instance caches independently — so in a horizontally scaled service the same user can see different values on consecutive requests, an invalidation reaches only the instance that handled the write, and total memory is multiplied by the instance count. Distributed caching in Redis gives one shared copy, coherent invalidation, and survival across restarts and deploys, at the cost of a round trip, serialisation, and a new dependency that must be kept available and whose failure mode you have to design for. The usual answer is both: a distributed cache for shared mutable data and a short in-process cache for immutable reference data.
A: Because correctness depends on every path that mutates the underlying data remembering to invalidate, and that set of paths grows silently — a new endpoint, an admin tool, a background job, a database migration, or another service writing to the same table all bypass your invalidation without any signal that they have. Explicit invalidation is therefore the freshest and the most fragile. A robust strategy pairs it with a TTL as a backstop, so a missed invalidation causes staleness bounded by the TTL rather than staleness forever. Making the acceptable staleness an explicit product decision, rather than an accident of implementation, is what turns this from a bug into a documented trade.
A: Not for a single operation. Awaiting genuine I/O returns the thread to the pool while the operation is in flight and resumes on a pool thread when it completes, so the operation takes exactly as long as it did before — what changes is that the thread is available to serve other work in the meantime. So async improves throughput and server scalability, not per-request latency. It follows that wrapping CPU-bound work in async accomplishes nothing but state-machine overhead, since there is no wait to give the thread back during, and that a service with plenty of threads and no concurrency pressure will see no benefit at all.
A: It is when pool threads are blocked rather than doing work, so incoming tasks queue while the pool adds replacement threads only slowly, by design, to avoid runaway thread creation. The signature is distinctive: latency rises with load while CPU utilisation stays low, the thread-pool queue length grows, and thread count climbs gradually. The usual cause is synchronous blocking on asynchronous work — .Result, .Wait(), GetAwaiter().GetResult() — or a genuinely synchronous I/O API somewhere in the path. The correct fix is to make that path async all the way down; raising MinThreads only masks the symptom, consumes a megabyte of stack per thread, and increases context switching.
A: In classic ASP.NET or a UI application it deadlocks: the continuation needs the captured synchronisation context to resume, that context is the very thread you have blocked, and neither side can proceed. In ASP.NET Core there is no synchronisation context so it does not deadlock, but it still blocks a pool thread for the entire duration of the I/O, which is exactly the waste async was introduced to eliminate — and at load those blocked threads accumulate into thread-pool starvation. It is dangerous in a subtler way too: it wraps any exception in an AggregateException, which changes what a caller's catch clauses match, so error handling that looked correct silently stops working.
A: It helps whenever there is a fixed per-operation cost that can be amortised — a network round trip, a transaction commit, a disk flush, a lock acquisition — which is why one batched insert of a thousand rows is orders of magnitude faster than a thousand individual inserts. What it costs is latency for the individual items, since each one waits for the batch to fill or for a timer to expire, so it is a direct throughput-for-latency trade. It also costs failure granularity: a batch that fails may fail entirely, so you need to decide whether partial success is acceptable and how to retry without duplicating, which usually means idempotent operations.
A: With Parallel.ForEachAsync and a MaxDegreeOfParallelism, a SemaphoreSlim gating the calls, or a bounded Channel<T> with a fixed number of consumers. You must because unbounded fan-out is not an optimisation but a failure mode: ten thousand concurrent outbound calls exhaust the connection pool, trip the dependency's rate limits, and can take down the service you are calling — turning your throughput improvement into someone else's outage, and then into your own when the retries arrive. A bounded channel additionally provides backpressure, slowing the producer when consumers fall behind rather than buffering unboundedly until the process runs out of memory.
Key takeaways
- Caching trades freshness for speed — decide the acceptable staleness explicitly; it is a product question.
- In-process caching diverges per instance: three instances, three answers, one invalidated. Distributed gives coherence for a round trip.
- Explicit invalidation is freshest and most fragile; pair it with a TTL so a missed invalidation is bounded, not permanent.
- Stampede: single-flight with a re-check inside the lock, background refresh, or jittered expiry.
- Async buys throughput, not latency — it frees the thread, it does not shorten the wait.
.Resultdeadlocks where a sync context exists and starves the pool where one does not. Async all the way down.- Thread-pool starvation looks like: latency scales with load, CPU idle, queue growing. Fix the blocking, don't raise MinThreads.
- Bound every fan-out. Unbounded concurrency is how your optimisation becomes a downstream outage.