Profiling & Benchmarking
9 min readProfiling & Benchmarking
TL;DR
Profiling tells you where time goes in a real workload; benchmarking tells you which of two implementations is faster under controlled conditions. They answer different questions and are not interchangeable. The practical skills are knowing which tool answers which question, and knowing why a hand-rolled Stopwatch loop is almost always lying to you — JIT warm-up, dead-code elimination, and GC timing conspire to produce numbers that are confidently wrong.
How it works
The tools and what each is for
| Question | Tool |
|---|---|
| Where does wall-clock time go in production? | Sampling profiler (dotnet-trace, PerfView, Visual Studio, dotTrace) |
| What is allocating and why is GC running? | dotnet-counters, memory profiler, dotnet-gcdump |
| Is implementation A faster than B? | BenchmarkDotNet |
| Why is this one request slow? | Distributed tracing (OpenTelemetry spans) |
| Is the problem even in my process? | Metrics first — CPU, GC %, thread-pool queue, DB duration |
Start with metrics, narrow with tracing, confirm with a profiler, and only then benchmark a candidate fix.
Sampling versus instrumenting profilers
A sampling profiler interrupts periodically and records the stack, so overhead is low and it can run in production, but it misses very short methods and gives statistical rather than exact counts. An instrumenting profiler injects timing into every method call, giving exact counts but with overhead high enough to distort the very behaviour being measured — inlined methods stop being inlined, and a cheap method called ten million times can dominate the profile purely through measurement cost. Default to sampling; reach for instrumentation only when you need exact call counts.
Why hand-rolled Stopwatch benchmarks lie
// Wrong in at least five ways.
var sw = Stopwatch.StartNew();
for (int i = 0; i < 1_000_000; i++) DoWork(i);
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);- JIT warm-up — the first calls run unoptimised, and tiered compilation may not have promoted the method to the optimised tier yet.
- Dead-code elimination — if the result is unused, the JIT may remove the work entirely, and you time an empty loop.
- GC timing — a collection triggered by the previous test lands inside this one, or vice versa.
- No statistics — one run gives no variance, so you cannot tell a 3% real difference from noise.
- Environment — debug build, debugger attached, or a background process stealing CPU.
BenchmarkDotNet handles all of it
[MemoryDiagnoser] // reports allocations per operation and GC counts
public class LookupBenchmarks
{
private List<int> _list = null!;
private HashSet<int> _set = null!;
[Params(100, 10_000)] // run the whole matrix at both sizes
public int N;
[GlobalSetup]
public void Setup()
{
_list = Enumerable.Range(0, N).ToList();
_set = _list.ToHashSet();
}
[Benchmark(Baseline = true)]
public bool ListContains() => _list.Contains(N - 1);
[Benchmark]
public bool SetContains() => _set.Contains(N - 1);
}It runs a warm-up phase until timings stabilise, executes enough iterations for statistical confidence, reports mean with standard deviation and confidence intervals, isolates each benchmark in its own process, forces a release build, refuses to run under a debugger, and — with [MemoryDiagnoser] — reports bytes allocated per operation. Returning a value from the method is what prevents dead-code elimination, which is why benchmark methods should never be void.
Allocation is often the more useful number
Time varies with machine and load; bytes allocated per operation is deterministic. If a change takes allocations from 2 KB per request to 200 bytes, that is a real, reproducible improvement in GC pressure regardless of whose laptop ran it. In CI, allocation regressions are far more stable to assert on than timing regressions, which are notoriously flaky on shared build agents.
Profiling in production
You cannot reproduce production load on a laptop, and the interesting problems only appear at real concurrency, real data volumes, and real cache-hit ratios. dotnet-trace, dotnet-counters, and dotnet-gcdump attach to a running process with low overhead and no restart. The essential counters: CPU usage, gen 0/1/2 collection counts, % Time in GC, allocation rate, thread-pool queue length, and lock contention. A thread-pool queue that grows under load is the signature of blocking calls starving the pool — one of the highest-value diagnoses in .NET.
A: Profiling measures where time and memory go in a real, complete workload — it answers "what should I fix". Benchmarking compares specific implementations under controlled, repeatable conditions — it answers "did this change help, and by how much". They are not interchangeable: a benchmark of an isolated method tells you nothing about whether that method matters to overall runtime, and a profile tells you where the cost is but not whether an alternative implementation would be faster. The correct order is profile first to locate the hotspot, then benchmark candidate fixes for that specific hotspot.
A: Because at least five effects distort the result. JIT warm-up means early iterations run unoptimised code, and tiered compilation may not yet have promoted the method. Dead-code elimination means the JIT can remove work whose result is unused, so you time an empty loop. Garbage collections triggered by earlier work land arbitrarily inside the measured window. A single run yields no variance, so you cannot distinguish a real few-percent difference from noise. And the environment — debug build, attached debugger, background load — silently changes everything. BenchmarkDotNet exists specifically to control all of these, which is why hand-rolled timings routinely report a "win" that disappears in production.
A: A sampling profiler periodically interrupts execution and records the call stack, then infers where time is spent from the distribution of samples. Overhead is low enough to run in production, but very short-lived methods may be missed and the numbers are statistical. An instrumenting profiler injects timing code into method entry and exit, giving exact call counts and per-method times, but its overhead is high enough to change the behaviour it measures — methods that would have been inlined no longer are, and a trivial method called millions of times can dominate the profile purely through measurement cost. Sampling is the default; instrumentation is for when you specifically need exact call counts.
A: Because it is deterministic and machine-independent. Elapsed time varies with CPU, background load, thermal state, and the mood of a shared CI agent, so small timing differences are unreliable and timing-based regression tests are notoriously flaky. Bytes allocated per operation is the same on every run and on every machine, so a reduction from two kilobytes to two hundred bytes per request is a real, reproducible result that will translate into reduced GC pressure everywhere. It is also frequently the underlying cause of the timing variance in the first place, since allocation rate drives collection frequency and therefore the pauses that show up in tail latency.
A: Attach the .NET diagnostic tools to the live process rather than trying to reproduce it locally, because the interesting behaviour depends on real concurrency, data volumes, and cache-hit ratios that a laptop cannot recreate. dotnet-counters gives live metrics with negligible overhead, dotnet-trace collects a sampled CPU trace over a window, and dotnet-gcdump captures the object graph for memory problems — none of which require a restart or a special build. I would start with metrics to characterise the shape of the problem, use distributed tracing to find which span in a slow request holds the time, and only then collect a trace or dump during the window when it is actually happening.
A: CPU usage, to establish whether the process is compute-bound or waiting. Percentage of time in GC together with the gen 0, 1, and 2 collection counts, since a high gen-2 rate points at objects surviving into the old generation and is a strong signal of a leak or an oversized cache. Allocation rate, which drives all of that. Thread-pool queue length and thread count, because a growing queue is the signature of blocking calls starving the pool. And lock contention. Those five characterise almost every .NET performance problem well enough to choose the next tool.
A: Not without knowing the variance. A 5% difference is well within the run-to-run noise of most machines, so the meaningful question is whether the confidence intervals of the two measurements overlap — which is exactly what BenchmarkDotNet reports, and why its output includes standard deviation and error rather than a single number. I would also check whether the result reproduces across runs and on a different machine, whether the allocation figures moved in the same direction, and whether the benchmark is representative of the real workload rather than a microbenchmark hitting a warm cache. And even if it is real, a 5% win on a component that is 10% of total runtime is a 0.5% overall improvement, which may not justify the change.
A: That work is arriving faster than the pool can start it, which in a .NET service almost always means threads are being blocked rather than freed — synchronous I/O, .Result or .Wait() on a task, a lock held across an await, or a blocking database driver. Blocked threads cannot serve other requests, so the pool injects new threads slowly, by design, and the queue grows in the meantime, which shows up as latency that increases with load rather than a constant slowness. The fix is to make the blocking path genuinely asynchronous rather than raising the minimum thread count, which merely masks the problem and consumes memory in stacks. It is one of the highest-value diagnoses available because the symptom — everything is slow under load — points nowhere obvious on its own.
Key takeaways
- Profiling finds what to fix; benchmarking confirms whether a fix helped. Different questions, different tools.
- Hand-rolled
Stopwatchloops lie: JIT warm-up, dead-code elimination, GC timing, no variance, wrong environment. - BenchmarkDotNet handles all of that — and a non-
voidreturn is what stops the JIT deleting your work. - Allocation per operation is deterministic where timing is not — better for CI assertions and usually the root cause anyway.
- Sampling profilers can run in production; instrumenting ones distort what they measure.
- Five counters characterise most .NET problems: CPU, % time in GC, gen-2 rate, allocation rate, thread-pool queue.
- A growing thread-pool queue means blocking, not insufficient threads.