Node Right-Sizing Study Portal

Key takeaways

• Measure before you attribute (the headline lesson): a capacity plan is a hypothesis — verify each assumption against live ground truth (kubectl top, describe node, in-pod printenv, free -h) before acting. Here the centerpiece assumption was false.
• Memory binds small nodes, not CPU or container count.
• Requests schedule, limits cap. Requests near allocatable ⇒ nothing schedules; overcommitted limits on a no-swap node ⇒ OOM-kills. Same limit on two services with different usage proves the limit is a ceiling, not the cause.
• GC theory is real but verify the mode: Server GC = one heap per logical core (higher baseline); Workstation = single heap, lower. Never assume which a service runs — kubectl exec POD -- printenv | grep -i gc. In our cluster the fat services were already on Workstation GC; flipping would have saved nothing.
• Committed slack vs live working set: GC mode and DOTNET_GCConserveMemory only return free/committed pages; they cannot reclaim live objects (EF models, broker channels, caches). When footprint is live, GC tuning yields little.
• Uptime accumulation is the real driver (A/B-proven): same content-api at 94 MB fresh vs 284 MB aged, kefi-api 419 MB at 16h, and GCConserveMemory=5 made a fresh pod worse (121 MB). Aged .NET pods accumulate caches/LOH/broker buffers the GC won't return.
• Validated levers, in order: (1) lower the memory limit (GC heap ≈ 75% of cgroup limit → tighter cap forces the GC compact; roll one service at a time + swap cushion; risk = managed OOM); (2) periodic restart (non-durable band-aid); (3) app-level reduction (cache eviction, MassTransit PrefetchCount, LOH); (4) scale-to-zero (KEDA) — sidesteps accumulation entirely, strongest for low-traffic. GC-mode flipping is NOT a lever if already on Workstation GC.
• DATAS (DOTNET_GCDynamicAdaptationMode=1) keeps Server-GC throughput but collapses heaps when idle — the middle option for bursty services.
• A tight limits.memory caps the managed heap (75% default) → catchable OutOfMemoryException beats a silent kernel SIGKILL.
• KEDA for HTTP scale-to-zero on tiny nodes (not Knative); never scale stateful workloads or queue consumers to zero.
• Scale-to-zero frees RAM, not the hardware bill — only per-request serverless bills $0 at idle.
• Don't rewrite to chase a footprint that's live: a Go rewrite of the same features doesn't remove live objects either; AOT shrinks the baseline, not the working set. Reserve Go for lean always-on edge.
• Disk hygiene: crictl rmi --prune reclaims a stale-image-bloated store (in-use protected); quarterly chore.
• Swap with low vm.swappiness is a safe OOM cushion (k3s --fail-swap-on=false, NodeSwap GA in 1.34).
• Prometheus agent mode (~120 MB) for capacity observability instead of a full local stack.
• Methodology: validate assumptions against ground truth, pilot one service, verify, roll out; keep every change reversible.