Node Right-Sizing Study Portal

How it works

The scenario

A real production single-node K3s cluster:

The instinct is "too many containers" or "need a bigger CPU." Both wrong. The diagnosis showed CPU 26% used but memory 95% requested with only ~880 MB free, and limits overcommitted to 244%. The binding constraint was memory, and the lack of swap turned every spike into an abrupt kernel kill.

Step 1 — Diagnose: memory is the binding constraint

Three commands tell the whole story:

kubectl top nodes                # live actual usage (CPU + memory)
kubectl describe node <node>     # "Allocated resources" → requests vs limits
free -h                          # host-level free/used/available memory

The tell-tale pattern on this box: CPU low, memory requested near 100%, free memory tiny. Once requests approach allocatable, nothing new can schedule even if actual usage is lower — the scheduler reserves against requests, not live usage.

Step 2 — Requests vs limits (the single most important scheduling concept)

Rule of thumb: set requests to honest steady-state usage; set limits only modestly above the request (limits are blast-radius control, not a wishlist). Over-fat requests waste reserved capacity and break bin-packing; reckless limits on a no-swap node invite the OOM killer.

Step 3 — .NET garbage-collector mode (the theory — true, but verify it)

The textbook theory is correct: ASP.NET Core defaults to Server GC — one heap per logical core, optimized for throughput, with a higher memory baseline that scales roughly with core count. Workstation GC uses a single heap with slightly more frequent collections — a lower footprint, often a fit for low-traffic services. It flips via an environment variable, no rebuild required:

# Switch a deployment to Workstation GC
kubectl set env deployment/my-service DOTNET_gcServer=0

# Revert (trailing dash deletes the env var)
kubectl set env deployment/my-service DOTNET_gcServer-

But never assume which mode a service is already running — VERIFY it. Check the live pod before you "fix" anything:

kubectl exec <pod> -- printenv | grep -i gc
# look for DOTNET_gcServer / DOTNET_SYSTEM_GC_SERVER (=0 means Workstation)
# also check the app's runtimeconfig.json (System.GC.Server)

Step 3b — The plot twist: ground truth contradicted the plan (the centerpiece)

The capacity plan assumed the fat services were heavy because they ran Server GC, and that flipping them to Workstation GC would save ~150 MB each. We inspected the live cluster (kubectl top, kubectl exec ... printenv) and the assumption was exactly backwards:

The fat services were already on Workstation GC. The lean services (poueni, dynalux) were on default Server GC and were tiny anyway. The cheap "win" the plan was built around was already applied — and it hadn't made the big apps small.

The memory limit wasn't the driver either: poueni had the same 512Mi limit as kefi but used 85 MB vs 416 MB. Same ceiling, 5× the usage.

Conclusion: the footprint difference was real application working set, not GC mode. The big apps simply hold more live memory — bigger EF Core models, MassTransit/RabbitMQ consumer prefetch buffers, S3/storage clients, more loaded assemblies and caches. poueni/dynalux are just leaner apps. No GC-mode change touches genuinely-live objects.

Step 3c — Committed memory vs live working set (why GC tuning sometimes does nothing)

GC mode and DOTNET_GCConserveMemory only affect committed/free pages returned to the OS — slack the GC is holding onto. They cannot reclaim genuinely-live objects (open EF Core contexts/models, open broker channels, populated caches). If most of a service's footprint is live working set, GC tuning yields little.

So the diagnostic question is always: is this footprint reclaimable slack, or is it live? If it's live, GC knobs are the wrong lever.

Step 3d — When the footprint is real working set, what ARE the levers?

If a service is already on the right GC mode and most of its footprint is live, GC-mode flipping is not a lever. The real options, in order of safety:

1. Scale-to-zero for idle / long-tail services (KEDA) — idle = 0 RAM. Safest, no app change. (See Step 6.)
2. App-level reduction — trim the EF Core model, lower MassTransit PrefetchCount, lazy-load features, drop unused caches. Real savings but per-service and riskier (needs testing, can affect throughput).
3. DOTNET_GCConserveMemory=0..9 — returns free slack to the OS more aggressively, trading a little CPU. Bounded gains — only reclaims slack, not live objects.

Step 3e — The A/B test: the real driver was UPTIME ACCUMULATION

We A/B-tested on the live cluster with content-api (.NET 10, already Workstation GC) and a long-running pod, controlling for everything but uptime and the conserve-memory knob:

The dominant memory driver was uptime accumulation, not GC mode and not GCConserveMemory. Long-running .NET services accumulate working set over time — caches, large-object-heap (LOH) growth, broker prefetch buffers — and the GC holds onto it rather than returning it to the OS. A fresh pod was 3-4× smaller than the aged one. This is why the GC-mode assumption looked plausible (the fat services were big) but was the wrong attribution: they were big because they'd been up for hours, not because of their GC mode.

So the validated levers, in order:

1. Lower the container memory LIMIT. Because the .NET GC heap hard limit defaults to ~75% of the cgroup limit, a tighter limit (e.g. 512Mi → 256Mi) forces the GC to stay compact and return memory — it caps the climb. Risk: a managed OOM if the genuine live set exceeds the cap, so roll one service at a time and keep a swap cushion.
2. Periodic restart — cheap but a non-durable band-aid; memory climbs back as the pod ages again.
3. App-level reduction — cache eviction policy, MassTransit PrefetchCount, investigate the LOH — durable but per-service work.
4. Scale-to-zero for idle services (KEDA) — sidesteps accumulation entirely: a pod at 0 replicas accumulates nothing. The strongest lever for low-traffic services.

Step 4 — DATAS: the middle option (.NET 8+)

DATAS (Dynamic Adaptation To Application Sizes) keeps Server GC but auto-collapses the heap count under low load:

kubectl set env deployment/my-service DOTNET_GCDynamicAdaptationMode=1

Reported ~8× memory reduction when idle while retaining Server-GC throughput when load returns. Use it when a service genuinely needs Server-GC throughput sometimes but not always — the middle ground between full Server GC and Workstation GC.

Step 5 — GC memory knobs and the cgroup heap limit

# Trade a little CPU to return memory to the OS more aggressively (0..9)
kubectl set env deployment/my-service DOTNET_GCConserveMemory=5

Crucially, inside a cgroup the GC heap hard limit defaults to max(20 MB, 75% of the container memory limit). So simply setting a tight resources.limits.memory caps the managed heap — and a managed OutOfMemoryException (catchable, logged) is far preferable to a kernel OOM SIGKILL (silent, takes the whole process with no stack trace).

Step 6 — Scale-to-zero for "0 RAM when idle"

KEY CAVEAT: stateful workloads (databases, brokers, object stores) and message-queue consumers must NOT scale to zero — a sleeping consumer stops draining its queue, so the backlog grows unbounded. Mitigate cold-start latency on hot paths with ReadyToRun/AOT and a minReplicaCount: 1.

Step 7 — Cost honesty (a frequently-confused point)

Scale-to-zero gives 0 RAM per idle container — but not a $0 hardware bill. A self-hosted always-on node costs the same whether idle or full. Literal "$0 when no traffic" only exists on per-request serverless (Cloud Run, AWS Lambda, Fly.io auto-stop).

Practical split: keep the stable platform self-hosted (it's paid for anyway) and scale-to-zero its idle services to free RAM for others; send genuinely bursty, isolated workloads to serverless where you actually stop paying at idle.

Step 8 — Runtime footprint comparison & when to rewrite

These are idle baselines; a busy app's real footprint is baseline plus its live working set. AOT is Go-class only if AOT-compatible — no EF Core, no heavy runtime reflection. The lesson: don't rewrite a whole service in another language to chase the delta and forfeit a shared framework/library ecosystem. First find out why it's heavy — if it's GC slack on an untuned service, tuning is the free win; if it's real working set (our case), neither a GC flip nor a Go rewrite of the same feature set removes the live objects — you'd reduce the app or scale it to zero. Reserve Go for genuinely lean, always-on edge/proxy components where its tiny baseline matters.

Step 9 — AI/LLM placement on constrained infra

Don't self-host large models on a small GPU-less node: a quantized 7B model alone blows an 8 GB box and is unusably slow CPU-only. Use an external inference API plus a light local orchestrator (prompt-building, tenant quotas, storing results). A dedicated GPU node or serverless GPU is warranted only for data-residency requirements, high steady volume, or self-hosted fine-tuned models.

Step 10 — Node disk hygiene

The container-image store bloats because every digest-pinned deploy leaves stale images behind — here it reached 50 GB / 426 images.

# k3s containerd image store lives here:
#   /var/lib/rancher/k3s/agent/containerd
crictl rmi --prune              # remove unused images (in-use images are protected)
# reclaimed ~35 GB, 426 → 88 images

journalctl --vacuum-size=500M   # trim systemd journal logs

crictl rmi --prune protects in-use images. Treat image-store and journal cleanup as a recurring / quarterly chore.

Step 11 — Swap as an OOM cushion

A no-swap node gets abrupt kernel OOM-kills (SIGKILL) rather than a soft landing. Add a swap file with a low vm.swappiness so it's used only under genuine pressure:

fallocate -l 4G /swapfile && chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile        # does NOT restart kubelet → node stays Ready
sysctl vm.swappiness=10                      # only swap under real pressure

k3s defaults --fail-swap-on=false, and Kubernetes NodeSwap is GA in v1.34, so enabling swap is safe. swapon doesn't restart kubelet, so the node stays Ready throughout.

Step 12 — Observability for capacity

Run Prometheus in agent mode — it remote-writes metrics to a central store with a ~120 MB footprint — instead of a full local Prometheus + Grafana on a tight node. View capacity via Grafana's built-in "Kubernetes / Compute Resources / Cluster" dashboards.

Step 13 — The meta-lesson: measure before you attribute (the most valuable takeaway)

A capacity plan is a hypothesis, not a fact. Validate every assumption against live ground truth — kubectl top, kubectl describe node, in-pod printenv — before you act. In this real case the headline assumption ("the fat services are heavy because of Server GC; flip them and save ~150 MB each") was false: the fat services were already on Workstation GC, the lean ones were on Server GC, and the difference was real application working set. The "cheap win" the whole plan rested on was already applied and changed nothing.

plan = a set of HYPOTHESES, each one falsifiable
  → for each assumption: verify against the live cluster
      (kubectl top · describe node · kubectl exec ... printenv · free -h)
  → only act on assumptions that survive contact with ground truth
  → pilot ONE low-risk service → apply → verify (top, Grafana, E2E smoke) → roll out
keep every change reversible (env vars deletable with a trailing dash)

Change one thing, measure, then proceed — never big-bang on production. Reversibility makes it safe: kubectl set env deploy/x DOTNET_gcServer- cleanly undoes the flip. But the deeper discipline is attribution: don't apply a fix until you've confirmed the cause is what you think it is — otherwise you "fix" something that was never broken and the metric doesn't move.