Devops Observability Quality · TL;DR
1 min readTL;DR
The Platform runs on a self-managed Kubernetes cluster (K3s) on Hetzner, in a single dloizides namespace, with a separate staging cluster used to run nightly tests against production. Container images are built, pushed to a private registry over a WireGuard VPN, and deployed by digest pin (the manifest references the image by its sha256 digest, not a mutable tag) so every deploy is reproducible and rollback is exact. A single manage.sh deploy <app> script drives deploys; Traefik is the ingress (TLS via Let's Encrypt, host routing, middleware).
Locally, Tilt orchestrates every dev server, test watcher, build, lint, typecheck, and container rebuild — all on manual trigger mode so nothing rebuilds in a runaway loop. End-to-end coverage is Playwright, one suite per product/area, runnable on a dev PC, via Tilt, or as an in-cluster Kubernetes Job + nightly CronJob that targets production.
Observability stands on the classic pillars — Sentry (errors/crashes), OpenTelemetry (traces), Prometheus (metrics), Loki (logs) — plus health checks, a canary, and Alertmanager critical-alert emails. Scheduled smoke tests prove deployed services are actually up, and a Daily Environment Report email summarises health. Public web apps must pass a mandatory quality baseline: Umami analytics, an SEO baseline, spam-safe contact, a Lighthouse score ≥ 80, and axe-core accessibility checks — with a cron static-smoke audit flagging any app that drifts.