DevOps & Quality Study Portal

How it works

The whole system is a loop: you change code, verify it through Tilt-driven gates, deploy a digest-pinned image with manage.sh, and then watch the deployed result through observability and smoke tooling. Here is the high-level pipeline.

                 LOCAL                                CI                          CLUSTER
   ┌───────────────────────────┐   ┌──────────────────────────┐   ┌────────────────────────────┐
   │  edit code                │   │  GitHub Actions          │   │  K3s on Hetzner            │
   │   │                       │   │   • lint / typecheck     │   │  namespace: dloizides       │
   │   ▼                       │   │   • unit tests           │   │                             │
   │  Tilt (manual trigger)    │   │   • prod build           │   │   Traefik ingress           │
   │   • lint   • typecheck    │──▶│   • security audit       │──▶│    (TLS / host / mw)        │
   │   • unit   • prod build   │   │   • e2e (selected)       │   │        │                    │
   │   • security audit        │   └──────────────────────────┘   │        ▼                    │
   │   • e2e suites            │              │                    │   Deployments / Services    │
   └───────────────────────────┘              │ build + push       │   (image pinned by DIGEST)  │
            │                                  ▼ (WireGuard VPN)    │        ▲                    │
            │ manage.sh deploy <app>      ┌─────────────────┐       │        │                    │
            └────────────────────────────▶│ private registry│──────▶│  manage.sh deploy <app>     │
                                          └─────────────────┘       └────────────────────────────┘
                                                                              │
                          OBSERVE / VERIFY  ◀───────────────────────────────────┘
              Sentry · OpenTelemetry · Prometheus · Loki · health/canary
              Alertmanager email · smoke tests · Daily Environment Report
              Umami analytics · Lighthouse · axe-core a11y · static-smoke audit

The cluster & deploys

The Platform is hosted on a self-managed Kubernetes cluster running K3s on Hetzner — a lightweight, single-binary Kubernetes distribution well-suited to a small, owner-run fleet. Everything ships into one namespace, dloizides.

A separate staging cluster exists alongside production. Its main job is to run the nightly test load against production — keeping that load off the production node while still exercising the real production endpoints.

Deploys follow a deliberately boring, reproducible recipe:

(the registry is never exposed to the public internet).

not by a mutable tag like :latest or :prod.

1. Build a container image locally / in CI.
2. Push it to a private registry that is only reachable over a WireGuard VPN
3. Update the Kubernetes manifest to reference the image by its sha256 digest,
4. Apply via the manage.sh deploy <app> script.

Why digest pinning? A tag is mutable — myapp:prod can point at a different image tomorrow, so "what's running" is ambiguous and rollbacks are guesswork. A digest is the content hash of the exact image bytes; pinning to it means the manifest is an exact, reproducible record of what is deployed and a rollback is just "pin back to the previous digest."

# Deploy a single app (builds/pushes if needed, then pins the digest in the manifest)
manage.sh deploy katalogos-web

# A digest-pinned container reference looks like:
#   registry.internal/katalogos-web@sha256:<digest>
# NOT:
#   registry.internal/katalogos-web:prod      <-- mutable, avoid

Traefik is the ingress controller. It terminates TLS (certificates from Let's Encrypt), routes by host to the right Service, and applies middleware (e.g. security headers, path rewrites, host-routed passthroughs for custom domains).

Local development with Tilt

Locally, Tilt is the single orchestrator. Instead of every developer juggling a pile of npm run dev, dotnet watch, and jest --watch processes by hand, Tilt owns them all: dev servers, test watchers, container rebuilds, and one resource for every check in the pipeline (lint, typecheck, unit tests, prod build, security audit, each E2E suite).

The critical design choice is that Tilt resources use MANUAL trigger mode. A rebuild happens only when you explicitly trigger it. This avoids two failure modes that plague file-watcher setups: runaway rebuild storms (a save triggers a chain of rebuilds) and the multi-gigabyte RAM cost of many always-on watchers. Developers (and AI agents) interact with Tilt rather than spawning their own servers, so there is exactly one shared instance of each resource.

# The loop is "trigger a named resource, then read its result"
tilt trigger frontend-lint
tilt trigger erevna-web-typecheck
tilt trigger questioner-unit-tests
tilt trigger katalogos-web-prod-build

Because there is a resource per check, "did my change pass?" becomes a series of explicit triggers rather than guesswork — and the same resource names are what CI and agents use, so local and pipeline behaviour stay aligned.

CI/CD

GitHub Actions runs the CI/CD pipelines. The same logical checks you trigger locally in Tilt — lint, typecheck, unit tests, production build, security/dependency audit, and selected E2E suites — run in Actions on push/PR. CI is the gate that protects the main branch; the build-push-deploy step then produces the digest-pinned image that manage.sh deploy rolls out.

The division of labour is roughly:

End-to-end testing with Playwright

E2E coverage is Playwright, organised as one suite per product/area rather than one giant monolith. Suites include, among others:

• health — is the surface reachable / authenticated at all
• identity — login, signup, session, password flows
• questioner / Erevna — surveys & forms product
• online-menus / Katalogos — online-menus product
• content — file upload / content API
• notifications — in-app + email notification flows
• billing — subscription & pricing
• themes — tenant theming
• cross-product isolation — one tenant/product can't see another's data
• a11y — axe-core accessibility (see Quality gates below)

Each suite can run in three places:

suites against production (from the staging cluster, so production isn't burdened with running its own tests).

1. On a dev PC — npx playwright test ... for tight local iteration.
2. Via Tilt — a Tilt resource per suite, so it's part of the same trigger workflow.
3. In-cluster — as a Kubernetes Job, plus a nightly CronJob that runs the

Two practices keep the suites trustworthy:

reproduction before it's "fixed," so fixes target the real cause, not the symptom.

hammering shared signup/login endpoints in parallel trips rate limits and produces false failures. Isolation removes that class of flake.

• Repro-first — a flaky or failing test is reduced to a minimal, reliable
• Run in isolation — suites are run independently rather than all-at-once, because

# Local, isolated, one area at a time (avoids cross-suite rate-limit flakes)
npx playwright test --project=questioner-active
npx playwright test --project=online-menus-crud-lifecycle

# Or as a Tilt resource:
tilt trigger playwright-e2e-health

Observability — logs, metrics, traces, errors

Observability is built on the standard pillars, each with a dedicated tool:

ErrorBoundary** in the frontend — so an uncaught React error surfaces as a reported event with context, not a silent white screen.

as it hops across microservices — essential for finding which hop is slow or failing.

use); Loki aggregates logs for querying.

signal independent of user traffic.

a human is paged for the things that matter rather than drowning in noise.

• Sentry captures crash/error reports across **backends, web apps, and an
• OpenTelemetry provides distributed traces, so a single request can be followed
• Prometheus scrapes metrics (request rates, latencies, error ratios, resource
• Health checks and a canary give a fast "is it up / is the happy path working"
• Alertmanager sends critical-alert emails when something crosses a threshold, so

The mental model: metrics tell you something is wrong, traces tell you where, logs tell you what exactly, and Sentry gives you the error with a stack trace.

Smoke tests & the Daily Report

Pillars tell you about traffic that is happening; smoke tests proactively prove the deployed services are actually up even when no one is using them. Scheduled smoke tests hit the deployed surfaces and report success/failure.

A Daily Environment Report email then summarises overall health — pulling together the health/smoke signals into one digest the owner reads each day. Importantly it escalates email-send failures: if the platform's own outbound email is failing, that drives the report's overall health to Warning or Critical and surfaces it prominently (because a silent email outage would otherwise hide the very channel the report uses).

# Verify deployed services are up, on demand
manage.sh smoke-test production

# Trigger the Daily Environment Report on demand
manage.sh daily-report production

Quality gates — Lighthouse & accessibility

Every public web app must clear a mandatory baseline before it counts as "done." These are not suggestions; a cron static-smoke audit periodically checks deployed apps and flags any that drift from the standards.

catching regressions in load performance before users feel them.

rules (contrast, labels, roles) across public pages so accessibility is regression-tested like any other behaviour, not audited once and forgotten.

and raises a flag the moment one drifts (e.g. a missing analytics snippet, a broken sitemap.xml, or a Lighthouse drop).

• Lighthouse runs against the built app and enforces the ≥ 80 performance floor —
• axe-core runs as a dedicated a11y Playwright project, asserting on accessibility
• The static-smoke cron continuously re-checks deployed apps against all of the above

Analytics

Analytics is self-hosted Umami, served from an analytics subdomain — so visitor data stays on the platform's own infrastructure rather than a third-party tracker. It is consent-gated (no tracking until the visitor agrees). In addition to pageviews, Web Vitals (Core Web Vitals / performance signals) are reported into analytics per app, so real-user performance is observable alongside traffic — complementing the lab-measured Lighthouse score with field data.