Observability & SLOs Study Portal

A: Monitoring checks for conditions you anticipated — known metrics, known thresholds, known dashboards — and tells you that something is wrong. Observability is the property that lets you investigate conditions you did not anticipate, by slicing rich telemetry by arbitrary dimensions (customer, region, version, endpoint) and following a request across services. A system can be well monitored and still hard to debug if its telemetry cannot be broken down along the dimension the new problem lives in.

A: So that you can jump between signals. A latency alert points at a slow trace; the trace shows which service and span were slow; the trace id then retrieves every log line written for that request across all services, including the error messages and parameters that metrics and spans do not carry. Without the shared id, correlating logs across services means guessing by timestamp, which fails under concurrency.

A: Because latency distributions are skewed and the average hides the tail. A service where 95% of requests take 20 ms and 5% take 2 seconds has an average around 120 ms, which describes no real request. Users feel the tail: a page that makes 20 backend calls hits the p95 of each call often. Report p50, p95 and p99 (and track the maximum or p99.9 for critical paths), computed from histograms so they can be aggregated across instances — averaging per-instance percentiles is mathematically wrong.

A: Because every distinct label value creates a separate time series that the metrics system must index, store and query. A user id with millions of values multiplies the series count by millions, exhausting the memory of Prometheus-style systems or generating a large bill in hosted ones, and making queries slow. Per-user detail belongs in traces or structured logs, which store attributes per event; metrics should use bounded labels such as route, status class, region or plan.

A: Head-based sampling decides at the start of a request whether to record its trace, typically a fixed percentage, and propagates the decision downstream. It is cheap and simple but blind: a rare error is as likely to be dropped as a boring success. Tail-based sampling buffers all spans of a trace until it completes, then decides — keeping every error, every slow request, and a small share of normal ones. It captures the interesting traces but needs a collector tier with memory to hold in-flight traces.

A: The producer injects the current trace context into the message headers or attributes, exactly as it would into HTTP headers, and the consumer extracts it when it processes the message and starts its span as a child of, or a link to, the producer's span. Because the consumer may run seconds or hours later, and one batch may contain messages from many traces, span links are often used rather than a strict parent-child relationship. Without this, every consumer starts a new unrelated trace and the request's path stops at the queue.

A: An SLI is the measurement — the fraction of valid requests that were good, for a definition of good such as "succeeded within 300 ms". An SLO is the internal target for that measurement over a window, such as 99.9% over 28 days. An SLA is an external contract, usually with financial penalties, promising a level of service to customers; it is set looser than the SLO so the team is alerted and acts before a contractual breach.

A: It turns reliability from an argument into a number both product and engineering agreed on. If the SLO is 99.9%, then 0.1% of requests may fail, and that allowance can be spent on releases, experiments and migrations. While budget remains, the team ships at full speed; when a period's budget is exhausted, the agreed policy kicks in — freeze risky launches, prioritise reliability work — until it recovers. It also argues against over-investing: reliability far above the SLO is budget left unspent that could have bought velocity.

A: Burn rate is the ratio between the current error rate and the error rate the SLO allows. At a 99.9% SLO the allowed rate is 0.1%, so a 1.44% error rate is a burn rate of 14.4 — if it continued, the 30-day budget would be gone in about two days. Alerting on burn rate ties pages directly to the promise made to users: a brief spike that uses little budget does not page, a sustained moderate problem does, and the thresholds mean the same thing on every service regardless of its traffic.

A: Because causes are many and often harmless, while symptoms are what users feel. CPU at 95%, one replica down or a queue growing may self-heal or be absorbed by redundancy; paging on each trains on-call to ignore pages. An SLO-based symptom alert — checkout success below target, latency budget burning — fires only when users are affected and covers causes nobody anticipated. Cause-based signals still belong on dashboards and in low-urgency tickets to aid diagnosis.

A: Name the user journeys and their SLOs first — for example 99.9% of checkouts succeed within 1 second over 28 days — and say alerts page on error-budget burn rate. Then list RED metrics for every service and endpoint, USE and saturation metrics for the stateful parts (database connections, cache hit ratio, queue consumer lag), distributed tracing with OpenTelemetry propagated across HTTP and messages with tail sampling to keep errors and slow requests, and structured logs carrying the trace id. Finish with the one or two metrics specific to this design's riskiest component.