Observability & SLOs · TL;DR

1 min read
Senior14 min read
Rapid overview

TL;DR

Observability is the ability to answer a new question about a running system — "why are checkout requests from Android slow since 14:10?" — from the telemetry it already emits. Three signals do the work: metrics (cheap numeric aggregates, good for dashboards and alerts), logs (detailed events, good for what exactly happened), and traces (one request's path across services, good for where the time went). RED (rate, errors, duration) describes a service from its users' side; USE (utilisation, saturation, errors) describes a resource. Metrics cost grows with cardinality, the number of label combinations, so a user id in a label can cost more than the system it watches. Reliability targets are set as SLIs (a ratio of good events), SLOs (a target for that ratio) and an error budget (the failures allowed), and alerts fire on the burn rate of that budget — on user pain, not on every CPU spike.

See also