Queues & messaging · TL;DR

1 min read
Senior15 min read
Rapid overview

TL;DR

A message broker decouples the service that produces work from the service that does it: the producer can finish quickly, the consumer can fall behind and catch up, and either can fail without taking the other down. The design questions are always the same five. Which shape — a work queue (each message to one worker), pub/sub (each message to every subscriber) or a log (replayable, ordered per partition)? Which delivery guarantee — at-most-once, at-least-once, or the effectively-once you build from at-least-once plus idempotent consumers? Which ordering — none, per key, or global? How do the database write and the message publish stay consistent — the outbox pattern? What happens to a message that keeps failing — retries with backoff, then a dead-letter queue?

See also