Queues & messaging · How it works
9 min readHow it works
Three shapes
- Work queue (SQS, RabbitMQ queue, Celery): each message is handled by exactly one of the competing workers and then deleted. Use for jobs: resize an image, send an email.
- Pub/sub (SNS, RabbitMQ fanout exchange, Google Pub/Sub): each subscriber gets its own copy. Use for events that several services react to independently:
OrderPlaced. - Partitioned log (Kafka, Kinesis, Pulsar): messages are appended to partitions and retained; each consumer group tracks its own offset and can replay. Ordered within a partition. Use for event streams, change data capture, and anything that needs replay.
A: When you need replay, per-key ordering at high throughput, or many independent consumers reading the same stream at their own pace. A log keeps messages after they are consumed, so a new service can read history, and a bug fix can reprocess a day of events by resetting an offset. A queue deletes a message once it is acknowledged and is better for task distribution with per-message acknowledgement, delays, priorities and flexible routing. Rule of thumb: events that are facts go on a log; jobs that are commands go on a queue.
The request path, with a broker in the middle
Two acknowledgements define the guarantees: the broker's ack to the producer (the message is stored) and the consumer's ack to the broker (the message is done).
Delivery guarantees
- At-most-once: the consumer acks before processing (or the producer does not retry). A crash loses the message. Fine for metrics samples and presence pings.
- At-least-once: the consumer acks after processing; the producer retries until it gets an ack. A crash between processing and ack causes a redelivery, so the same message can be processed twice. The default for almost everything.
- Exactly-once: impossible end-to-end across independent systems in general. What systems offer is effectively-once: at-least-once delivery plus deduplication or idempotent processing, or a transaction that spans the read offset and the write within one system (Kafka transactions for read-process-write inside Kafka).
A: Because the consumer cannot atomically do its side effect and tell the broker it did it: if it crashes after the effect but before the ack, the broker must choose between redelivering (possible duplicate) or not (possible loss), and it cannot tell which case happened. What systems provide is effectively-once processing: at-least-once delivery combined with idempotent consumers or deduplication by message id, so a duplicate delivery has no additional effect. Kafka's exactly-once semantics are this idea applied inside Kafka, by committing consumed offsets and produced messages in one transaction.
Idempotent consumers
A consumer is idempotent if processing the same message twice has the same effect as once. Techniques:
- Natural idempotency: set operations (
status = shipped) instead of increments. - Dedup table: record each processed message id in the same database transaction as the effect; skip ids already present. Expire old ids after the maximum redelivery window.
- Idempotency keys on outbound calls: pass the message id to the payment provider so a retried charge is recognised.
- Version checks: apply an update only if the record's version is older than the event's.
A: Give every charge a stable idempotency key derived from the message — usually the order or payment id, not a random value generated per attempt — and send it to the payment provider, which returns the original result on a repeat instead of charging again. Locally, record the processed message id and the outcome in the same transaction that updates the order, so a redelivery finds the record and acks without calling the provider. A key generated fresh on each attempt defeats the purpose.
The dual-write problem and the outbox pattern
A service that writes to its database and then publishes an event has two writes to two systems. If it crashes between them, the database says the order exists and no one else ever hears about it — or, publishing first, others hear about an order that was rolled back.
The transactional outbox: write the event into an outbox table in the same transaction as the business change. A relay — a poller or change data capture such as Debezium reading the database log — publishes outbox rows to the broker and marks them sent. The relay can crash and republish, so delivery is at-least-once and consumers must be idempotent.
A: A service must change its database and notify other services, and there is no transaction spanning the database and the broker, so a crash between the two leaves them inconsistent: either an event for data that was never committed, or committed data that was never announced. The outbox pattern turns two writes into one: the event is inserted into an outbox table in the same local transaction as the business change, so both commit or neither does. A separate relay reads the outbox and publishes, retrying until the broker acks. It trades immediate publication for guaranteed eventual publication, and it makes delivery at-least-once.
A: The consumer-side mirror. The consumer records each incoming message id in an inbox table in the same transaction as the effect it causes, and skips messages whose id is already there. The outbox guarantees the event is published at least once; the inbox guarantees it is applied at most once; together they give effectively-once processing between two services without a distributed transaction.
Ordering
- A single queue with many competing consumers does not preserve order: message 2 can finish before message 1.
- A log preserves order within a partition. Partition by the key whose events must be ordered (order id, account id), and ordering holds per key while throughput scales across partitions.
- Retries break ordering: if message 1 fails and goes to a retry queue, message 2 for the same key may be processed first. Either block the key until 1 succeeds, or make handlers tolerant (version numbers, ignore older events).
- Global ordering means one partition and one consumer — a throughput ceiling. It is rarely needed.
A: Partition by order id. In a log such as Kafka, all events with the same key land in the same partition, and a partition is consumed by one member of a consumer group at a time, so events for one order are processed in sequence while different orders are processed in parallel across partitions. With queues, use the equivalent feature — SQS FIFO message groups or consistent-hash exchanges in RabbitMQ. The cost is that one slow or hot key delays everything behind it in its partition.
Retries, backoff and dead-letter queues
- Transient failures (timeouts, 503s, lock contention) are retried with exponential backoff and jitter: 1 s, 2 s, 4 s… plus randomness so thousands of consumers do not retry in lockstep.
- Permanent failures (malformed payload, validation error, a bug) will never succeed; retrying them wastes capacity and blocks ordered partitions. They go straight to a dead-letter queue (DLQ).
- A message that fails repeatedly is a poison message. A max-attempts limit stops it from looping forever.
- A DLQ needs an owner, an alert on depth, and a redrive tool to replay messages after the fix. A DLQ nobody reads is a data-loss mechanism with extra steps.
A: A message that fails every time it is processed — malformed data, a payload the code cannot handle, or a bug triggered by specific content. Without a limit, it is redelivered forever, burning capacity, and on an ordered partition it blocks every message behind it. Stop it with a maximum delivery count after which the broker moves it to a dead-letter queue, classify errors so permanent failures are dead-lettered immediately instead of after all retries, and alert on DLQ depth so someone fixes the cause and redrives.
A: Without jitter, every consumer that failed at the same moment — because a dependency went down — retries at exactly the same moments: all at 1 s, all at 2 s, all at 4 s. Each retry wave hits the recovering dependency together and can knock it over again. Randomising each delay (full jitter picks uniformly between zero and the backoff cap) spreads the retries out, so the dependency sees a smooth trickle it can absorb.
Backpressure and consumer lag
Queues absorb bursts, but they do not create capacity. If producers outpace consumers for long enough, the queue grows until messages expire, memory runs out, or latency becomes unacceptable. Watch consumer lag (messages or seconds behind), autoscale consumers on it, cap queue length or reject at the edge when it is too deep, and remember that a log's partition count caps its parallelism.
A: The distance between the newest message produced and the last message a consumer group has processed, measured in messages or, more usefully, in seconds of delay. It directly measures user-visible staleness (how long until an order confirmation email goes out) and whether consumers are keeping up. Steadily rising lag means capacity is short or a consumer is stuck; a lag spike at one partition points to a hot key or a poison message. Autoscaling and alerting should use lag, not CPU.
Choosing messaging vs a direct call
Use a synchronous call when the caller needs the answer to continue (price check before checkout). Use a message when the work can happen later, may be slow, fans out to several services, or must survive the downstream being down (send receipt, update search index, notify warehouse).
A: When the caller needs the result to respond to the user, when the work is fast and the downstream is reliable, or when the team cannot operate the extra infrastructure. A queue adds latency, a new failure mode (the broker), duplicate handling, and asynchronous error reporting — a user who submitted a form now needs a way to learn it failed later. Putting a request-reply RPC over a queue usually combines the downsides of both.