Consensus & coordination ยท How it works

9 min read
Senior15 min read
Rapid overview

How it works

Why agreement is hard

Messages can be delayed, lost or reordered; processes can pause (garbage collection, VM migration) for seconds and then resume thinking nothing happened; clocks drift. A node cannot tell a crashed peer from a slow one. The FLP result proves that no deterministic protocol can guarantee agreement in a fully asynchronous system with even one faulty process. Practical protocols escape by using timeouts: they are always safe (never disagree) and are live (make progress) whenever the network behaves well enough for long enough.

Q: Why can a node not reliably tell whether a peer has crashed?

A: Because a crashed peer and a slow peer look identical from the outside: both stop answering. The peer may be paused in garbage collection, its network link may be congested, or its reply may be queued, and any timeout is a guess. Declaring it dead too early risks two nodes acting as leader; waiting too long extends outages. Consensus protocols are built so that a wrong guess costs availability but never correctness โ€” a node wrongly declared dead cannot make decisions without a majority.

Quorums

flowchart LR subgraph Cluster[Five nodes] N1[N1] N2[N2] N3[N3] N4[N4] N5[N5] end W[Write quorum - N1 N2 N3] --> N1 W --> N2 W --> N3 R[Read quorum - N3 N4 N5] --> N3 R --> N4 R --> N5 N3 -. overlap guarantees the read sees the write .-> R

With N nodes, a majority is โŒŠN/2โŒ‹ + 1. Two majorities always share at least one node, so information from one decision reaches the next. A cluster of 2f + 1 nodes tolerates f failures: 3 nodes survive 1, 5 survive 2. Even sizes add cost without adding tolerance โ€” 4 nodes still survive only 1.

Q: Why do consensus clusters usually have an odd number of nodes?

A: Because fault tolerance depends on how many nodes can fail while a majority remains, and adding one node to an odd-sized cluster does not change that. Three nodes need two for a majority and tolerate one failure; four need three and still tolerate only one. The fourth node adds cost, network traffic and another machine that can fail, without improving availability. It also makes a two-two network split leave neither side with a majority.

Raft in three parts

  1. Leader election: nodes start as followers. A follower that hears no heartbeat within a randomised election timeout (for example 150โ€“300 ms) becomes a candidate, increments the term, votes for itself and requests votes. A node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own. A majority of votes makes a leader.
  2. Log replication: clients send commands to the leader; the leader appends them to its log and sends them to followers. An entry is committed once a majority has stored it; then every node applies it to its state machine in log order.
  3. Safety: a newer term always wins; a leader from an old term that hears of a higher term steps down. Because a new leader needs votes from a majority, and committed entries are on a majority, the new leader always has every committed entry.
sequenceDiagram participant A as Node A participant B as Node B participant C as Node C Note over A,C: Leader of term 4 has crashed A->>A: Election timeout fires, term 5, vote for self A->>B: RequestVote term 5 A->>C: RequestVote term 5 B-->>A: Vote granted Note over A: 2 of 3 votes - leader for term 5 A->>B: AppendEntries heartbeat term 5 A->>C: AppendEntries heartbeat term 5 C-->>A: Ack, C learns term 5 and becomes follower
Q: Why does Raft randomise election timeouts?

A: To avoid split votes. If every follower timed out at the same moment, they would all become candidates in the same term, each vote for itself, and no one would win a majority; the same would repeat every round. Randomising the timeout within a range means one node usually times out first, requests votes before the others become candidates, and wins, so elections settle in one round most of the time.

Q: When is an entry committed in Raft, and why is that enough to survive a leader crash?

A: When the leader of the current term has replicated it to a majority of nodes. Any future leader must win votes from a majority, and a node only votes for a candidate whose log is at least as up to date as its own; since the two majorities overlap, at least one voter holds the committed entry and would refuse a candidate missing it. So every future leader already has every committed entry, and nothing acknowledged to a client is lost.

What a consensus service gives you

etcd (Raft), ZooKeeper (ZAB) and Consul (Raft) expose consensus as a small, strongly consistent key-value store with primitives:

  • Linearizable reads and writes and compare-and-set.
  • Leases / ephemeral nodes that disappear when the owning session stops heartbeating.
  • Watches that notify clients of changes.
  • Monotonic revision numbers usable as fencing tokens.

On top of these you build leader election, service discovery, configuration, and locks. Keep the data small (megabytes, not gigabytes) and the write rate modest โ€” every write goes through the leader and a quorum.

Q: Why would you use etcd or ZooKeeper instead of implementing leader election in your own service?

A: Because correct consensus is subtle and the failure cases are rare enough that bugs survive testing: split votes, stale leaders after pauses, lost acknowledgements, and membership changes. A consensus service has been verified in production and often by formal methods or Jepsen, and it exposes the primitives you need โ€” linearizable compare-and-set, leases that expire with the session, watches and monotonic revisions. Your service then only needs to campaign for a key and respect the lease, which is far less code to get wrong.

Distributed locks and the pause problem

A lock service grants a lease-based lock: the holder owns it until the lease expires. The danger:

sequenceDiagram participant C1 as Client 1 participant L as Lock service participant C2 as Client 2 participant S as Storage C1->>L: Acquire lock L-->>C1: Granted, token 33 Note over C1: Long GC pause, lease expires C2->>L: Acquire lock L-->>C2: Granted, token 34 C2->>S: Write with token 34 S-->>C2: Accepted, highest token now 34 C1->>S: Write with token 33 after pause S-->>C1: Rejected, token older than 34

Client 1 believes it still holds the lock after its pause. Without a check at the storage, both clients write. The fix is a fencing token: the lock service returns a number that increases with each grant, the client sends it with every write, and the protected resource rejects tokens older than the newest it has seen. The lock gives efficiency; the fencing check gives correctness.

Q: Why is a lock with a lease not enough to guarantee mutual exclusion?

A: Because the holder cannot know when its lease has expired if it pauses. A process can acquire a lock, stall for longer than the lease in garbage collection, a VM pause or a slow disk, then resume and act as if it still holds the lock โ€” while another client has legitimately acquired it. Checking the time before acting does not help, since the pause can happen after the check. Correctness needs the protected resource to participate: the lock hands out monotonically increasing fencing tokens and the resource rejects writes carrying an older token.

Q: What is the difference between using a lock for efficiency and using it for correctness?

A: An efficiency lock prevents duplicated work โ€” two workers generating the same report โ€” and occasional double execution is harmless, so a simple lease in Redis is fine. A correctness lock protects an invariant โ€” two writers must never modify an account concurrently โ€” and a violation corrupts data, so it needs a consensus-backed lock service plus fencing tokens checked by the resource. Deciding which kind you need up front tells you how much machinery is justified.

Leases and leader election in applications

Common pattern for "only one instance runs the scheduler":

  1. Each instance tries to create a key (/scheduler/leader) with a lease, using compare-and-set so only one succeeds.
  2. The winner keeps the lease alive with heartbeats and does the work; the others watch the key.
  3. If the leader dies or cannot renew, the lease expires, the key disappears, and a watcher takes over.
  4. The leader stops working as soon as renewal fails โ€” before the lease could have expired on the server โ€” and passes its revision as a fencing token to anything it writes.
Q: A leader holds a 10-second lease and renews every 3 seconds. What should it do when renewal fails?

A: Stop acting as leader immediately, not when it believes 10 seconds have passed. Its clock and the server's may differ, and the failure may mean the server already considers the lease expired. It should stop issuing leader-only actions, cancel in-flight work where possible, and keep trying to re-acquire leadership as a normal candidate. Downstream writes should carry its fencing token so that anything still in flight after a new leader starts is rejected.

Split-brain

Split-brain is two nodes acting as leader at once. It happens when a leader is cut off from the majority but still reachable by some clients. Majority quorums prevent it for the consensus log itself (the minority side cannot commit), but systems built on top can still split: an old leader continues writing to an external database, or a failover tool promotes a replica while the old primary is merely partitioned. Defences: majority-based leadership, leader step-down when it loses contact with a majority, fencing tokens at every resource, and STONITH for shared storage.

Q: How can split-brain still happen in a system that uses a correct Raft cluster for leader election?

A: Raft prevents two leaders from committing to the Raft log, but an application leader also acts on external systems. If the old leader is partitioned from the Raft cluster but still connected to the database, it may not yet know it has lost leadership and continues writing while a new leader, elected on the majority side, also writes. The fix is to carry the leadership term or revision as a fencing token on every external write and have the external system reject older tokens, and to make the old leader step down as soon as it cannot renew its lease.

Consensus vs two-phase commit

Two-phase commit (2PC) coordinates a transaction across different participants (prepare, then commit), where every participant must agree. Consensus replicates one state across identical replicas, where a majority is enough. 2PC blocks if the coordinator fails after prepare โ€” participants hold locks until it returns โ€” unless the coordinator itself is replicated with consensus, which is what Spanner does.

Q: What is the difference between consensus and two-phase commit?

A: They solve different problems. Consensus (Raft, Paxos) makes a group of replicas of the same data agree on the next value, and needs only a majority to proceed, so it survives minority failures. Two-phase commit makes different participants โ€” say two database shards โ€” either all commit or all abort one transaction, and needs every participant's yes, so any single participant or the coordinator failing at the wrong moment blocks it. Production distributed databases combine them: each participant is a consensus group, and 2PC runs across groups.

See also