Consensus & coordination ยท How it works
9 min readHow it works
Why agreement is hard
Messages can be delayed, lost or reordered; processes can pause (garbage collection, VM migration) for seconds and then resume thinking nothing happened; clocks drift. A node cannot tell a crashed peer from a slow one. The FLP result proves that no deterministic protocol can guarantee agreement in a fully asynchronous system with even one faulty process. Practical protocols escape by using timeouts: they are always safe (never disagree) and are live (make progress) whenever the network behaves well enough for long enough.
A: Because a crashed peer and a slow peer look identical from the outside: both stop answering. The peer may be paused in garbage collection, its network link may be congested, or its reply may be queued, and any timeout is a guess. Declaring it dead too early risks two nodes acting as leader; waiting too long extends outages. Consensus protocols are built so that a wrong guess costs availability but never correctness โ a node wrongly declared dead cannot make decisions without a majority.
Quorums
With N nodes, a majority is โN/2โ + 1. Two majorities always share at least one node, so information from one decision reaches the next. A cluster of 2f + 1 nodes tolerates f failures: 3 nodes survive 1, 5 survive 2. Even sizes add cost without adding tolerance โ 4 nodes still survive only 1.
A: Because fault tolerance depends on how many nodes can fail while a majority remains, and adding one node to an odd-sized cluster does not change that. Three nodes need two for a majority and tolerate one failure; four need three and still tolerate only one. The fourth node adds cost, network traffic and another machine that can fail, without improving availability. It also makes a two-two network split leave neither side with a majority.
Raft in three parts
- Leader election: nodes start as followers. A follower that hears no heartbeat within a randomised election timeout (for example 150โ300 ms) becomes a candidate, increments the term, votes for itself and requests votes. A node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own. A majority of votes makes a leader.
- Log replication: clients send commands to the leader; the leader appends them to its log and sends them to followers. An entry is committed once a majority has stored it; then every node applies it to its state machine in log order.
- Safety: a newer term always wins; a leader from an old term that hears of a higher term steps down. Because a new leader needs votes from a majority, and committed entries are on a majority, the new leader always has every committed entry.
A: To avoid split votes. If every follower timed out at the same moment, they would all become candidates in the same term, each vote for itself, and no one would win a majority; the same would repeat every round. Randomising the timeout within a range means one node usually times out first, requests votes before the others become candidates, and wins, so elections settle in one round most of the time.
A: When the leader of the current term has replicated it to a majority of nodes. Any future leader must win votes from a majority, and a node only votes for a candidate whose log is at least as up to date as its own; since the two majorities overlap, at least one voter holds the committed entry and would refuse a candidate missing it. So every future leader already has every committed entry, and nothing acknowledged to a client is lost.
What a consensus service gives you
etcd (Raft), ZooKeeper (ZAB) and Consul (Raft) expose consensus as a small, strongly consistent key-value store with primitives:
- Linearizable reads and writes and compare-and-set.
- Leases / ephemeral nodes that disappear when the owning session stops heartbeating.
- Watches that notify clients of changes.
- Monotonic revision numbers usable as fencing tokens.
On top of these you build leader election, service discovery, configuration, and locks. Keep the data small (megabytes, not gigabytes) and the write rate modest โ every write goes through the leader and a quorum.
A: Because correct consensus is subtle and the failure cases are rare enough that bugs survive testing: split votes, stale leaders after pauses, lost acknowledgements, and membership changes. A consensus service has been verified in production and often by formal methods or Jepsen, and it exposes the primitives you need โ linearizable compare-and-set, leases that expire with the session, watches and monotonic revisions. Your service then only needs to campaign for a key and respect the lease, which is far less code to get wrong.
Distributed locks and the pause problem
A lock service grants a lease-based lock: the holder owns it until the lease expires. The danger:
Client 1 believes it still holds the lock after its pause. Without a check at the storage, both clients write. The fix is a fencing token: the lock service returns a number that increases with each grant, the client sends it with every write, and the protected resource rejects tokens older than the newest it has seen. The lock gives efficiency; the fencing check gives correctness.
A: Because the holder cannot know when its lease has expired if it pauses. A process can acquire a lock, stall for longer than the lease in garbage collection, a VM pause or a slow disk, then resume and act as if it still holds the lock โ while another client has legitimately acquired it. Checking the time before acting does not help, since the pause can happen after the check. Correctness needs the protected resource to participate: the lock hands out monotonically increasing fencing tokens and the resource rejects writes carrying an older token.
A: An efficiency lock prevents duplicated work โ two workers generating the same report โ and occasional double execution is harmless, so a simple lease in Redis is fine. A correctness lock protects an invariant โ two writers must never modify an account concurrently โ and a violation corrupts data, so it needs a consensus-backed lock service plus fencing tokens checked by the resource. Deciding which kind you need up front tells you how much machinery is justified.
Leases and leader election in applications
Common pattern for "only one instance runs the scheduler":
- Each instance tries to create a key (
/scheduler/leader) with a lease, using compare-and-set so only one succeeds. - The winner keeps the lease alive with heartbeats and does the work; the others watch the key.
- If the leader dies or cannot renew, the lease expires, the key disappears, and a watcher takes over.
- The leader stops working as soon as renewal fails โ before the lease could have expired on the server โ and passes its revision as a fencing token to anything it writes.
A: Stop acting as leader immediately, not when it believes 10 seconds have passed. Its clock and the server's may differ, and the failure may mean the server already considers the lease expired. It should stop issuing leader-only actions, cancel in-flight work where possible, and keep trying to re-acquire leadership as a normal candidate. Downstream writes should carry its fencing token so that anything still in flight after a new leader starts is rejected.
Split-brain
Split-brain is two nodes acting as leader at once. It happens when a leader is cut off from the majority but still reachable by some clients. Majority quorums prevent it for the consensus log itself (the minority side cannot commit), but systems built on top can still split: an old leader continues writing to an external database, or a failover tool promotes a replica while the old primary is merely partitioned. Defences: majority-based leadership, leader step-down when it loses contact with a majority, fencing tokens at every resource, and STONITH for shared storage.
A: Raft prevents two leaders from committing to the Raft log, but an application leader also acts on external systems. If the old leader is partitioned from the Raft cluster but still connected to the database, it may not yet know it has lost leadership and continues writing while a new leader, elected on the majority side, also writes. The fix is to carry the leadership term or revision as a fencing token on every external write and have the external system reject older tokens, and to make the old leader step down as soon as it cannot renew its lease.
Consensus vs two-phase commit
Two-phase commit (2PC) coordinates a transaction across different participants (prepare, then commit), where every participant must agree. Consensus replicates one state across identical replicas, where a majority is enough. 2PC blocks if the coordinator fails after prepare โ participants hold locks until it returns โ unless the coordinator itself is replicated with consensus, which is what Spanner does.
A: They solve different problems. Consensus (Raft, Paxos) makes a group of replicas of the same data agree on the next value, and needs only a majority to proceed, so it survives minority failures. Two-phase commit makes different participants โ say two database shards โ either all commit or all abort one transaction, and needs every participant's yes, so any single participant or the coordinator failing at the wrong moment blocks it. Production distributed databases combine them: each participant is a consensus group, and 2PC runs across groups.