Consistency & CAP · How it works
9 min readHow it works
Why copies disagree
A write reaches one replica first and the others later. Between those moments, a read from a lagging replica returns old data. Replication lag is usually milliseconds, but under load, during a failover, or across regions it can be seconds or minutes. Every consistency decision is a decision about what users may observe in that window.
A: The delay between a write being committed on the leader and being applied on a follower. It is normally milliseconds, so users rarely notice, but it grows under write bursts, long-running transactions on the follower, network trouble and cross-region links. It becomes visible when a request writes to the leader and the next request reads from a follower: the user saves a profile, the page reloads from a replica, and the old value appears. That is a read-your-writes violation, and it is the most common consistency bug in replicated systems.
Replication topologies
- Single-leader (Postgres, MySQL, most managed databases): all writes go to one node; followers replicate its log. Simple, supports strong consistency on the leader, but the leader is a write bottleneck and a failover point.
- Multi-leader (multi-region active-active, collaborative editors, offline clients): several nodes accept writes and exchange them. Low write latency everywhere, but concurrent writes to the same key conflict and need a resolution rule.
- Leaderless (Dynamo, Cassandra, Riak): the client writes to several replicas and reads from several; overlapping quorums give tunable consistency.
A: Write conflicts. Two leaders can accept different writes to the same record at the same time, and neither is wrong from its own point of view. The system needs a conflict resolution rule: last-writer-wins by timestamp (simple, silently drops a write), application-level merge, or conflict-free replicated data types (CRDTs) that merge mathematically. Single-leader avoids it because one node orders every write to a key.
Synchronous vs asynchronous replication
- Synchronous: the leader waits for a follower to confirm before acknowledging. No data loss on leader failure, but a slow or dead follower stalls writes.
- Asynchronous: the leader acknowledges immediately. Fast, but a leader crash loses writes that had not shipped.
- Semi-synchronous: wait for one follower, the rest async. The common production compromise.
A: Every write the old leader acknowledged but had not yet shipped to the follower that gets promoted. Clients were told those writes succeeded, yet the new leader never saw them. If the old leader later comes back and rejoins, its extra writes conflict with the new history and are usually discarded. That is why systems that cannot lose acknowledged writes use synchronous or semi-synchronous replication to at least one replica, or a consensus protocol.
Failover
Failover hazards: the timeout is a guess (too short causes needless failovers during a GC pause, too long extends the outage); async replication loses the tail of writes; and if the old leader is not fenced, both nodes accept writes — split-brain.
A: Two nodes both believe they are the leader and both accept writes, so history diverges. It happens when the old leader was only unreachable, not dead, and keeps serving clients that can still reach it. Prevention is fencing: each leadership term has a monotonically increasing epoch or fencing token, storage and downstream services reject writes carrying an older epoch, and a leader that cannot confirm it still holds a majority lease steps down. STONITH-style fencing (power off the old node) is the blunt version.
CAP, stated correctly
CAP says: when a network partition splits the nodes, a system must either refuse some requests (stay consistent) or answer them with possibly stale or conflicting data (stay available). It is not "pick two of three" in normal operation, since partitions are not optional. Consistency here means linearizability, and availability means every non-failed node answers.
- CP examples: ZooKeeper, etcd, a single-leader database that rejects writes on the minority side.
- AP examples: Cassandra and DynamoDB in their default modes, DNS, shopping carts that merge later.
A: Because partition tolerance is not optional in a distributed system: networks do partition, so the real choice only exists during a partition, between consistency and availability. Outside a partition a system can be both consistent and available. The slogan also hides that C in CAP means linearizability specifically and A means every non-failed node responds, so many real systems are neither strictly CP nor AP, and many choose per operation.
PACELC
PACELC extends CAP: if Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The "else" half is the one you pay every day: a linearizable write in a multi-region cluster waits for a cross-region quorum on every request, partition or not.
| System | During partition | Normal operation |
|---|---|---|
| DynamoDB, Cassandra default | A | L |
| Spanner, etcd | C | C |
| MongoDB default | C | L-ish, depends on read preference |
| Postgres with async replicas | C on leader | L when reading replicas |
A: It names the trade-off that exists when there is no partition: lower latency or stronger consistency. To be linearizable, a write must reach a quorum and a read must confirm it is current, which costs round trips, and across regions each one is tens to hundreds of milliseconds. Systems that choose latency answer from the nearest replica and accept staleness. CAP only talks about the rare partition; PACELC describes the cost every request pays.
Consistency models, strongest to weakest
- Linearizable: once a write completes, every later read sees it; the system behaves like one copy. Needed for locks, leader election, unique constraints, balances.
- Sequential: all nodes see operations in the same order, but that order need not match wall-clock time.
- Causal: if write B depends on write A (a reply to a comment), everyone sees A before B; unrelated writes may appear in different orders.
- Session guarantees: per-client promises — read-your-writes, monotonic reads (never go back in time), monotonic writes, writes-follow-reads.
- Eventual: no ordering promise, only convergence.
A: They answer different questions. Serializability is a transaction isolation property: concurrent transactions produce a result equal to some serial order, but that order need not match real time. Linearizability is a recency property of single objects: once a write completes, every later read returns it. A database can be serializable but not linearizable (a snapshot read from a stale replica can still be serializable), and the combination of both is called strict serializability, which Spanner provides.
A: Because it can be enforced without coordinating with remote replicas on each operation: a replica only needs to delay showing a write until it has shown the writes that write depended on, which it can do with metadata (version vectors, dependency lists) carried with the write. Linearizability, in contrast, requires contacting a majority, which the minority side of a partition cannot do. Causal consistency preserves the orderings users actually notice — a reply never appears before its question — while still letting each side of a partition accept writes.
Session guarantees in practice
Read-your-writes is what users complain about when it is missing. Ways to provide it on a leader-plus-replicas setup:
- Read from the leader for a short window after the user writes (for example 10 seconds, tracked in the session).
- Carry the write's log position (LSN) in the session and only read from a replica that has applied it.
- Pin a user's reads to the same replica (sticky sessions) for monotonic reads.
- Read anything the user can edit from the leader; read everything else from replicas.
A: Make the read path aware of the user's last write. The simplest version sends that user's reads to the leader for a short window after a write. A more precise version records the log sequence number returned by the write in the session or a cookie, and routes reads only to replicas whose applied position is at least that LSN, falling back to the leader otherwise. Either way, other users can still read from replicas, so the read-scaling benefit mostly survives.
Conflict resolution
- Last-writer-wins (LWW): highest timestamp wins. Simple, but clock skew picks the wrong winner and the losing write disappears silently.
- Version vectors: detect that two writes were concurrent and keep both as siblings for the application to merge.
- CRDTs: data types (counters, sets, maps, sequences) whose merge is commutative, associative and idempotent, so replicas converge regardless of order.
- Application merge: domain rules, such as "a shopping cart keeps the union of items".
A: It silently discards data. Two concurrent writes each think they succeeded, and the one with the lower timestamp vanishes; with clock skew between nodes, it may be the later write that loses. It is acceptable when writes are idempotent overwrites of the whole value and losing a concurrent update is harmless — a user's last-seen timestamp, a cache entry, a presence status — and unacceptable for counters, balances, or anything that accumulates.
Multi-region
Three common shapes:
- Single write region, read replicas elsewhere: simple, consistent writes, but remote users pay cross-region latency on every write and a region loss means a failover.
- Home region per user or tenant (geo-partitioning): each record has one home region that owns its writes; most requests stay local. Good for data residency.
- Active-active multi-leader: every region writes locally; conflicts need CRDTs or LWW; best latency and availability, hardest correctness.
A: Start from what needs strong consistency and where users are. If most writes belong to one user or tenant, geo-partitioning gives each record a home region, so writes are local and consistent without cross-region conflicts. If writes are rare compared with reads, a single write region with local read replicas is simplest, and remote users pay latency only on writes. Active-active is justified when every region must keep writing through a region outage and the data can merge safely (counters, carts, presence); for balances, inventory or unique constraints it needs a consensus-based database such as Spanner or CockroachDB, and every write pays a cross-region quorum.