Scaling & Sharding Study Portal

A: Because every write must still be applied by the single primary, and replicas are downstream consumers of that same write stream rather than additional write capacity. In fact adding replicas makes the write path slightly worse: the primary has to ship the log to more destinations, consuming network and CPU it would otherwise spend on writes, and with synchronous replication it must also wait for acknowledgements. So a system saturated on writes gets no relief and marginally more load. The mechanisms that do address write throughput are reducing the write volume itself — batching, coalescing repeated updates, moving high-frequency low-value writes such as counters into a cache or a stream and aggregating before persisting — or partitioning the data so that different writes are handled by different primaries, which is sharding. It is worth being precise about this in an interview because "add replicas" is a common reflex answer to any scaling question, and identifying that the constrained resource is write capacity rather than read capacity is the actual insight.

A: The write went to the primary and the subsequent read was served by an asynchronous replica that had not yet applied it, so the user is shown a version of the world from a moment before their own change — the read-your-own-writes anomaly. It is particularly damaging because it looks exactly like a failed save, so users retry, which doubles the write load and can create duplicate records if the operation is not idempotent. The cheapest fix is to route reads to the primary for a bounded window, typically a few seconds, after that user performs a write, using a session flag or a cookie; it is simple and correct, at the cost of some extra primary load. A more precise approach is to capture the replication position at the time of the write, store it in the session, and require any replica serving that user to have applied at least that position, falling back to the primary if none has — this gives correctness without sending all post-write reads to the primary. A third option, which is often the best user experience, is to update the client optimistically from the response of the write itself rather than re-fetching, so no read is needed at all. I would also alert on lag, because a lag large enough for users to notice usually indicates a long-running transaction or write saturation rather than normal operation.

A: Because it determines which queries stay efficient and which become fan-outs, and it is extremely expensive to change once data is distributed — altering it means rewriting essentially every row into a new placement, usually with dual-writes and a long backfill while the system stays live. A well-chosen key means the overwhelming majority of queries include it and therefore touch exactly one shard, which is what makes the system fast. A poorly-chosen one means common queries lack it, so each has to be broadcast to every shard and merged in the application, and that pattern gets worse with every shard added — the opposite of scaling. The key also determines distribution: if it has low cardinality or skewed values, some shards receive far more data and traffic than others, and a single-shard bottleneck limits the whole system no matter how many shards exist. And it determines what can be transactional, since atomicity is normally only available within a shard, so entities that must change together should share a key. This is why the honest interview answer starts from the access patterns and the entities that need atomicity, then derives the key, rather than picking something evenly distributed and discovering the query implications later.

A: It solves the remapping problem in hash(key) % N: because the node count is the divisor, changing it changes the result for nearly every key, so adding or losing a single node reshuffles almost the entire keyspace. For a cache that means an instantaneous collapse of the hit rate and a full-load stampede onto the origin; for a sharded datastore it means moving nearly all the data. Consistent hashing places nodes and keys on a ring and assigns each key to the next node clockwise, so adding or removing a node only reassigns the keys in that node's arc — about 1/N of the keyspace — while every other key stays exactly where it was. Virtual nodes fix the distribution weakness of the basic scheme: with a single point per node, the randomly sized arcs mean some nodes own several times more of the keyspace than others, so load is uneven no matter how many nodes you add. Giving each node many positions — typically one to two hundred — averages the arc sizes so ownership is close to even, spreads a departing node's keys across many remaining nodes rather than dumping them all on one neighbour, and allows weighting by assigning more positions to more capable hardware.

A: Row count is not load. One shard is receiving a disproportionate share of traffic — typically because it holds a single very high-volume tenant, a celebrity account, or the current time range in a time-partitioned scheme where all new writes target the newest partition. Even key distribution says nothing about request distribution, so the metric to look at is operations and bytes per shard, not size. The fixes depend on the cause. If a single large tenant is responsible, the answer is either to give that tenant its own dedicated shard, which is straightforward with directory-based routing, or to change the key so that tenant's data is itself split — a composite of tenant plus something high-cardinality within it, accepting that queries for the whole tenant now fan out. If it is a hot individual entity, the same key-splitting logic applies at entity level. If it is time-based skew, the leading component of the key must not be the timestamp; prefixing with a hash or a bucket spreads new writes across shards, at the cost of making time-range scans a fan-out. In all cases I would check first whether the traffic is legitimate, because a hot shard is also what a misbehaving client or a missing cache looks like, and adding shards to absorb a bug is expensive.

A: No. Every write is still applied by one primary, and shipping the log to more replicas adds load. Only sharding scales writes.

A: Almost all of them, because the node count is the divisor. Going from ten nodes to eleven reassigns roughly ninety percent of the keyspace.

A: About one over N, because only the arc belonging to the added or removed node changes owner.

A: Even distribution, by giving each physical node many ring positions so arc sizes average out. They also allow weighting larger machines.

A: The dual write. It records the event in the same database transaction as the data, so a crash cannot leave one without the other.

A: A user writes to the primary then reads from a lagging replica and sees their old data, which looks to them like a failed save.