API Design for Distributed Systems · How it works
9 min readHow it works
Choosing between REST, gRPC and GraphQL
This is a "who is calling, over what network" question, not a taste question.
| REST/JSON | Universally supported, cacheable via HTTP, trivially debuggable | Public APIs, third-party integrators, browser clients, anything cached by CDNs |
|---|---|---|
| gRPC | Binary protobuf, HTTP/2 streaming, generated typed clients, low overhead | Internal service-to-service calls, high call volume, polyglot backends |
| GraphQL | Client selects exactly the fields it needs; one round trip for a graph | Many heterogeneous clients with divergent data needs, especially mobile |
REST's underrated advantage is that it works with the HTTP caching ecosystem. A GET with a sensible Cache-Control header can be served by a CDN, a reverse proxy, and the browser without your servers being involved. GraphQL loses this almost entirely, because everything is a POST to a single endpoint and the response depends on the query body — so caching moves into your application, where you have to build it.
gRPC's cost is that it is inconvenient from a browser and opaque to ordinary HTTP tooling, which is precisely why it belongs behind your edge rather than at it.
A: Because REST identifies a resource by its URL, which is exactly the cache key the whole HTTP ecosystem already uses — CDNs, reverse proxies, and browsers can cache GET /v1/products/42 with a Cache-Control header and never involve your servers. GraphQL sends queries as a POST body to a single endpoint, so every request has the same URL and a different meaning, and POST is not cacheable by default; intermediaries have no way to know that two requests are equivalent or that a mutation invalidated a previous result. That pushes caching from free infrastructure into your application, where you need per-field or per-entity caching, a data loader layer to batch and deduplicate within a request, and your own invalidation logic. There are mitigations — persisted queries give each query a stable ID that can appear in a GET URL, and automatic persisted queries do this transparently — but they are work you would not have had to do with REST.
Pagination: why offset breaks
LIMIT 20 OFFSET 10000 has two problems. The database must scan and discard 10,000 rows to return 20, so deep pages get progressively slower. Worse, on data that changes while the user pages, offsets shift: if a row is inserted before the current page, one item moves to the next page and the user sees a duplicate; if a row is deleted, an item is skipped entirely and never seen.
Cursor-based (keyset) pagination fixes both by remembering where you were rather than how many you skipped.
-- Offset pagination: page 500 forces the database to walk 10,000 rows first.
SELECT id, title, created_at
FROM posts
WHERE feed_id = $1
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 10000;
-- Keyset pagination: uses the index to seek directly, and cannot skip or duplicate.
SELECT id, title, created_at
FROM posts
WHERE feed_id = $1
AND (created_at, id) < ($2, $3) -- the cursor from the previous page
ORDER BY created_at DESC, id DESC
LIMIT 20;
Two details make this work. The sort key must be unique, which is why id is included as a tiebreaker — with only created_at, two rows sharing a timestamp can straddle a page boundary and be lost or repeated. And the cursor should be opaque to the client, typically base64-encoded, so that clients do not build logic on its internals and you can change the sort without breaking them.
A: Offset counts rows to skip, so it is evaluated fresh against whatever the data looks like at the moment of each request. Suppose the user has read page one, items 1 to 20, and an item ranked within that first page is deleted before they request page two. Every subsequent row shifts up by one position, so what was item 21 becomes item 20 — inside the page already read — and the request for OFFSET 20 now returns what was originally item 22. Item 21 is never returned on any page, and the user has no way to know. The same mechanism in reverse causes duplicates: an insertion near the top pushes an already-seen item down into the next page. Keyset pagination avoids this because the cursor names a position in the sort order rather than a count, so inserts and deletes elsewhere do not move the boundary.
Idempotency, and why it is mandatory
When a client sends a request and the connection times out, the client cannot tell whether the server processed it. The request may have succeeded and the response been lost. The only safe options are to retry — which risks doing the operation twice — or to give up, which risks losing it.
GET, PUT and DELETE are idempotent by definition: repeating them yields the same state. POST is not, which is why payment and order-creation endpoints need an explicit mechanism: the client generates a unique key and sends it with the request, and the server records the key with the result.
// The server side of an idempotency key. The important part is that
// the key is reserved ATOMICALLY before any work happens.
public async Task<IActionResult> CreatePayment(PaymentRequest request, string idempotencyKey)
{
var existing = await _store.TryGetAsync(idempotencyKey);
if (existing is not null)
{
// Replay the original outcome verbatim — do not re-execute.
return StatusCode(existing.StatusCode, existing.Body);
}
// Atomic insert-if-absent. If this loses the race, another request
// with the same key is already in flight.
var reserved = await _store.TryReserveAsync(idempotencyKey, ttl: TimeSpan.FromHours(24));
if (!reserved)
{
return StatusCode(409, new { error = "request_in_progress" });
}
var result = await _payments.ChargeAsync(request);
await _store.CompleteAsync(idempotencyKey, result);
return Ok(result);
}
The subtle requirement is that the reservation must be atomic. A naive "check whether the key exists, then insert it" has a window in which two concurrent retries both see no key and both charge the card. The check and the reservation must be one operation — a conditional insert, or a unique constraint that the second writer violates.
A: Because it is a read followed by a write with a gap between them, and two concurrent requests carrying the same key can both perform the read before either performs the write. Both see no existing key, both conclude they are the first, and both proceed to execute the operation — so the card is charged twice, which is precisely the outcome idempotency exists to prevent. Retries make this concurrency likely rather than theoretical: a client that times out and immediately retries can easily have both requests in flight simultaneously, and a load balancer will happily route them to different instances so no in-process lock helps. The fix is to make reservation a single atomic operation — an insert relying on a unique constraint, a conditional write such as INSERT ... ON CONFLICT DO NOTHING, or a SET NX in Redis — so exactly one caller wins and the loser learns it lost and either waits or returns a conflict.
Versioning without breaking clients
Once a third party depends on your API you cannot change it, only add to it. The practical rules are narrower than the versioning debate suggests.
Adding an optional field is safe. Adding a required request field, removing a field, renaming anything, changing a type, or tightening validation are all breaking. So is changing the meaning of an existing field while keeping its name and type — which is the most dangerous kind, because nothing fails at compile time and nobody notices until the data is wrong.
URL-path versioning (/v1/, /v2/) is the most common choice because it is visible, trivially routable at layer 7, and easy to reason about in logs and caches. Header-based versioning keeps URLs stable and is arguably purer, but it is easier to get wrong operationally, since a missing header silently selects a default.
A: Because removing a field fails loudly and immediately — clients get a missing value, deserialisation errors, or obviously empty UI, so the breakage is discovered in testing or within minutes of deploy and can be rolled back. Changing what a field means while keeping its name and type produces no error anywhere: the field is present, the type matches, parsing succeeds, and every client continues to consume it while silently misinterpreting it. If amount changes from major units to minor units, or from pre-tax to post-tax, or a status value acquires a new meaning, then integrations keep working and produce wrong numbers, and by the time anyone notices, incorrect data has been written into downstream systems and possibly acted upon financially. The correct approach is always to add a new, differently named field and deprecate the old one on a published timeline, because that makes the change explicit and gives every client an unambiguous migration.
Designing errors as part of the contract
Error responses are consumed by code, so they need structure. A useful shape has a stable machine-readable code that clients branch on, a human-readable message that you are free to reword, and enough detail to act — plus, critically, an indication of whether retrying is sensible.
The distinction that matters most operationally is between errors the client caused and errors the server caused, because it determines who is paged and whether a retry helps. A 400 means the request was wrong and retrying it unchanged will fail identically. A 503 or 429 means try again later, ideally with Retry-After so the client does not have to guess and does not retry aggressively enough to prolong the outage.
A: A stable machine-readable code that the client can branch on, kept distinct from the human-readable message so you can reword the message without breaking anyone's conditional logic. An appropriate HTTP status, because that is what proxies, retry libraries, and monitoring interpret automatically. A clear signal about retryability — ideally an explicit flag or a Retry-After header on 429 and 503 — since the single most important thing a client must decide is whether to retry, and guessing produces either lost work or a retry storm that deepens the outage. Field-level detail for validation failures, so the caller can show the user which input was wrong rather than a generic failure. And a correlation or request ID that also appears in your logs, so a user can report a failure and support can find the exact request. What must not be included is internal detail — stack traces, SQL, or hostnames — which leaks implementation and is a security problem.
Quick recall
Short-answer versions of the same material, for spaced repetition.
A: GET, PUT, DELETE, and HEAD. POST is not, which is why side-effecting POST endpoints need an explicit idempotency key.
A: Offset counts rows to skip, so an insert or delete earlier in the sort order shifts the boundary. Users then see duplicates or miss items entirely.
A: The cursor names a position in the sort order rather than a count, so inserts and deletes elsewhere do not move the page boundary.
A: 202 Accepted, with a Location header pointing at a job resource the client can poll or subscribe to.