Networking Essentials for System Design · How it works
9 min read- How it works
- Latency has a floor, and it is not a tuning problem
- TCP, UDP, and why the difference shows up in design
- HTTP versions, briefly, and the one thing that matters
- Load balancing at layer 4 versus layer 7
- Health checks and the failure they usually hide
- Getting data to the client: four options
- Quick recall
How it works
Latency has a floor, and it is not a tuning problem
Light travels about 200,000 km/s in fibre. A round trip from London to Sydney is roughly 34,000 km of cable, which is about 170ms of pure propagation before any processing. No amount of faster hardware changes this.
| Round trip | Approximate latency |
|---|---|
| Same datacentre | 0.5 ms |
| Same region, different availability zone | 1–2 ms |
| Cross-continent (e.g. US East → Europe) | 80–100 ms |
| Antipodal (e.g. UK → Australia) | 150–200 ms |
The design consequence is that you count round trips, not milliseconds. A request that makes five sequential cross-region calls has bought 500ms of latency that cannot be recovered. This is why the answer to "our European users see slow page loads" is usually to move the data or the computation closer to them, not to make the queries faster.
A: Because most of it is propagation delay, which is a function of distance and the speed of light in fibre, not of how fast your code runs. A UK-to-Australia round trip is roughly 170ms of pure signal travel time, so even an instantaneous server still yields a 170ms response. The only real levers are reducing the number of round trips, moving the data or computation geographically closer to the user via replicas or edge caching, or removing the cross-region dependency from the synchronous path entirely by making it asynchronous. This is why "count the round trips" is the useful analytical habit: a design with three sequential cross-region hops has a latency floor of about 500ms regardless of implementation quality, and no profiling exercise will find it because the time is not being spent in your process.
TCP, UDP, and why the difference shows up in design
TCP gives you an ordered, reliable, connection-oriented byte stream. It costs a handshake before any data flows, and it enforces ordering, which means a lost packet stalls everything behind it until it is retransmitted — head-of-line blocking.
UDP gives you none of those guarantees and therefore none of those costs. It is the right choice when late data is worthless: live voice and video, where retransmitting a 200ms-old audio frame is pointless because the moment has passed, and real-time gaming for the same reason.
A: I would choose UDP when data that arrives late is worse than data that never arrives — live audio and video, real-time game state, and high-volume metrics or telemetry where losing a sample is acceptable. TCP's retransmission and in-order delivery actively hurt these: a dropped audio packet retransmitted 200ms later cannot be played because that moment has passed, and worse, TCP's ordering guarantee means it blocks the newer packets behind it. What I take on is that I must handle in the application anything I still need: sequence numbers if I care about ordering or detecting loss, my own acknowledgement scheme if some messages are important, congestion awareness so I do not flood the network, and typically an encryption layer such as DTLS. So the trade is real work in exchange for removing latency I cannot otherwise remove.
HTTP versions, briefly, and the one thing that matters
HTTP/1.1 allows one in-flight request per connection, so browsers open six or so connections per host and developers used to concatenate assets to reduce request count. HTTP/2 introduced multiplexing — many concurrent streams over one TCP connection — which removed that need, but because it still runs over TCP, a single lost packet stalls every stream sharing that connection. HTTP/3 moves to QUIC over UDP, giving each stream independent loss recovery, which matters most on lossy mobile networks.
The design-relevant summary: with HTTP/2 or HTTP/3, request count is much less costly than it used to be, so "batch everything into one giant endpoint" is no longer automatically the right instinct.
Load balancing at layer 4 versus layer 7
This distinction comes up constantly and is worth being precise about.
A layer 4 load balancer routes on IP and port. It does not read the request, so it is fast and protocol-agnostic, and it can forward anything — but it cannot make a decision based on a URL path, a header, or a cookie.
A layer 7 load balancer terminates the connection and reads the HTTP request. That lets it route /api/ to one pool and /images/ to another, retry a failed idempotent request against a different backend, terminate TLS, and inject headers. It costs more CPU and adds a little latency.
# A layer-7 routing decision: only possible because the proxy reads the path.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-routes
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
spec:
rules:
- host: api.example.com
http:
paths:
- path: /v1/search
pathType: Prefix
backend:
service:
name: search-service
port:
number: 8080
- path: /v1/orders
pathType: Prefix
backend:
service:
name: orders-service
port:
number: 8080
A: It can make routing decisions based on the content of the request, because it terminates the connection and parses the HTTP layer rather than only seeing IP addresses and ports. That enables path- and host-based routing to different backend pools, routing on headers or cookies for A/B tests and canary releases, TLS termination so backends serve plain HTTP, header injection and rewriting, response compression, and — importantly — safe retries, since it understands request boundaries and can resend an idempotent request to a healthy backend when one fails. The cost is CPU and a small latency addition from parsing, plus it must handle the protocol correctly, whereas a layer 4 balancer just forwards packets and so is faster and works for any protocol including raw TCP and UDP.
Health checks and the failure they usually hide
A load balancer removes a backend from rotation when its health check fails, which means the health check's definition of "healthy" determines the system's behaviour under partial failure. A check that only confirms the process is listening will keep sending traffic to an instance whose database connection pool is exhausted. A check that verifies every downstream dependency creates the opposite problem: a brief blip in one dependency fails every instance's check simultaneously and the load balancer removes the entire fleet, converting a degradation into a total outage.
The usual resolution is two distinct checks. Liveness asks "is this process wedged and in need of a restart?" and must not depend on anything external. Readiness asks "should this instance receive traffic right now?" and may consider local conditions such as warm-up state or queue depth, but should be cautious about hard-failing on shared dependencies.
A: Because the dependency is shared, every instance evaluates the same condition and fails at the same moment, so the load balancer removes the entire fleet from rotation at once. The result is that a partial degradation — where the service could still serve cached reads, health endpoints, or the subset of routes that do not touch that dependency — becomes a complete outage returning nothing at all. It also makes recovery worse, because when the dependency returns, every instance passes its check simultaneously and the full load lands on a cold, unwarmed fleet, which can immediately re-fail. The better pattern is for readiness to reflect only conditions local to that instance, and to handle dependency failures inside the request path with timeouts, circuit breakers, and degraded responses, so the blast radius matches the actual damage.
Getting data to the client: four options
This is one of the most reliably asked questions, and the correct answer is almost never "WebSockets" without qualification.
| Approach | How it works | Best when |
|---|---|---|
| Short polling | Client requests on a timer | Updates are rare, simplicity matters, minutes of delay acceptable |
| Long polling | Server holds the request open until data or timeout | Occasional updates, need near-real-time, want plain HTTP |
| Server-Sent Events | One long-lived HTTP response, server pushes text events | Server-to-client only, e.g. notifications, live scores, progress |
| WebSockets | Full-duplex persistent connection | Genuinely bidirectional and frequent, e.g. chat, collaborative editing, games |
The reason to resist WebSockets by default is that a persistent stateful connection is operationally expensive. Every connection consumes server memory and a file descriptor for its whole life, load balancers must be configured for long-lived upgrades, and a deploy or scale-down drops every connection so you need reconnection with backoff and state resynchronisation. If the data only flows one way, Server-Sent Events gets you the same user experience over ordinary HTTP, with automatic browser reconnection built in.
# SSE is just an HTTP response that never ends. You can watch it with curl:
curl -N -H "Accept: text/event-stream" https://api.example.com/v1/notifications/stream
# The wire format is deliberately trivial:
# event: order_shipped
# data: {"orderId":"A-1934","carrier":"DHL"}
# id: 88213
#
# The client sends Last-Event-ID on reconnect, so the server can resume.
A: Because notifications flow only from server to client, and SSE provides exactly that over plain HTTP while WebSockets adds bidirectional capability I would not use plus real operational cost. SSE runs over a normal HTTP response, so it passes through existing proxies, load balancers, and CDNs without special upgrade handling, works with standard HTTP authentication and compression, and the browser's EventSource implements automatic reconnection with a Last-Event-ID header so the server can resume from where the client left off. WebSockets requires protocol-upgrade support in every hop, needs its own heartbeat and reconnection logic because the browser gives you none, and its bidirectional channel invites protocol design that HTTP semantics would otherwise handle. I would switch to WebSockets when the client genuinely needs to send frequent messages too — chat with typing indicators, collaborative editing, multiplayer game input — because then the second direction is doing real work.
A: It is when the first item in a queue cannot be processed and everything behind it waits, even though the later items are ready. In HTTP it appears at two distinct levels. At the application level in HTTP/1.1, a connection handles one request at a time, so a slow response blocks every subsequent request on that connection — which is why browsers open several connections per host. HTTP/2 fixed that with multiplexing, but reintroduced it at the transport level: many streams share one TCP connection, and because TCP guarantees in-order delivery of the byte stream, a single lost packet halts delivery of all streams until it is retransmitted, so one dropped packet stalls unrelated requests. HTTP/3 addresses this by running over QUIC on UDP, where each stream has independent loss recovery, so a lost packet only affects its own stream. This is why HTTP/3's benefit is largest on lossy networks such as mobile.
Quick recall
Short-answer versions of the same material, for spaced repetition.
A: Roughly 80 to 100 milliseconds. It is propagation delay set by distance and the speed of light in fibre, so no code change reduces it.
A: The HTTP request itself, so it can route on path, host, header, or cookie, terminate TLS, and safely retry idempotent requests.
A: UDP, because a retransmitted audio frame arrives too late to play, and TCP's ordering guarantee would also stall the newer frames behind it.
A: Liveness asks whether the process is wedged and needs restarting. Readiness asks whether this instance should receive traffic right now.