Networking Essentials for System Design · How it works

9 min read
Mid-level10 min read
Rapid overview

How it works

Latency has a floor, and it is not a tuning problem

Light travels about 200,000 km/s in fibre. A round trip from London to Sydney is roughly 34,000 km of cable, which is about 170ms of pure propagation before any processing. No amount of faster hardware changes this.

Round tripApproximate latency
Same datacentre0.5 ms
Same region, different availability zone1–2 ms
Cross-continent (e.g. US East → Europe)80–100 ms
Antipodal (e.g. UK → Australia)150–200 ms

The design consequence is that you count round trips, not milliseconds. A request that makes five sequential cross-region calls has bought 500ms of latency that cannot be recovered. This is why the answer to "our European users see slow page loads" is usually to move the data or the computation closer to them, not to make the queries faster.

Q: Why can't you fix cross-region latency by optimising your application code?

A: Because most of it is propagation delay, which is a function of distance and the speed of light in fibre, not of how fast your code runs. A UK-to-Australia round trip is roughly 170ms of pure signal travel time, so even an instantaneous server still yields a 170ms response. The only real levers are reducing the number of round trips, moving the data or computation geographically closer to the user via replicas or edge caching, or removing the cross-region dependency from the synchronous path entirely by making it asynchronous. This is why "count the round trips" is the useful analytical habit: a design with three sequential cross-region hops has a latency floor of about 500ms regardless of implementation quality, and no profiling exercise will find it because the time is not being spent in your process.

TCP, UDP, and why the difference shows up in design

TCP gives you an ordered, reliable, connection-oriented byte stream. It costs a handshake before any data flows, and it enforces ordering, which means a lost packet stalls everything behind it until it is retransmitted — head-of-line blocking.

UDP gives you none of those guarantees and therefore none of those costs. It is the right choice when late data is worthless: live voice and video, where retransmitting a 200ms-old audio frame is pointless because the moment has passed, and real-time gaming for the same reason.

Q: When would you deliberately choose UDP over TCP, and what do you take on by doing so?

A: I would choose UDP when data that arrives late is worse than data that never arrives — live audio and video, real-time game state, and high-volume metrics or telemetry where losing a sample is acceptable. TCP's retransmission and in-order delivery actively hurt these: a dropped audio packet retransmitted 200ms later cannot be played because that moment has passed, and worse, TCP's ordering guarantee means it blocks the newer packets behind it. What I take on is that I must handle in the application anything I still need: sequence numbers if I care about ordering or detecting loss, my own acknowledgement scheme if some messages are important, congestion awareness so I do not flood the network, and typically an encryption layer such as DTLS. So the trade is real work in exchange for removing latency I cannot otherwise remove.

HTTP versions, briefly, and the one thing that matters

HTTP/1.1 allows one in-flight request per connection, so browsers open six or so connections per host and developers used to concatenate assets to reduce request count. HTTP/2 introduced multiplexing — many concurrent streams over one TCP connection — which removed that need, but because it still runs over TCP, a single lost packet stalls every stream sharing that connection. HTTP/3 moves to QUIC over UDP, giving each stream independent loss recovery, which matters most on lossy mobile networks.

The design-relevant summary: with HTTP/2 or HTTP/3, request count is much less costly than it used to be, so "batch everything into one giant endpoint" is no longer automatically the right instinct.

Load balancing at layer 4 versus layer 7

This distinction comes up constantly and is worth being precise about.

A layer 4 load balancer routes on IP and port. It does not read the request, so it is fast and protocol-agnostic, and it can forward anything — but it cannot make a decision based on a URL path, a header, or a cookie.

A layer 7 load balancer terminates the connection and reads the HTTP request. That lets it route /api/ to one pool and /images/ to another, retry a failed idempotent request against a different backend, terminate TLS, and inject headers. It costs more CPU and adds a little latency.

# A layer-7 routing decision: only possible because the proxy reads the path.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-routes
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /v1/search
            pathType: Prefix
            backend:
              service:
                name: search-service
                port:
                  number: 8080
          - path: /v1/orders
            pathType: Prefix
            backend:
              service:
                name: orders-service
                port:
                  number: 8080
Q: What can a layer 7 load balancer do that a layer 4 one cannot?

A: It can make routing decisions based on the content of the request, because it terminates the connection and parses the HTTP layer rather than only seeing IP addresses and ports. That enables path- and host-based routing to different backend pools, routing on headers or cookies for A/B tests and canary releases, TLS termination so backends serve plain HTTP, header injection and rewriting, response compression, and — importantly — safe retries, since it understands request boundaries and can resend an idempotent request to a healthy backend when one fails. The cost is CPU and a small latency addition from parsing, plus it must handle the protocol correctly, whereas a layer 4 balancer just forwards packets and so is faster and works for any protocol including raw TCP and UDP.

Health checks and the failure they usually hide

A load balancer removes a backend from rotation when its health check fails, which means the health check's definition of "healthy" determines the system's behaviour under partial failure. A check that only confirms the process is listening will keep sending traffic to an instance whose database connection pool is exhausted. A check that verifies every downstream dependency creates the opposite problem: a brief blip in one dependency fails every instance's check simultaneously and the load balancer removes the entire fleet, converting a degradation into a total outage.

The usual resolution is two distinct checks. Liveness asks "is this process wedged and in need of a restart?" and must not depend on anything external. Readiness asks "should this instance receive traffic right now?" and may consider local conditions such as warm-up state or queue depth, but should be cautious about hard-failing on shared dependencies.

Q: Why is it dangerous for a readiness check to fail when a shared downstream dependency is unavailable?

A: Because the dependency is shared, every instance evaluates the same condition and fails at the same moment, so the load balancer removes the entire fleet from rotation at once. The result is that a partial degradation — where the service could still serve cached reads, health endpoints, or the subset of routes that do not touch that dependency — becomes a complete outage returning nothing at all. It also makes recovery worse, because when the dependency returns, every instance passes its check simultaneously and the full load lands on a cold, unwarmed fleet, which can immediately re-fail. The better pattern is for readiness to reflect only conditions local to that instance, and to handle dependency failures inside the request path with timeouts, circuit breakers, and degraded responses, so the blast radius matches the actual damage.

Getting data to the client: four options

This is one of the most reliably asked questions, and the correct answer is almost never "WebSockets" without qualification.

ApproachHow it worksBest when
Short pollingClient requests on a timerUpdates are rare, simplicity matters, minutes of delay acceptable
Long pollingServer holds the request open until data or timeoutOccasional updates, need near-real-time, want plain HTTP
Server-Sent EventsOne long-lived HTTP response, server pushes text eventsServer-to-client only, e.g. notifications, live scores, progress
WebSocketsFull-duplex persistent connectionGenuinely bidirectional and frequent, e.g. chat, collaborative editing, games

The reason to resist WebSockets by default is that a persistent stateful connection is operationally expensive. Every connection consumes server memory and a file descriptor for its whole life, load balancers must be configured for long-lived upgrades, and a deploy or scale-down drops every connection so you need reconnection with backoff and state resynchronisation. If the data only flows one way, Server-Sent Events gets you the same user experience over ordinary HTTP, with automatic browser reconnection built in.

# SSE is just an HTTP response that never ends. You can watch it with curl:
curl -N -H "Accept: text/event-stream" https://api.example.com/v1/notifications/stream

# The wire format is deliberately trivial:
#   event: order_shipped
#   data: {"orderId":"A-1934","carrier":"DHL"}
#   id: 88213
#
# The client sends Last-Event-ID on reconnect, so the server can resume.
Q: Why would you choose Server-Sent Events over WebSockets for a notification feed?

A: Because notifications flow only from server to client, and SSE provides exactly that over plain HTTP while WebSockets adds bidirectional capability I would not use plus real operational cost. SSE runs over a normal HTTP response, so it passes through existing proxies, load balancers, and CDNs without special upgrade handling, works with standard HTTP authentication and compression, and the browser's EventSource implements automatic reconnection with a Last-Event-ID header so the server can resume from where the client left off. WebSockets requires protocol-upgrade support in every hop, needs its own heartbeat and reconnection logic because the browser gives you none, and its bidirectional channel invites protocol design that HTTP semantics would otherwise handle. I would switch to WebSockets when the client genuinely needs to send frequent messages too — chat with typing indicators, collaborative editing, multiplayer game input — because then the second direction is doing real work.

Q: What is head-of-line blocking, and where does it appear in HTTP?

A: It is when the first item in a queue cannot be processed and everything behind it waits, even though the later items are ready. In HTTP it appears at two distinct levels. At the application level in HTTP/1.1, a connection handles one request at a time, so a slow response blocks every subsequent request on that connection — which is why browsers open several connections per host. HTTP/2 fixed that with multiplexing, but reintroduced it at the transport level: many streams share one TCP connection, and because TCP guarantees in-order delivery of the byte stream, a single lost packet halts delivery of all streams until it is retransmitted, so one dropped packet stalls unrelated requests. HTTP/3 addresses this by running over QUIC on UDP, where each stream has independent loss recovery, so a lost packet only affects its own stream. This is why HTTP/3's benefit is largest on lossy networks such as mobile.

Quick recall

Short-answer versions of the same material, for spaced repetition.

Q: What is the approximate round-trip latency between Europe and the US east coast?

A: Roughly 80 to 100 milliseconds. It is propagation delay set by distance and the speed of light in fibre, so no code change reduces it.

Q: What does a layer 7 load balancer read that a layer 4 one does not?

A: The HTTP request itself, so it can route on path, host, header, or cookie, terminate TLS, and safely retry idempotent requests.

Q: Which transport should carry live voice or video, and why?

A: UDP, because a retransmitted audio frame arrives too late to play, and TCP's ordering guarantee would also stall the newer frames behind it.

Q: What is the difference between a liveness and a readiness check?

A: Liveness asks whether the process is wedged and needs restarting. Readiness asks whether this instance should receive traffic right now.

See also