Networking essentials

Every box in a design is joined to another by a network. The network is slower than memory, slower than disk, and it fails in ways the code on either side cannot see. You do not need to know every protocol. You need to know what a request costs, which protocol fits which traffic, and what to do when a request goes missing.

Crux: every network hop costs time and can fail, so count the round trips, pick the simplest protocol that fits the traffic, and assume any request may be lost, slow or delivered twice.

What a request costs

Before a browser can show the first byte of a page, it usually does four things in order. Each one waits for the one before it.

• 1. DNS lookup. Turn a name (example.com) into an IP address.
• 2. TCP handshake. Open a connection to that address.
• 3. TLS handshake. Agree on keys, so the traffic is encrypted and the server is proven to be the right one.
• 4. HTTP request. Ask for the page and read the answer.

The lesson from the figure is about connections. Opening one is expensive, using an open one is cheap. That is why servers keep connections alive, clients reuse them, and a load balancer or gateway often holds long-lived connections to the servers behind it.

Distance sets a floor

Signals in optical fibre travel at roughly two thirds of the speed of light, about 200,000 km per second. That gives a floor that no software can beat.

1,000 km   ÷ 200,000 km/s  =  5 ms one way    →  ~10 ms round trip
10,000 km  ÷ 200,000 km/s  = 50 ms one way    →  ~100 ms round trip

Real routes are longer than a straight line and add routers, so real numbers are higher. The point for a design: a user on another continent pays about 100 ms per round trip no matter how fast your server is. To fix that, move data closer (a CDN, a regional deployment), or ask for fewer round trips.

Bandwidth and latency are different things

• Latency is how long one trip takes.
• Bandwidth is how much fits through per second.

A big file is limited by bandwidth. Many small requests are limited by latency. Arithmetic for a 1 GB file on a 1 Gbit/s link:

1 GB = 8 Gbit.  8 Gbit ÷ 1 Gbit/s  =  8 seconds   (1 Gbit/s is about 125 MB/s)

The layers you need

You will hear of seven layers. Four ideas carry almost every discussion.

• IP moves packets from one address to another. It is best effort: a packet can be lost, repeated or arrive out of order.
• TCP builds a reliable, ordered stream on top of IP. It numbers the data, resends what is lost and slows down when the network is congested. The price is the handshake and a delay when a packet is lost, because later data waits.
• UDP sends packets with no promises. It is quick and simple, and the application decides what to do about loss. Live video, voice calls and games use it because a late packet is worth less than a new one.
• HTTP is the request and reply format almost every API uses, on top of TCP and TLS.

Which transport to name? Default to TCP. Pick UDP only when a late answer is worthless (live voice, games, telemetry that is sampled anyway), and say how the application copes with loss: skip it, or resend only what matters. Do not claim UDP "because it is faster" without that second half.

Two newer versions are worth one line each. HTTP/2 sends many requests at once over a single TCP connection. HTTP/3 runs over QUIC, which is built on UDP, so one lost packet no longer holds up unrelated requests and connections start faster.

DNS

DNS maps a name to one or more IP addresses. Answers are cached on the way (in the browser, the operating system and the resolver) for a time set by the TTL (time to live) on the record.

• Use it for spreading traffic and routing users to a nearby region: the same name can return different addresses.
• Careful: a long TTL means a changed address takes a long time to reach everyone. Do not count on DNS for fast failover.

Choosing a protocol

Start with plain HTTP request and reply. Move only when the traffic has a shape that HTTP does not fit.

The options people mix up are the ones for pushing data to the client:

• Polling. The client asks every few seconds. Simple, works everywhere, wasteful. Arithmetic: 1 million clients polling every 10 seconds is 100,000 requests per second, most of which return "nothing new".
• Long polling. The client asks and the server holds the request open until it has something. Fewer empty replies, still one request per message.
• Server-sent events (SSE). One long HTTP response that the server keeps writing to. One direction only (server to client). Good for feeds and notifications.
• WebSocket. A long-lived two-way connection. Use it when the client also sends often (chat, shared editing). Cost: connections are stateful, so a load balancer must keep them on the same server and you must plan for reconnects.

• WebRTC. Peer-to-peer media between browsers, over UDP, with a server only to introduce the two sides. It is for audio and video calls. Naming it for a chat or a feed is a sign you are reaching.

For traffic between your own services, gRPC is common: a typed contract, a compact binary format and streaming. The cost is that you cannot test it with a browser or curl as easily.

Load balancers work at two levels. A layer 4 balancer routes by address and port, which suits WebSockets and raw TCP. A layer 7 balancer reads the HTTP request, so it can route by path or header. The key technologies module covers them in detail.

Design for a network that fails

A request can fail in three ways, and the caller cannot tell the last two apart.

• It never arrived. Safe to send again.
• It arrived and was processed, but the reply was lost. Sending again would do the work twice.
• It is slow. Maybe it will finish, maybe not.

So every call needs the same four habits.

• A timeout. Never wait forever. A call without a timeout turns one slow service into a stuck caller, then a stuck caller's callers.
• A circuit breaker. After enough failures in a row, stop calling the sick service for a short time and fail fast (or use a fallback), then let a trial call through to see if it has recovered. It keeps a slow dependency from eating every thread of its callers.
• Retries with backoff and jitter. Wait longer after each failure (1 s, 2 s, 4 s) and add a random amount, so a thousand clients do not retry in the same instant.
• A retry limit. Retries multiply load. If three layers each retry three times, one user action can become 3 × 3 × 3 = 27 calls on the database that is already struggling.
• Idempotency. An operation is idempotent if doing it twice has the same effect as doing it once. PUT a value and DELETE a record are. Charging a card is not, unless you send a unique idempotency key and the server remembers it. The API design lesson shows how.

Spreading traffic: load balancers and regions

One server cannot take all the traffic, and one server is a single point of failure. A load balancer sits in front of several and shares requests out.

• Layer 4 or layer 7. Layer 4 forwards connections by address and port: fast, protocol-blind, right for WebSockets and raw TCP. Layer 7 reads each HTTP request, so it can route by path, terminate TLS and retry on another server, at a higher cost per request.
• Health checks. The balancer probes each server and stops sending to one that fails. Without checks it keeps feeding a dead node.
• Algorithm. Round robin is fine when requests are short and alike. With long-lived connections (WebSockets, streams) it can pile many onto one server, so prefer least connections. For caches, hash on a key so the same object reaches the same server.
• The balancer is a component too. One instance is a single point of failure, so run a pair or use a managed one.

Distance is the other lever. Run copies of the service in more than one region and send each user to the nearest (DNS or an anycast address does the steering). Put static and cacheable content on a CDN. Cost of regions: data now lives in several places, so you must decide which data is local to a region (a user's profile, partitioned by home region) and which is global (and so pays the cross-region delay or a consistency trade-off, see the CAP lesson).

Common mistakes

• Ignoring distance: a design with six sequential cross-region calls costs about 6 × 100 ms = 600 ms before any work is done.
• No timeouts, or timeouts longer than the caller's own.
• Retrying without backoff, which turns a small outage into a large one.
• Using WebSocket where server-sent events or plain polling would do, and then having to scale stateful connections.
• Treating DNS as instant failover.
• Opening a new connection for every request.
• Retrying a non-idempotent write, or retrying at every layer at once.
• Round robin in front of long-lived connections, so one server ends up with most of them.
• One load balancer, and no thought for what happens when it dies.

Check yourself

• What are the four steps before the first byte of a page, and which can be skipped on the second request?
• Why can a user in another continent not get under about 100 ms per round trip, whatever you do?
• When would you choose UDP over TCP?
• Polling, SSE and WebSocket: which one fits a live score ticker, and which fits a chat?
• Why do retries need backoff, jitter and a limit?
• What does a circuit breaker do that a timeout alone does not?
• Layer 4 or layer 7 for WebSockets, and why? What replaces round robin for long connections?
• What makes an operation safe to retry, and how do you make a payment safe to retry?