Load balancers

One server has a ceiling and a single point of failure. A load balancer lets many identical servers act as one, and removes the broken ones from rotation.

Crux: a load balancer gives you horizontal scale and failover for stateless servers. Choose layer 4 for raw speed and long-lived connections, layer 7 for routing by request content, and always run the balancer redundantly.

What it is and why it exists

A load balancer (LB) accepts traffic at one address and forwards each request or connection to one of many backends. It gives you:

• Scale: add servers to add capacity.
• Availability: unhealthy servers stop receiving traffic.
• Easy deploys: drain a server, update it, put it back.

It works best when servers are stateless, meaning any server can handle any request, with session data kept in a shared store such as Redis or in a token.

How it works, at the level you must explain

Layer 4 and layer 7

The numbers come from the network layers model. The only part you must know is what each one can see.

• Layer 4 (transport). Sees IP addresses and ports, not the HTTP content. It forwards TCP or UDP connections. It is very fast and cheap, and holds long-lived connections such as WebSockets naturally. It cannot route by URL.
• Layer 7 (application). Reads HTTP: path, host, headers, cookies. It can route /api and /images to different server groups, terminate TLS, rewrite headers, retry, and apply limits. It costs more per request.

A common setup is a layer 4 balancer at the very front for speed and failover, with layer 7 routing behind it.

Choosing a backend

• Round robin: each server in turn. Simple, fine when requests cost about the same.
• Weighted round robin: bigger servers get more.
• Least connections (or least load): send to the one with the fewest open connections. Better when requests vary a lot in duration.
• Hash-based: hash the client IP or a request key, so the same client reaches the same server. A consistent hash limits reshuffling when servers are added or removed. Useful for caches and stateful servers.
• Random with two choices: pick two at random, use the less loaded. Cheap and surprisingly effective.

Health checks

The LB probes each server (a TCP connect, or an HTTP GET /health). After a few failures it stops sending traffic, and after successes it returns. Make the health endpoint check what matters, and keep it cheap. A check that depends on a shared database can take every server out at once if that database slows down.

Sticky sessions

Sticky sessions pin a client to one server, with a cookie or an IP hash. Avoid them if you can: they unbalance load, and the user loses state when that server dies. Prefer a shared session store. They are justified for stateful connections such as WebSockets, where the connection is the state anyway.

Making the balancer reliable

The LB is a single point of failure unless you remove that.

• Run two or more, behind one virtual IP that fails over (active-passive), or behind DNS or anycast that spreads clients (active-active).
• Cloud load balancers do this for you, and scale automatically.
• DNS load balancing returns different IPs for one name. It is coarse, since caching delays changes, so it is often used to pick a region and not a server.
• Global load balancing sends users to the nearest healthy region.
• Client-side balancing: the client holds a list of servers and picks one, as in gRPC and service-to-service calls. It removes a hop. A service mesh can do it for you.

Where to draw them

Draw one at the front, and one more only where a group of servers sits behind another (for example between an app tier and an internal service). Do not put one before every box.

Smoother deploys and failures

• Connection draining: before removing a server, stop sending it new requests but let in-flight ones finish. A deploy then drops nobody.
• Slow start: a new or recovered server gets a small share first. Otherwise least-connections sends it everything at once, and a cold server falls over.
• Health check tuning: require several failed checks before removing a server and several passes before returning it. One-shot checks flap, and a flapping server hurts more than a slow one.
• Retries: the balancer can retry a failed request on another server. Do this only for safe (idempotent) requests, and cap it, or a struggling backend gets a retry storm.
• Overload: a balancer can shed load with a quick error (or a queue limit) instead of passing everything to servers that cannot keep up.

TLS and the two layers

Layer 7 balancers usually terminate TLS: they decrypt, read the request, then talk to backends over a private network (or re-encrypt). Layer 4 balancers can pass the encrypted stream through, so the backend holds the certificate. Pick by where the certificate should live and whether you need to read the request.

Autoscaling and capacity

Autoscaling adds servers when a metric (CPU, requests per server) crosses a threshold, and the balancer starts sending them traffic after their health check passes. Scaling takes minutes, so keep headroom for a spike. Size by arithmetic: peak requests per second divided by what one server sustains, then add one or two servers so losing one still leaves enough capacity.

Balancing across regions

Global balancing picks a region by DNS, anycast or latency, and fails over when a region is unhealthy. Inside each region, a regional balancer does the rest. Plan the failover capacity: if one of three regions fails, the other two receive its traffic, so each must have room for about half again its normal load.

When to use it, trade-offs, numbers

• What: a component that distributes requests over identical servers and removes failed ones.
• Use when: more than one server does the same job.
• Know: layer 4 versus 7, algorithms, health checks, stickiness and its cost, redundancy.
• Careful: it does not fix a slow database, it adds a hop, and it does not by itself make stateful servers safe to scale.

Which tool?

• Cloud managed (AWS ALB for layer 7, NLB for layer 4): the default choice on AWS.
• NGINX, HAProxy, Envoy: self-run, flexible, also usable as an API gateway or reverse proxy.
• API gateway: adds auth, limits and API-level rules in front of or alongside it.

Numbers: I am leaving out per-instance connection and throughput limits. They vary by product and I cannot give reliable figures. If numbers matter, estimate your peak requests per second and divide by what one server handles, then add headroom for failures.

Common mistakes

• Drawing a load balancer before every service.
• Using layer 7 for WebSockets and not thinking about long-lived connections.
• Sticky sessions as a substitute for a shared session store.
• A health check that is too heavy or depends on a shared dependency.
• Forgetting that the load balancer needs its own redundancy.

• Removing servers abruptly in a deploy, dropping in-flight requests.
• Sending full traffic to a freshly started server.
• Retrying non-idempotent calls through the balancer.
• No spare capacity for losing a server or a region.

Check yourself

• What does a layer 4 balancer see that a layer 7 one does not, and the reverse?
• When would you choose least connections over round robin?
• Why are stateless servers easier to balance?
• What is wrong with a health check that queries the main database?
• How do you avoid the load balancer being a single point of failure?
• Why might WebSockets suit a layer 4 balancer?
• What is connection draining?
• Why use slow start?
• If one of three regions fails, how much extra load does each of the others take?