ZooKeeper and coordination

Distributed systems keep needing small answers that everyone agrees on: who is the leader, which servers are alive, what is the current config. Getting that right with plain databases or Redis is hard. A coordination service exists to do just that.

Crux: ZooKeeper is a small, replicated, strongly consistent store for coordination data. Use it for leader election, membership and locks, never for bulk data.

What it is and why it exists

Apache ZooKeeper is a coordination service. Behind it sits a small cluster, called an ensemble, of an odd number of servers (3 or 5 is typical). They agree on every write using a consensus protocol (ZAB, close to Paxos and Raft in spirit). One server is the leader and takes all writes. The others are followers and can serve reads.

It is built for correctness first. It is not built for storage volume or high write rates.

How it works, at the level you must explain

Data model: znodes

Data is a tree of znodes, like a file system. Each holds a small amount of data (up to about 1 MB, and meant to be far smaller) and can have children.

Three node features do most of the work.

• Ephemeral nodes exist only while the client's session is alive. If the client dies, the node disappears. That is a built-in "is this process still alive" signal.
• Sequential nodes get an increasing number appended to their name. That gives a global ordering.
• Watches let a client ask to be told once when a node changes. Clients react to change instead of polling.

Consensus and quorum

A write is committed when a majority (quorum) of the ensemble has it. With 3 servers you tolerate 1 failure. With 5 you tolerate 2. An even count adds cost with no gain in tolerance, so use odd numbers. If a majority is lost, writes stop. ZooKeeper chooses consistency over availability when partitioned. Reads from a follower can lag slightly unless the client asks to sync first.

Common recipes

• Leader election. Each candidate creates an ephemeral sequential node under /election. The lowest number is the leader. The others watch the node just before theirs. When the leader's session ends, its node vanishes and the next one takes over.
• Distributed lock. The same trick: the lowest sequential node holds the lock. Crash-safe because the node is ephemeral. Pair it with a fencing idea, such as using the node's number, if stale holders could do damage.
• Service discovery and membership. Each instance registers an ephemeral node under /services/payments. Clients list the children and watch for changes.
• Configuration. Store small shared settings and push changes through watches.

Where it is used today

Many systems used it inside: Kafka (in older versions), HBase, Hadoop and others. Newer Kafka versions use their own built-in consensus instead, and many teams now pick etcd (Raft, the base of Kubernetes) or Consul for new work. In an interview you rarely need to run ZooKeeper yourself. You need to say "a coordination service such as ZooKeeper or etcd".

A motivating problem

A chat service starts on one server and everything is easy: all users are connected to it. Add four servers. Now a message for user A, connected to server 3, arrives on server 1. Someone must know where A is. Someone must also decide which server runs the nightly cleanup, and every server must learn when a server dies or a setting changes. These small shared facts need strong agreement, and that is the job of a coordination service. Keep the bulk data (the messages) elsewhere.

What it guarantees

• Writes are linearizable: they go through the leader in one global order.
• Each client sees its own operations in order, and sees updates in that global order.
• A read from a follower can lag. If a client needs the very latest value, it first issues a sync.
• A watch fires before the client can read the new state, so a client never sees later data first.
• A write is atomic: it either applies fully or not at all.

Sessions

A client connects and gets a session with a timeout. It sends heartbeats. If the ensemble hears nothing for the whole timeout, the session expires and its ephemeral nodes are deleted. A client that is merely slow can have its session expire, and must then assume it lost whatever it held (a lock, the leader role). A reconnect to another server inside the timeout keeps the session.

Storage and recovery

• The whole tree is kept in memory for fast reads, which is one reason the data must stay small.
• Every write is appended to a transaction log on disk before it is acknowledged, and periodic snapshots record the tree. Restart means load the snapshot and replay the log.
• If a follower fails, the ensemble carries on while a majority is alive. The follower catches up when it returns.
• If the leader fails, the remaining servers run an election, and a new leader is chosen once a majority agree. Writes pause for that short period.
• Some deployments add observers: servers that serve reads and take no part in voting, so read capacity grows without slowing writes.

Limits worth naming

Every write goes through one leader and a majority, so write throughput does not rise with more servers. It is the wrong tool for large values, high write rates, or every request path. Operating it needs care (disk latency matters a lot, because of the transaction log). In a new design, say what the job is (lock, election, discovery) and "a coordination service such as ZooKeeper or etcd".

When to use it, trade-offs, numbers

• What: a replicated, strongly consistent store for small coordination data.
• Use when: leader election, locks that must be correct, membership, and shared config across a fleet.
• Know: ephemeral and sequential nodes, watches, quorum maths, odd sizes, and that writes go through one leader.
• Careful: it does not scale for writes by adding servers (more servers means more agreement work), and it is not for large data or request-time lookups on a hot path. Clients must handle session expiry.

ZooKeeper or something else?

• Redis lock: pick it when the lock is only for efficiency. Faster and simpler, weaker guarantees after failover.
• A database row lock or a unique constraint: if you already have one relational database, a lock there is often enough and has no extra part to run.
• etcd or Consul: the same job with a more modern API, and common on Kubernetes.
• A managed lock or leader service from your cloud: pick it when you do not want to run a cluster.

Numbers: ensembles of 3 or 5 are standard, and the quorum is a majority. I am leaving out throughput and latency figures, since they depend on setup and I cannot state them reliably.

Common mistakes

• Using it as a general database or a cache.
• An even number of servers.
• Ignoring session expiry, so a client acts as leader after losing its lock.
• Many clients polling or watching the same node, creating a thundering herd when it changes.
• Adding it to a design that a database lock would serve.

• Treating a lost session as a minor glitch, and carrying on as leader.
• Storing large or frequently changing data in znodes.
• Expecting a follower read to be the latest without a sync.

Check yourself

• Why do ensembles have an odd number of servers?
• What is an ephemeral node, and why does it help leader election?
• Describe leader election with sequential nodes.
• How many failures does a 5-server ensemble tolerate?
• Why is a Redis lock weaker than a ZooKeeper lock?
• What would you use instead of running ZooKeeper yourself?
• What happens to ephemeral nodes when a session expires?
• Why does adding more servers not raise write throughput?
• What does a sync do, and when do you need it?