Dealing with contention

Two people click "buy" on the last ticket in the same millisecond. Both read "1 left", both write "sold". One of them has paid for a seat that is not theirs.

Crux: contention is a race between "check" and "act". Make them one step, or make one request wait. Solve it inside one database first, and go distributed only when the data cannot live in one place.

The problem

Contention appears when several requests read a value, decide, and write based on what they read. Typical places:

• the last item in stock, the last seat, a limited coupon
• an account balance that must not go below zero
• the highest bid in an auction
• a username that must be unique

The bug is called a race condition, and a version of it is the lost update: both requests read 10, both write 9, and one decrement vanishes.

Ask three things: what is the shared resource, how often is it fought over (the hot case), and what is worse, a wrong answer or a slow one?

Options, from simple to complex

Option 0: let the database do it in one statement

Do the check and the write in a single atomic step:

UPDATE seats
SET    status = 'sold', buyer = :user
WHERE  id = :seat AND status = 'free'
-- rows affected: 1 means you won, 0 means someone else did

• Use when: the rule is simple and the data is in one database. This solves a large share of problems.
• Know: the database serialises writes to a row, so the second request sees sold and affects zero rows.

Pessimistic locking

Lock the row first, then read and decide, then write and release. In SQL this is SELECT ... FOR UPDATE inside a transaction.

• Use when: conflicts are frequent and the work between read and write is more than one statement.
• Know: others wait their turn. It is safe and simple to reason about.
• Careful: keep the transaction short, because waiters pile up behind it. Locks taken in different orders cause deadlocks, so lock rows in a consistent order.

Optimistic concurrency

Do not lock. Read the row with a version number, do the work, and write only if the version is unchanged:

UPDATE accounts
SET    balance = :new, version = version + 1
WHERE  id = :id AND version = :seen_version
-- 0 rows affected: someone changed it; re-read and retry

• Use when: conflicts are rare, so most writes succeed first time and nobody waits.
• Careful: when conflicts are common, retries pile up and throughput collapses. Cap retries and add jitter.

Higher isolation levels

A database can run transactions as if they ran one at a time (serialisable isolation). It is the least work for you and the most expensive for the database, and some transactions will be aborted and must be retried. Know what your database's defaults actually guarantee before relying on it. Behaviour differs between products.

Reservations with an expiry

For "hold this seat for ten minutes while the buyer pays": write a held state with a held_until time, and treat expired holds as free. This turns one hot race into two short ones, and avoids holding a database lock for a human's pace.

When the data is spread over several nodes

If the resource lives in one database, none of the below is needed. If it spans services or shards, you have three families of options:

• Distributed lock: a lock kept in a shared store (Redis, ZooKeeper, etcd, or a database row). Give every lock an expiry, otherwise a crashed holder blocks everyone. An expiry creates a new danger: the holder may be paused (a long garbage collection), lose the lock, and still write. A fencing token, an ever-increasing number the storage checks, rejects the stale writer.
• Two-phase commit (2PC): a coordinator asks every participant to prepare, then tells all to commit. It gives atomicity across nodes, but it blocks if the coordinator dies at the wrong moment, and it is slow. Prefer to avoid it. A saga (a chain of local steps with undo steps) is the common alternative; see multi-step processes.
• Serialise through a queue: send every request for one key (one seat, one auction) to one queue partition, and one consumer handles them in order. No locks needed, and the order is explicit. The cost is latency, and that one consumer is the throughput cap for that key.

A worked example: the last concert ticket

Requirements we assume: 50,000 people want 1,000 seats; a person who picks a seat has ten minutes to pay; no seat may be sold twice.

• 1. Hold the seat. The "pick" call runs the atomic update: set held, held_by = user, held_until = now + 10 min, but only where the seat is free or its hold has expired. Zero rows affected means "taken", and the user sees that at once.
• 2. Payment succeeds. A second conditional update moves held to sold, only where held_by is this user and the hold has not expired.
• 3. Payment fails or time runs out. A background sweep (or just the "expired counts as free" rule) releases the seat.

All the locking is one conditional row update in one database. No distributed lock is needed, and it is easy to explain. The hot spot is the few popular seats, and the waiting room for 50,000 users is a separate problem: let people in gradually through a queue, so the database sees a trickle instead of a flood.

Choosing

• One database, simple rule: conditional update. Start here.
• One database, multi-step logic, frequent conflicts: pessimistic lock.
• One database, rare conflicts, you prefer no waiting: optimistic version check.
• Need a time-limited claim: reservation with expiry.
• Resource shared across services: consider whether it can live in one place first. Then a distributed lock with fencing, a saga, or one queue partition per key.
• Truly atomic across databases: 2PC, reluctantly.

State the cost every time: pessimistic costs waiting, optimistic costs retries, distributed options cost latency and failure modes.

When to bring this up, and when not to

• Signals in the prompt: "last", "limited", "no double booking", "balance must not go negative", "highest bid", "unique".
• Do not overbuild. If a rare conflict only costs a retry or an apology, a plain check with a unique constraint as a safety net is enough. A distributed lock for a counter nobody audits is wasted effort.

Deep dives interviewers use

Deadlocks with pessimistic locks

Two requests each hold a row the other wants, and both wait forever. Prevent it by taking locks in one global order (say, ascending ID, so a transfer locks the lower account first). Add a lock timeout, so a stuck request fails and retries, and keep transactions short and free of network calls. Most databases also detect deadlocks and abort one victim, so callers must be ready to retry.

The ABA problem with version checks

A value goes from A to B and back to A. A check "is it still A?" passes, though things changed in between. A monotonic version number (or a last-updated timestamp that only moves forward) avoids it, because the version never repeats even when the value does. This is why optimistic checks compare a version and not the data.

Everyone wants the same row

A flash sale on one product turns one row into the bottleneck, whatever locking you choose. Options, from least to most change:

• Shorten the critical section: do only the conditional update inside the transaction, and do the rest (emails, receipts) afterwards.
• Split the stock into several rows (say, ten buckets of 100) and let each request pick one at random. The total is the sum; the cost is that the last few items are hard to hand out fairly, because one bucket can run empty while others have stock.
• Queue in front: admit requests one key at a time, or admit them at a rate the row can bear, and tell the rest "you are in line".
• Approximate where allowed: a "likes" counter does not need to be exact on every request, so batch it (see scaling writes).

What the isolation level does and does not give

Under read committed, two transactions can both read the stock as 1 and both proceed, so the bug above survives. Repeatable read and snapshot modes differ by product on whether they catch it, and serializable does catch it, at the price of aborts. Do not claim a level "prevents races" without naming which anomaly you mean and checking the database in use. Often the conditional update is clearer than depending on a level.

Crashes while holding something

A client takes a lock, then dies. An expiry (a lease) frees the lock, a hold with held_until frees the seat, and a sweeper cleans leftover state. Whatever you choose, the question to answer aloud is "what releases this if its owner disappears?"

Common mistakes

• Check then act in application code: read the count, test it, then write. That is the race itself.
• Locks without expiry, or locks held across a slow call (an external payment). A hung call then blocks everyone.
• A distributed lock without a fencing token, used where a wrong write is costly.
• Splitting the data across nodes early, creating a distributed problem the single database would have solved.
• Deadlocks from inconsistent lock order.
• Retrying optimistic conflicts forever, or with no jitter.
• Forgetting the hot key: one very popular item serialises everything behind one row. Options are splitting the stock into several rows, or queueing in front of it.
• Believing that "the cache is faster" solves it. A cache read followed by a database write has the same race.

Check yourself

• Why is "read, check, write" unsafe?
• Write the one-statement fix for the last ticket.
• When does optimistic concurrency perform worse than locking?
• What does a fencing token protect against?
• Why avoid two-phase commit if you can?
• How does a reservation with expiry reduce contention?