Multi-step processes

Placing an order means reserving stock, charging a card and creating a shipment. Each step may live in a different service. Any of them can fail, time out, or run twice. The system must end up in a sensible state regardless.

Crux: a process that spans services has no single transaction to save it. Write down its state durably, make every step safe to repeat, and define what "undo" means for each step.

The problem

Within one database, a transaction makes several changes succeed or fail together. Across services, you lose that. The failure cases:

• A step fails after earlier steps succeeded (card declined after stock was reserved).
• The process crashes between steps. Which steps happened?
• A call times out. Did it work or not? You cannot tell.
• A retry runs a step a second time.

Ask: how many steps, how long can the process run (seconds, or days with a human approval), and what must be undone if a later step fails?

Options, from simple to complex

One transaction

If all the steps touch data in one database, use one transaction and stop. Many "multi-step" problems are this.

A state machine in a database

One service owns the process. It stores a row per instance with the current state, and moves it forward step by step, saving after each.

orders: id, state, updated_at
state: created -> stock_reserved -> paid -> shipped
                          \-> payment_failed -> stock_released

• Use when: a handful of steps and one team owns the flow.
• Know: after a crash, a recovery loop finds rows stuck in a non-final state and resumes them. That loop is where the reliability lives.
• Careful: you are building a small workflow engine. It grows by accident, so watch the code.

Saga

A saga is a sequence of local transactions, each with a compensating action that undoes it: reserve stock has release stock, charge card has refund. If step 3 fails, run the compensations of steps 2 and 1, newest first.

• Use when: steps live in different services and you cannot hold a distributed lock or two-phase commit.
• Know: a saga is not atomic. Other users can see the in-between state (stock reserved, not yet paid), so design the states to be visible and safe.
• Careful: some steps cannot be undone (an email already sent). Put those last, or make the compensation a follow-up action (a correction email). Compensations can fail too, and need retries.

Two ways to run a saga:

• Orchestration: one coordinator calls each step and decides what next. Easy to follow and debug, one place to look. The coordinator must be durable.
• Choreography: no coordinator. Each service reacts to events from the previous one (StockReserved triggers payment). Loosely coupled, but the flow is spread over many places and is hard to see or change.

Workflow engine

A durable workflow engine (Temporal, AWS Step Functions and similar) stores every step's progress for you. Your code describes the process as a function; the engine retries failed steps, persists state across crashes, supports timers (wait 3 days) and human steps, and shows each run's history.

• Use when: many steps, long waits, branching, or several workflows. The engine replaces the recovery loop you would otherwise write.
• Know: you adopt its model and its operational cost. Workflow code often has rules (it must be deterministic in some engines), and you must plan how to change a workflow while old runs are still going.

Event sourcing

Instead of storing the current state, store the sequence of events (OrderPlaced, PaymentTaken), and derive state by replaying them. It gives a full history and an audit trail, and fits choreography. It also brings its own complexity (event schemas that change, rebuilding views). Mention it, and do not volunteer it as a default.

The rule that ties them together: idempotency

Any step can run twice, because retries after timeouts are the only safe way to deal with "did it work?". So every step must be idempotent: doing it again has no further effect.

• Give each logical operation an idempotency key (for example order-42-charge) and send it with the call. The receiver stores the key with the result, and on a repeat returns the stored result instead of acting again.
• Prefer "set to" over "add to". Prefer "create with this ID" over "create".
• Record a step's outcome before the next step starts, and use the stored outcome after a crash.

A worked example: placing an order

Steps (assumed): reserve stock, charge card, create shipment. We pick orchestration by a workflow engine or a state machine.

• 1. The order row is created in state created, with an ID that doubles as the idempotency key base.
• 2. Reserve stock, key order-42-reserve. State becomes stock_reserved.
• 3. Charge card, key order-42-charge. If it times out, retry with the same key, and the payment provider charges once. If it is declined, run the compensation: release stock, state cancelled.
• 4. Create shipment, key order-42-ship. If this fails for good (it is in our control, so rare), refund and release stock.
• 5. A step that is stuck gets an alert after a timeout, and a person can see exactly which step it is on.

The point to make in a discussion is not the tool, but the three properties: the state is durable, each step is safe to repeat, and each step has a defined undo.

Choosing

• One database: a transaction.
• A few steps, one owner: state machine.
• Across services, bounded steps, undo makes sense: saga with orchestration.
• Long running, timers, human steps, many flows: workflow engine.
• Teams that must stay decoupled and accept a harder-to-follow flow: choreography.

Avoid two-phase commit across services, unless the participants are all databases built for it. It blocks on failure and is slow.

How a workflow engine survives a crash

The engine does not snapshot your program's memory. It keeps an append-only history of what happened: step started, step result, timer set, signal received. After a crash, a worker starts your workflow function again from the top, and the engine feeds recorded results to each step that already finished, so only unfinished steps really run. This is called replay.

Consequences to know:

• The workflow code itself must behave the same on every replay: no reading the clock, no random numbers, no direct network calls. Those go into activities (the steps with side effects) whose results are recorded.
• Activities can still run more than once, because a crash may come after the side effect and before the result was recorded. They must be idempotent. Even an engine gives at-least-once execution of steps, and anyone promising "exactly once" means "at least once plus idempotency".
• Managed services (such as AWS Step Functions) move the definition into configuration, and take the hosting away from you. Open-source engines give you code and portability, and you run the cluster. Pick on who will operate it.

Deep dives interviewers use

The coordinator crashes mid-saga

State is stored after each step, so a new instance picks up the instance from its last recorded state and continues. Without an engine, a recovery loop scans for sagas that have been in a non-final state longer than expected. The step that was in flight at the crash is repeated with its idempotency key, which is why that key must be stable and saved before the call.

Changing a workflow with runs in flight

Runs started under the old code must finish under logic they understand. Options: versioning (branch in the code on a version marker recorded at start, so old runs take the old path), or run old and new definitions side by side until the old ones drain. Migrating a live run to new logic is rarer and needs care, because replay will compare it to its recorded history. Keep versions few and delete a path only when no run uses it.

State and history growing without limit

A workflow that loops forever (a subscription that bills monthly) piles up history. Break it into generations: after some number of events, finish the run and start a new one carrying a small state. Keep large payloads in blob storage and pass references, not bytes.

Waiting for the outside world

Approval by a person, a payment callback, a shipping event: the workflow waits for a signal or a callback with a timeout, and decides what to do on silence (remind, escalate, cancel). The waiting costs nothing while it sleeps, which is the main advantage over holding a thread. External events need an idempotency key as well, since webhooks repeat.

Making sure a step takes effect once

You cannot make a network call exactly-once. You make the effect once: pass an idempotency key, or have a unique constraint on the effect (one row per order and step), and treat a duplicate as "already done, return the old result". For effects the other side cannot de-duplicate, such as sending an email, narrow the window and accept a small chance of a duplicate.

When this is too much

Two steps in one service do not need a saga or engine. A transaction or a small state column is enough. Reaching for a workflow engine in a CRUD app is a sign of pattern-matching, not design.

Common mistakes

• Steps that are not idempotent, so a retry double-charges.
• Treating a timeout as a failure and running the compensation, while the original step actually succeeded. After a timeout, the state is unknown; ask or retry with the same key.
• The dual-write problem: updating a database and publishing an event as two separate actions, so a crash does one but not the other. Use the outbox pattern (see long-running tasks).
• Compensations that are missing, or that cannot be run safely twice.
• Hiding the in-between states from the user and the rest of the system.
• A process with no timeout, so a lost step waits forever.
• Choreography with ten services and no one who understands the whole flow.
• No way to see where each instance is.

Check yourself

• Why can you not rely on a database transaction across services?
• What is a compensating action? Give two examples and one that is impossible.
• When would you choose orchestration over choreography?
• What does an idempotency key do?
• After a timeout on the payment call, what should the process do?
• What does a workflow engine give you that a hand-built state machine does not?