Temporal and durable workflows

Some processes take minutes, days or months and touch many services: an order, a refund, a user onboarding, a subscription renewal. They must finish despite crashes, retries, and waiting for a human. A workflow engine does that bookkeeping for you.

Crux: a durable workflow engine saves each step's result, so after a crash your code replays and resumes exactly where it stopped. Workflow code must be deterministic. Side effects belong in activities, which must be idempotent.

What it is and why it exists

Temporal (a descendant of Uber's Cadence) is an open-source platform for durable execution. You write the process as ordinary code, and the engine makes it reliable. Without it, you build this yourself from a database table of states, a queue, timers, retries and a recovery job. That works for two steps and becomes a mess for ten.

Alternatives in the same family are AWS Step Functions (a state machine defined in configuration) and similar managed engines.

How it works, at the level you must explain

Workflows and activities

• A workflow is the orchestrating code: the order of steps, branches, loops, waits. It must be deterministic: the same history gives the same decisions. So no random numbers, no current time, no network calls inside it. Use the workflow API for time and randomness.
• An activity is one unit of real work: call a payment service, send an email, write a row. Activities may fail and are retried by the engine under a retry policy (backoff, maximum attempts, timeouts). Activities can do any side effect.

Event history and replay

• The Temporal service (the server) stores an append-only event history per workflow: workflow started, activity scheduled, activity completed with this result, timer fired, signal received.
• Workers are your processes. They poll a task queue, run workflow or activity code, and report results back.
• If a worker dies mid-workflow, another picks up the task, replays the code against the stored history, and skips activities that already completed, using their recorded results. The code arrives at the same state and carries on. This is why determinism matters. A change that makes replay diverge breaks running workflows, so changing live workflow code needs versioning.

Features you will be asked about

• Durable timers. Sleep for 30 days costs nothing while sleeping, and survives restarts.
• Signals. Send an event into a running workflow, such as "payment approved" or "user clicked confirm".
• Queries. Read a running workflow's current state.
• Child workflows and retries at workflow level.
• Compensation (the saga pattern). If step 4 fails permanently, run the undo for steps 3, 2 and 1. A workflow expresses this as ordinary code with a list of undo actions.
• Continue-as-new. History grows with every event, so a long-lived loop should start a fresh run with carried-over state now and then.
• Visibility. You can list and search workflows and see where each is stuck.

Exactly once, really?

The engine runs workflow code effectively once. Activities run at least once. A worker can finish the work and crash before reporting, so the activity runs again. So make activities idempotent, for example by passing an idempotency key to the payment call. This is the same rule as in the multi-step processes pattern.

Feeling the need: building it by hand

Take an order flow: charge the card, reserve stock, book a courier, then wait up to a week for delivery confirmation. Do it without an engine and you need, in turn: a table of order states, a queue per step, a retry rule per step, a timer that wakes the order after a week, a recovery job that finds orders stuck halfway, and code that remembers which steps already ran. Every crash between "stock reserved" and "state saved" is a new bug. A durable execution engine takes over exactly that bookkeeping. Say that in an interview: the reason is the cost of the state machine, not fashion.

Under the hood (optional depth)

You rarely need this, but a few terms help.

• The server is several services. The frontend takes requests from clients. The history service owns each workflow's event history and current state, split into history shards, and writes changes and the tasks they imply in one transaction (a transactional outbox). The matching service hands tasks to workers through task queues. Internal workers run housekeeping.
• Storage is a database (Cassandra, MySQL or PostgreSQL are supported options).
• Workers keep running workflows in a cache. With sticky execution the same worker keeps receiving the next task for a workflow, so it skips a full replay. Replay from history is the fallback.
• Scaling is by tier: more workers for your code, more shards for history, more matching capacity for task queues.
• A task queue is also how you separate workloads: give a heavy activity its own queue and its own workers.

Limits, and what stays your job

• Activities may run more than once, so idempotency stays your responsibility.
• The engine does not decide business rules. If payment fails, you still decide whether to retry, cancel or refund.
• It is not a stream processor, and not a fit for tens of thousands of tiny events per second per workflow.
• A single quick job on a queue (send one email) does not need it.
• Some interviewers want you to design the orchestrator itself. If so, state the pieces you would build (state store, timers, retries, recovery) and say Temporal packages them. Ask whether naming it is allowed.

When to use it, trade-offs, numbers

• What: a durable orchestration engine where workflows are code and history is persisted.
• Use when: multi-step flows across services, long waits (days), human approval, retries with backoff, sagas, and scheduled or recurring processes.
• Know: workflow versus activity, determinism, replay, signals and timers, idempotent activities, compensation.
• Careful: it is a system to run (or buy), it adds latency per step, and it is not for high-throughput data pipelines. Do not put large payloads in history. Pass references.

Temporal or something else?

• A request-time call: if everything finishes in a couple of seconds and a failure can simply return an error, do it inline.
• A queue and workers (Kafka, SQS): pick it for independent tasks with no ordering between steps.
• Step Functions: managed on AWS and visual, with configuration-based definitions. Less flexible for complex logic.
• Event-driven choreography: each service reacts to events and emits the next. No central brain, but the whole flow is hard to see and debug.
• Cron plus a database: fine for simple periodic jobs.

Numbers: I am leaving out the per-workflow history limits and throughput figures, because they depend on version and deployment and I cannot give reliable numbers. The design habit is what matters: keep histories bounded with continue-as-new and keep payloads small.

Common mistakes

• Calling the network or reading the clock in workflow code.
• Non-idempotent activities, so a retry charges twice.
• Reaching for a workflow engine for a two-second request.
• Changing workflow code with no versioning while runs are in flight.
• Putting large data in the workflow history.

• Treating the engine as if it removes the need to design failure handling and idempotency.
• Using it for event streams.
• One huge workflow doing all the work, instead of child workflows or continue-as-new.

Check yourself

• What is the difference between a workflow and an activity?
• Why must workflow code be deterministic?
• Describe what happens when a worker crashes halfway through a workflow.
• Why must activities be idempotent?
• How would you model "wait up to 3 days for approval, then cancel"?
• When is a queue plus workers enough?
• What pieces would you have to build yourself without a workflow engine?
• What does sticky execution save?
• What would you say if the interviewer does not allow Temporal?