Managing long-running tasks

Exporting a year of data, encoding a video, sending a million emails. The browser times out long before the work ends, and a server thread tied up for minutes is a thread not serving anyone else.

Crux: accept the request quickly, hand the slow part to workers through a queue, give the caller a job ID, and design for the job running more than once.

The problem

A request has a patience limit. Clients, load balancers and users all give up after seconds. The work may take minutes or hours, and it may fail halfway.

So the design must answer four questions:

• How does the caller learn that the work was accepted?
• Where does the work wait, and who does it?
• How does the caller find out the result?
• What happens when a worker crashes, or the task keeps failing?

Options, from simple to complex

Do it in the request

If the work takes a second or two, do it inline. It is the simplest thing that works and the user gets the result immediately.

• Use when: the work is short and the load is modest.
• Careful: do not add a queue to a task that does not need one. A queue means a status API, retries, monitoring and a harder-to-debug flow.

Background thread or in-process job

Return at once and finish the work on a thread in the same server.

• Use when: the work is small and losing it on a crash is acceptable.
• Careful: if the server restarts, the work is gone and nobody knows. This is rarely acceptable for anything the user paid for.

Job queue and worker pool

The standard shape:

• 1. The API validates the request and writes a job row with status queued.
• 2. It puts a message on a queue and returns a job ID (HTTP 202 Accepted).
• 3. A pool of workers takes messages, marks the job running, does the work, then done or failed.
• 4. The client uses the job ID to ask for status or the result.

• Use when: the work takes longer than a request should, load is bursty, or the work needs its own scaling (more CPU, a GPU).
• Know: SQS-style queues and Kafka both fit. A queue with per-message acknowledgement is easier for "do each job once, by any worker". A log is better when many consumers read the same events or you need replay.

Workflow engine

When a job is really several steps, with waits and branches, move to a durable workflow engine. That comes later in this module: multi-step processes.

The details that make it safe

Job states

queued -> running -> done
                 \-> failed -> (retry) -> queued
                          \-> dead-letter queue after N tries

Store the state in a database, not only in the queue. The queue tells workers what to do next. The database tells users what happened.

At-least-once delivery

Most queues deliver a message at least once. If a worker takes a message and dies before acknowledging, the queue hands it to another worker after a timeout (often called the visibility timeout). So the same job can run twice.

The fix is idempotency: running the job twice gives the same result as running it once. Use a job ID or key to skip work already done, write results with "set" rather than "add", and make external calls safe to repeat.

Retries and the dead-letter queue

• Retry with exponential backoff (wait longer each time) and jitter.
• After a fixed number of tries, move the message to a dead-letter queue (DLQ). A message that always fails is a poison message; without a DLQ it would block or loop forever.
• Alert on DLQ depth. Someone must look at it.

Reporting progress

• Polling GET /v1/jobs/{id} is the default. Return the state and, if useful, a percentage.
• Push (webhook, SSE or WebSocket) when the user is watching. See real-time updates.
• Return the result as a link to blob storage if it is large.

Long jobs and crashes

If a job runs for an hour, a crash near the end wastes the hour. Have the worker checkpoint progress (for example, "rows 0 to 4M done") and send a heartbeat, so a stuck worker is detected and its job resumed from the checkpoint.

A worked example: export a report

A user asks for a CSV of all orders from last year. We assume the query takes about a minute (an assumption, not a measurement).

• 1. POST /v1/exports with the date range. The API checks the caller's identity from the token, creates job j42 (queued), enqueues {job_id: j42}, and answers 202 with { "job_id": "j42" }.
• 2. A worker takes the message, sets running, streams the rows to a file in blob storage in chunks, and updates progress now and then.
• 3. It sets done and stores the file's location.
• 4. The page polls GET /v1/exports/j42. On done it receives a time-limited download link.
• 5. If the worker dies at step 2, the message reappears after the visibility timeout. The next worker sees the same job ID, overwrites the same output path, and finishes. Writing to a path chosen from the job ID is what makes the retry harmless.

Choosing

• Seconds: inline.
• Minutes, bursty, or needs separate scaling: queue and workers.
• Many steps, waits, human approvals: workflow engine.
• Different kinds of work (a 5-second thumbnail and a 2-hour encode): separate queues, so slow jobs do not block fast ones.
• Some work is more urgent: a priority queue, or a separate queue with its own workers.

What you gain and what you give up

• Gain: fast responses, workers that scale on their own, bursts absorbed, and failed work that can be retried without the user waiting.
• Give up: the answer is no longer immediate, so you need a status API and a way to tell the user. You run more parts (queue, workers, a job table, alerts), and a failure now happens out of sight, so someone must look at it.

When to bring this up

• Signals in the prompt: an export, a report, a video or image pipeline, a bulk import or send, anything described as "takes a while" or "heavy".
• Say what makes it slow (CPU, a big query, a slow third party), because that decides how workers scale.

Deep dives interviewers use

A worker keeps failing on the same job

Retry a few times with backoff, then park the job in the DLQ with the error attached. Distinguish transient errors (a timeout; retry) from permanent ones (a malformed file; do not retry, mark failed at once). A retry budget per job stops one bad input from burning workers.

The same job is submitted twice

The user double-clicks, or the client retries after a timeout. Let the caller send an idempotency key with the submit call, and keep a unique constraint on it in the job table. The second call finds the existing job and returns its ID. This is separate from the worker-side idempotency above: one stops duplicate jobs, the other makes a repeated run harmless.

The queue is growing

Producers are faster than workers. In order:

• Watch queue depth and the age of the oldest message. Age is the better alarm, since it is what the user feels.
• Scale workers on those signals, up to what the downstream database or API can bear.
• When it still grows, limit intake: reject or delay new jobs with a clear response, and cap the number of jobs per user.
• Remember that a bigger queue only delays the problem.

Fast and slow jobs share workers

A flood of two-hour jobs fills every worker, and a five-second job waits behind them. Use separate queues per class of work, each with its own worker pool, and set a per-user limit so one tenant cannot take all the capacity. A priority queue is the lighter option when the classes are close in size.

Jobs that depend on each other

"Render, then watermark, then publish" is a chain, and "wait for all ten parts, then merge" is a fan-in. A few lines of code that enqueue the next job on completion are fine for two steps. Beyond that, the dependency graph, partial failure and timeouts are what a workflow engine is for.

Common mistakes

• Using a queue for a three-second task.
• Losing jobs between the database and the queue. If you save the job row, then crash before enqueueing, the job never runs. Fix it with the outbox pattern (write the message into the same database transaction, and have a relay publish it), or a sweeper that re-enqueues old queued jobs.
• Not making workers idempotent, then charging the customer twice on a retry.
• No dead-letter queue, so one bad message blocks a worker forever.
• Ignoring backpressure. If producers outrun workers, the queue grows without bound. Watch queue depth and the age of the oldest message, scale workers on it, and decide when to reject new work.
• No way to cancel, or to tell the user what went wrong.
• Keeping state only in the queue, so "where is my job?" has no answer.

Check yourself

• What does the API return when it accepts a long job?
• Why can a job run twice, and how do you make that safe?
• What is a poison message, and where does it go?
• How can a job be lost between the database write and the enqueue?
• When is a queue the wrong answer?
• How do you stop slow jobs from delaying fast ones?