Durable execution

After a retry, a crash, or a durable wait, Workhorse calls your handler again — from the top. There is no saved JavaScript stack to resume; the process that held it may be gone, deployed over twice since. Durable execution is how a restarted handler skips the work it already finished: you name the boundaries, and the new activation replays their stored results.

Save completed stages with checkpoints

HandlerContext.checkpoint(name, operation) looks for a stored value under the name. The first activation finds none, runs the operation, and persists its JSON result under the current fenced lease. Every later activation finds the value and returns it without running the operation again.

const charge = await ctx.checkpoint("charge", () =>
  payments.charge(payload.orderId, { idempotencyKey: `charge:${payload.orderId}` }),
);
const shipment = await ctx.checkpoint("shipment", () =>
  logistics.createShipment(payload.orderId, charge.id),
);

If this handler fails while creating the shipment, the retry replays the stored charge result and only reruns shipment. The card is charged once across any number of attempts.

One honest caveat: a checkpoint commits after its operation finishes. If the process dies between the external effect and that commit, the next activation runs the operation again. Delivery stays at-least-once, so effects that must not repeat still need a provider idempotency key — that is why the example passes one to payments.charge. The checkpoint makes repeats rare; the provider key makes them harmless.

Reusing a checkpoint name with a different value raises CheckpointConflictError instead of silently changing durable evidence.

The synchronous Python worker accepts a regular callable, while Go accepts a function returning a JSON value and an error. Replay returns the stored value without invoking either callback again.

Release the worker slot with a durable wait

Some jobs need to pause — an hour before a reminder, until tomorrow morning. Sleeping inside the handler wastes a worker slot for the whole wait. A durable wait releases the job instead:

• HandlerContext.sleep(name, durationMs) stores a named relative timer.
• HandlerContext.sleepUntil(name, wakeAt) stores a named absolute target.

Python names the absolute form context.sleep_until(name, wake_at); its relative form remains context.sleep(name, duration_ms).

The wait drops the lease, frees the worker slot, and places the job in scheduled. It does not consume an attempt — waiting is normal work, not failure. A job that sleeps five times and succeeds has used one attempt.

await ctx.checkpoint("welcome", () => mailer.welcome(payload.to));
await ctx.sleepUntil("follow-up-window", new Date(payload.followUpAt));
await ctx.checkpoint("follow-up", () => mailer.followUp(payload.to));

When the timer is due, promotion makes the job ready and some worker — not necessarily the original — calls the handler from the top. The stored welcome checkpoint replays, execution reaches the elapsed follow-up-window wait and continues straight past it, and only then does the follow-up send. Everything before a wait runs at least twice, so checkpoint it or make it idempotent.

A wake time means "eligible from then", not "runs exactly then". Promotion runs on an interval and a worker must be free; expect seconds of slack, and do not build precise timing on top of it.

Treat names as immutable program state

Checkpoint and wait names are the job's durable control flow. The name is how a later activation recognizes a boundary it already passed, so keep names stable across deploys — renaming one creates a different boundary, and in-flight jobs will rerun the stage.

The database defends these rules explicitly:

• A repeated relative wait keeps its original duration and wake target. Changing an absolute wait's target — or switching a name between modes — raises WaitConflictError.
• Checkpoint values are size-capped, and one job's open waits are bounded; exceeding the wait bound raises WaitLimitExceededError.

For inspection, HandlerContext.getCheckpoint and getWait read one boundary from inside a handler, and Admin.listCheckpoints and Admin.listWaits expose the retained evidence to operator code.

Wait for an external result

Timers resume because time passed. Signals and human waits resume because another actor supplied a JSON value:

const event = await ctx.waitForSignal<{ id: string }>("provider-event");
const review = await ctx.waitForHuman("operator-review", { eventId: event.id });

Both calls store a named boundary, release the lease, and restart the handler after delivery. Applications deliver signals through Queue.sendSignal; authenticated operator surfaces complete human waits through Queue.completeHumanWait. Each delivery is idempotent and retained, so a network retry cannot resume the job twice. Read Signals and Human waits for timeout and authorization rules.

TypeScript handlers use ctx.sleep() for a relative timer and ctx.sleepUntil() for a fixed wake time. Both forms store the boundary before the worker releases its lease, then settle the wait when its timer or delivery wins.

Next

• Progress — report mutable status without creating a restart boundary
• Signals — resume from another process
• Human waits — pause for an operator decision
• Retries — understand why the handler starts again
• Cancellation — stop a job during execution or a durable wait

Exact checkpoint and wait limits, conflicts, fencing, and timer transitions: architecture reference.