Dead letters and redrive

A payment provider goes down for an hour. Two hundred jobs exhaust their retries and fail terminally. When the provider recovers, you want those jobs to run again — without editing rows, without losing the evidence of what happened, and without replaying anything twice. That is redrive: Workhorse keeps the failed job as immutable evidence and creates a new job from it, with an audit trail linking the two.

The operator client stays separate from application enqueueing in every SDK. TypeScript constructs Admin with a pool, Python uses Admin or AsyncAdmin, and Go calls NewAdmin with a caller-owned pgx or database/sql executor.

List terminal failures

admin.listDeadLetters(query) reads a cold failure index that stays separate from the ready and active indexes. A backlog of a million failures cannot slow workers as they claim jobs.

const page = await admin.listDeadLetters({
  queue: "billing",
  errorName: "ProviderTimeout",
  finishedAfter: new Date("2026-08-12T09:00:00Z"),
});

Filters cover queue, type, tags (every supplied tag must match), errorName, finishedAfter, and finishedBefore. Each DeadLetter carries the payload, the terminal error, the attempt counts, and redriveCount — how many times an operator has already redriven it. The cursor pages by immutable finish time and job identity, so pages stay stable while you work.

Redrive one job

admin.redrive(sourceJobId, request) requires three things: who asked, why, and a stable request ID. Workhorse records attribution; your operator layer must authorize the actor before calling.

const result = await admin.redrive(sourceJobId, {
  actor: actor.email, reason: "provider incident resolved", requestId: incidentId,
});

The new job copies what defines the work: queue, type, payload, tags, retry policy, attempt budget, and execution timeout. It deliberately does not copy attempts, errors, checkpoints, waits, cancellation state, or the old deadline — the new job starts clean.

The source job stays exactly as it failed. job_redrive stores the lineage edge, and admin.getRedriveLineage(jobId) walks the connected graph in either direction, so you can trace a running job back to the failure that spawned it — or forward through repeated redrives.

Repeat safely

Operator tools crash, and operators double-click. Redrive absorbs both. Repeating the same source and request ID returns the existing target with status replayed instead of creating a duplicate. If a different actor or reason arrives under the same request ID, Workhorse throws RedriveIdempotencyConflictError rather than overwrite audit evidence.

PostgreSQL stores a digest of the request ID, not the raw value, so incident identifiers never become durable payload — operator views still get safe diagnostics.

Work through a backlog

admin.redriveMany(filter, request, options) processes a bounded page of matching failures in stable order. Start with a dry run:

const filter = { queue: "billing", errorName: "ProviderTimeout" };
const audit = { actor: actor.email, reason: "provider incident resolved", requestId: incidentId };
const preview = await admin.redriveMany(filter, audit, { dryRun: true, limit: 100 });
const page = await admin.redriveMany(filter, audit, { limit: 100 });

A dry run returns eligible sources without creating jobs or lineage. Once you have confirmed the external dependency recovered, drop dryRun and reuse the same incident request ID across cursor pages.

If the loop crashes mid-backlog, run it again: pages that already redrove converge on their existing targets instead of duplicating them.

One honest caveat: a redriven job is another at-least-once execution. Payment, email, and webhook handlers still need provider-side idempotency before an operator replays them — redrive makes the operation auditable, not magically exactly-once.

Next

• Retries — the automatic attempts that precede a terminal failure
• Queries and timelines — inspect the source and target evidence
• Idempotency — enqueue keys versus redrive request IDs

Exact dead-letter filters, redrive copy rules, bounds, conflicts, and lineage: architecture reference.