# Dead letters and redrive
> Inspect terminal failures, then create fresh, fully audited jobs from them once the underlying incident is fixed.
A payment provider goes down for an hour. Two hundred jobs exhaust their retries and fail
terminally. When the provider recovers, you want those jobs to run again — without editing rows,
without losing the evidence of what happened, and without replaying anything twice. That is
redrive: Workhorse keeps the failed job as immutable evidence and creates a new job from it, with
an audit trail linking the two.
The operator client stays separate from application enqueueing in every SDK. TypeScript constructs
`Admin` with a pool, Python uses `Admin` or `AsyncAdmin`, and Go calls `NewAdmin` with a
caller-owned pgx or `database/sql` executor.
## List terminal failures
`admin.listDeadLetters(query)` reads a cold failure index that stays separate from the ready and
active indexes. A backlog of a million failures cannot slow workers as they claim jobs.
```ts
const page = await admin.listDeadLetters({
queue: "billing",
errorName: "ProviderTimeout",
finishedAfter: new Date("2026-08-12T09:00:00Z"),
});
```
```python
page = admin.list_dead_letters(DeadLetterQuery(
queue="billing",
error_name="ProviderTimeout",
finished_after=datetime.fromisoformat("2026-08-12T09:00:00+00:00"),
))
```
```go
finishedAfter := time.Date(2026, time.August, 12, 9, 0, 0, 0, time.UTC)
page, err := admin.ListDeadLetters(ctx, workhorse.DeadLetterQuery{
DeadLetterFilter: workhorse.DeadLetterFilter{
Queue: "billing", ErrorName: "ProviderTimeout", FinishedAfter: &finishedAfter,
},
})
```
Filters cover `queue`, `type`, `tags` (every supplied tag must match), `errorName`,
`finishedAfter`, and `finishedBefore`. Each `DeadLetter` carries the payload, the terminal error,
the attempt counts, and `redriveCount` — how many times an operator has already redriven it. The
cursor pages by immutable finish time and job identity, so pages stay stable while you work.
## Redrive one job
`admin.redrive(sourceJobId, request)` requires three things: who asked, why, and a stable request
ID. Workhorse records attribution; your operator layer must authorize the actor before calling.
```ts
const result = await admin.redrive(sourceJobId, {
actor: actor.email, reason: "provider incident resolved", requestId: incidentId,
});
```
```python
result = admin.redrive(
source_job_id,
AdminAudit(actor.email, "provider incident resolved", incident_id),
)
```
```go
result, err := admin.Redrive(ctx, sourceJobID, workhorse.AdminAudit{
Actor: actor.Email, Reason: "provider incident resolved", RequestID: incidentID,
})
```
The new job copies what defines the work: queue, type, payload, tags, retry policy, attempt
budget, and execution timeout. It deliberately does not copy attempts, errors, checkpoints,
waits, cancellation state, or the old deadline — the new job starts clean.
The source job stays exactly as it failed. `job_redrive` stores the lineage edge, and
`admin.getRedriveLineage(jobId)` walks the connected graph in either direction, so you can trace
a running job back to the failure that spawned it — or forward through repeated redrives.
## Repeat safely
Operator tools crash, and operators double-click. Redrive absorbs both. Repeating the same source
and request ID returns the existing target with status `replayed` instead of creating a
duplicate. If a different actor or reason arrives under the same request ID, Workhorse throws
`RedriveIdempotencyConflictError` rather than overwrite audit evidence.
PostgreSQL stores a digest of the request ID, not the raw value, so incident identifiers never
become durable payload — operator views still get safe diagnostics.
## Work through a backlog
`admin.redriveMany(filter, request, options)` processes a bounded page of matching failures in
stable order. Start with a dry run:
```ts
const filter = { queue: "billing", errorName: "ProviderTimeout" };
const audit = { actor: actor.email, reason: "provider incident resolved", requestId: incidentId };
const preview = await admin.redriveMany(filter, audit, { dryRun: true, limit: 100 });
const page = await admin.redriveMany(filter, audit, { limit: 100 });
```
```python
filter = DeadLetterFilter(queue="billing", error_name="ProviderTimeout")
audit = AdminAudit(actor.email, "provider incident resolved", incident_id)
preview = admin.redrive_many(filter, audit, BulkRedriveOptions(dry_run=True, limit=100))
page = admin.redrive_many(filter, audit, BulkRedriveOptions(limit=100))
```
```go
filter := workhorse.DeadLetterFilter{Queue: "billing", ErrorName: "ProviderTimeout"}
audit := workhorse.AdminAudit{
Actor: actor.Email, Reason: "provider incident resolved", RequestID: incidentID,
}
preview, err := admin.RedriveMany(ctx, filter, audit, workhorse.BulkRedriveOptions{
DryRun: true, Limit: 100,
})
page, err := admin.RedriveMany(ctx, filter, audit, workhorse.BulkRedriveOptions{Limit: 100})
```
A dry run returns eligible sources without creating jobs or lineage. Once you have confirmed the
external dependency recovered, drop `dryRun` and reuse the same incident request ID across cursor
pages.
If the loop crashes mid-backlog, run it again: pages that already redrove converge on their
existing targets instead of duplicating them.
One honest caveat: a redriven job is another at-least-once execution. Payment, email, and webhook
handlers still need provider-side idempotency before an operator replays them — redrive makes the
operation auditable, not magically exactly-once.
## Next
- [Retries](/docs/retries) — the automatic attempts that precede a terminal failure
- [Queries and timelines](/docs/queries) — inspect the source and target evidence
- [Idempotency](/docs/idempotency) — enqueue keys versus redrive request IDs
---
Exact dead-letter filters, redrive copy rules, bounds, conflicts, and lineage:
[architecture reference](https://github.com/stablemates/workhorse/blob/main/docs/architecture.md#job_redrive).