# Operations

> A map of Workhorse's operational surface — which page owns each concern, and the fleet controls that live nowhere else.

Running Workhorse in production splits into a few distinct concerns: supervising processes,
maintaining the database, controlling the fleet, and watching health. Each has its own page. This
page is the map, plus the fleet controls that belong to no single sibling.

## Process supervision

[Worker processes](/docs/worker-processes) owns startup, termination signals, drain behavior,
readiness and liveness probes, and database pool ownership. Deploy workers there first; every
other concern assumes a supervised process already exists.

## Database maintenance

[Maintenance and retention](/docs/maintenance) owns promotion, expired-lease recovery, history
partitions, cleanup, retention policy, and the health budgets that reveal lag. Workers drive this
work automatically; the page explains how to observe and tune it.

## Rolling statistics

The dashboard reads time-window statistics from minute, hour, or day summaries chosen for the
requested horizon. Queue-wait percentiles merge a logarithmic sketch across those summaries, so
long-window reads stay bounded without keeping every sample. Worker and tag breakdowns remain live
queries because their data-controlled cardinality would make aggregate storage unbounded.

A background rollup summarises fully elapsed periods and records how far it got as a watermark.
Window reads combine pre-computed rows below the watermark with live computation above it, so a
window is correct the instant a job runs; a lagging rollup means a longer live section and a
slower query, never a wrong answer. Retention never deletes history the rollup has not summarised
yet, so a stuck rollup surfaces as growing lag on the health page, not as a hole in your numbers.

## OpenTelemetry

TypeScript, Python, and Go workers emit the same worker span and metric names, so one set of
dashboards can cover a mixed-language fleet. A Go worker restores the W3C context stored by a
TypeScript enqueue, which keeps the handler in the enqueue trace across processes.

The Python package exposes telemetry through its optional `telemetry` extra. Applications supply
OpenTelemetry providers and exporters, while Go applications also pass an application-owned
`log/slog` logger through `WorkerOptions.Logger`. Without a configured provider, telemetry stays
inert and worker behavior is unchanged.

Worker metrics use queue, job type, and a closed outcome vocabulary. Job IDs remain on sampled
spans and structured events, while payloads, results, and error messages never enter telemetry.

Worker metrics cover only what each process does. Database-wide state — queue depth, the age of
ready work, expired leases, paused queues, and fleet capacity — needs a dedicated collector.
Run `WorkhorseMetricsObserver` against a pool, or `registerQueueMetrics` through a `Queue` when
you also want policy and orchestration gauges, in one long-lived service per database and
telemetry resource. Do not start a collector in every worker replica: each one reads the same
rows, so replicas would export duplicate gauges and add unnecessary queries.

## Fleet controls

Every worker registers itself in PostgreSQL on its `registryIntervalMs` cadence — 5 seconds by
default, `0` to opt out. That registry is how you see and steer a fleet you do not host:

<Tabs items={["TypeScript", "Python", "Go"]}>
  <Tab value="TypeScript">
    ```ts
    const admin = new Admin(pool);
    const workers = await admin.listWorkers();
    await admin.setWorkerPaused("billing-worker-1", true, {
      actor: actor.email, reason: "investigating slow provider", requestId: incidentId,
    });
    ```
  </Tab>
  <Tab value="Python">
    ```python
    admin = Admin(connection)
    workers = admin.list_workers()
    admin.set_worker_paused(
        "billing-worker-1",
        True,
        AdminAudit(actor.email, "investigating slow provider", incident_id),
    )
    ```
  </Tab>
  <Tab value="Go">
    ```go
    admin := workhorse.NewAdmin(workhorse.NewPGXExecutor(pool))
    workers, err := admin.ListWorkers(ctx)
    result, err := admin.SetWorkerPaused(ctx, "billing-worker-1", true, workhorse.AdminAudit{
        Actor: actor.Email, Reason: "investigating slow provider", RequestID: incidentID,
    })
    ```
  </Tab>
</Tabs>

Pause is cooperative: the worker notices on its next registry refresh, stops asking for jobs, and
any in-flight handler runs to completion. The pause is scoped to the running process — a
restarted worker comes back running, so a forgotten pause can never idle a future deployment. Use
queue pause when work must stop durably.

Use `AsyncAdmin.from_psycopg` or `AsyncAdmin.from_asyncpg` when the operator process is async.

## Job controls

[Cancellation](/docs/cancellation) owns durable requests to stop one job.
[Dead letters and redrive](/docs/dead-letters) owns replaying jobs that failed terminally.

## Inspection and authorization

[Queries and timelines](/docs/queries) owns read-only inspection: point lookups, bounded
listings, lifecycle evidence, and `queue.health`. [Dashboard](/docs/dashboard) renders the same
read models for humans and owns the authorization boundary for operator mutations. Embedded hosts
return a verified principal through `authorize`, while the standalone dashboard can enable its
built-in administrator login.

## Terminal operations

The packaged CLI carries the same operator surface into a shell. `workhorse admin` inspects
jobs, queues, schedules, failures, workers, and maintenance state — as aligned tables for
humans, or with `--json` as the exact objects the TypeScript operator API returns:

```sh
workhorse admin queues
workhorse admin failures --queue billing --json | jq '.items[].jobId'
```

Guarded commands — `admin cancel`, `admin redrive`, `admin pause`, `admin resume` — refuse to
run until the target database is named explicitly with `--env`, verified against
`current_database()` on the live connection, and the operation is confirmed interactively or
with `--yes`. Attribution and redrive idempotency follow the same contracts as the
programmatic APIs.

`workhorse tui` renders the same views as a live, self-refreshing terminal application. It is
read-only unless launched with a verified `--env`, which enables pausing and resuming the
selected queue behind an explicit confirmation.

Telemetry attributes producer calls such as `Queue.enqueue` to `deployment.environment.name` and
`service.name`; Go accepts a `*slog.Logger` for the same runtime context. Stable metrics include
`workhorse.jobs.enqueued`, `workhorse.jobs.claimed`, `workhorse.queue.paused`,
`workhorse.jobs.enqueue.outcomes`, `workhorse.handler.executions`, `workhorse.handler.duration`,
`workhorse.jobs.completed`, `workhorse.jobs.failed`, and `workhorse.jobs.retried`. The
`non_replaceable` outcome distinguishes a rejected keyed replacement from a transport failure.
Enqueue outcome labels remain `accepted`, `replayed`, `replaced`, and `coalesced` across runtimes.

## Next

- [Worker processes](/docs/worker-processes) — deploy and drain runtime processes
- [Maintenance and retention](/docs/maintenance) — keep PostgreSQL cleanup healthy
- [Compatibility](/docs/compatibility) — reject a mismatched runtime before startup

---

Exact process and maintenance boundaries:
[architecture reference](https://github.com/stablemates/workhorse/blob/main/docs/architecture.md#worker-process-lifecycle).