Operations

Running Workhorse in production splits into a few distinct concerns: supervising processes, maintaining the database, controlling the fleet, and watching health. Each has its own page. This page is the map, plus the fleet controls that belong to no single sibling.

Process supervision

Worker processes owns startup, termination signals, drain behavior, readiness and liveness probes, and database pool ownership. Deploy workers there first; every other concern assumes a supervised process already exists.

Database maintenance

Maintenance and retention owns promotion, expired-lease recovery, history partitions, cleanup, retention policy, and the health budgets that reveal lag. Workers drive this work automatically; the page explains how to observe and tune it.

Rolling statistics

The dashboard reads time-window statistics from minute, hour, or day summaries chosen for the requested horizon. Queue-wait percentiles merge a logarithmic sketch across those summaries, so long-window reads stay bounded without keeping every sample. Worker and tag breakdowns remain live queries because their data-controlled cardinality would make aggregate storage unbounded.

A background rollup summarises fully elapsed periods and records how far it got as a watermark. Window reads combine pre-computed rows below the watermark with live computation above it, so a window is correct the instant a job runs; a lagging rollup means a longer live section and a slower query, never a wrong answer. Retention never deletes history the rollup has not summarised yet, so a stuck rollup surfaces as growing lag on the health page, not as a hole in your numbers.

OpenTelemetry

TypeScript, Python, and Go workers emit the same worker span and metric names, so one set of dashboards can cover a mixed-language fleet. A Go worker restores the W3C context stored by a TypeScript enqueue, which keeps the handler in the enqueue trace across processes.

The Python package exposes telemetry through its optional telemetry extra. Applications supply OpenTelemetry providers and exporters, while Go applications also pass an application-owned log/slog logger through WorkerOptions.Logger. Without a configured provider, telemetry stays inert and worker behavior is unchanged.

Worker metrics use queue, job type, and a closed outcome vocabulary. Job IDs remain on sampled spans and structured events, while payloads, results, and error messages never enter telemetry.

Worker metrics cover only what each process does. Database-wide state — queue depth, the age of ready work, expired leases, paused queues, and fleet capacity — needs a dedicated collector. Run WorkhorseMetricsObserver against a pool, or registerQueueMetrics through a Queue when you also want policy and orchestration gauges, in one long-lived service per database and telemetry resource. Do not start a collector in every worker replica: each one reads the same rows, so replicas would export duplicate gauges and add unnecessary queries.

Fleet controls

Every worker registers itself in PostgreSQL on its registryIntervalMs cadence — 5 seconds by default, 0 to opt out. That registry is how you see and steer a fleet you do not host:

const admin = new Admin(pool);
const workers = await admin.listWorkers();
await admin.setWorkerPaused("billing-worker-1", true, {
  actor: actor.email, reason: "investigating slow provider", requestId: incidentId,
});

Pause is cooperative: the worker notices on its next registry refresh, stops asking for jobs, and any in-flight handler runs to completion. The pause is scoped to the running process — a restarted worker comes back running, so a forgotten pause can never idle a future deployment. Use queue pause when work must stop durably.

Use AsyncAdmin.from_psycopg or AsyncAdmin.from_asyncpg when the operator process is async.

Job controls

Cancellation owns durable requests to stop one job. Dead letters and redrive owns replaying jobs that failed terminally.

Inspection and authorization

Queries and timelines owns read-only inspection: point lookups, bounded listings, lifecycle evidence, and queue.health. Dashboard renders the same read models for humans and owns the authorization boundary for operator mutations. Embedded hosts return a verified principal through authorize, while the standalone dashboard can enable its built-in administrator login.

Terminal operations

The packaged CLI carries the same operator surface into a shell. workhorse admin inspects jobs, queues, schedules, failures, workers, and maintenance state — as aligned tables for humans, or with --json as the exact objects the TypeScript operator API returns:

workhorse admin queues
workhorse admin failures --queue billing --json | jq '.items[].jobId'

Guarded commands — admin cancel, admin redrive, admin pause, admin resume — refuse to run until the target database is named explicitly with --env, verified against current_database() on the live connection, and the operation is confirmed interactively or with --yes. Attribution and redrive idempotency follow the same contracts as the programmatic APIs.

workhorse tui renders the same views as a live, self-refreshing terminal application. It is read-only unless launched with a verified --env, which enables pausing and resuming the selected queue behind an explicit confirmation.

Telemetry attributes producer calls such as Queue.enqueue to deployment.environment.name and service.name; Go accepts a *slog.Logger for the same runtime context. Stable metrics include workhorse.jobs.enqueued, workhorse.jobs.claimed, workhorse.queue.paused, workhorse.jobs.enqueue.outcomes, workhorse.handler.executions, workhorse.handler.duration, workhorse.jobs.completed, workhorse.jobs.failed, and workhorse.jobs.retried. The non_replaceable outcome distinguishes a rejected keyed replacement from a transport failure. Enqueue outcome labels remain accepted, replayed, replaced, and coalesced across runtimes.

Next

• Worker processes — deploy and drain runtime processes
• Maintenance and retention — keep PostgreSQL cleanup healthy
• Compatibility — reject a mismatched runtime before startup

Exact process and maintenance boundaries: architecture reference.