> Section: [8. Observing it](https://jaque.sh/docs/ui/dashboard.md)
> Next: observability/sinks-and-perfdata
> Index: https://jaque.sh/llms.txt


jaque exposes its own vital signs in the Prometheus exposition format on
`/metrics`, so a jaque process is scraped the way every other target is.
This page is the reference for every `jaque_*` series: type, labels and
what a change in it means. The metrics are emitted by internal
instrumentation covering the engine's scheduling loop and the notifier's
delivery path, plus a collector that reports how far each event log
follower is behind. This page is exhaustive; a series not listed here
does not exist.

## 1. Where

`/metrics` is served on the same `-listen` address as the dashboard and
the command API; there is no separate flag or port. An empty `-listen`
turns all three off. `-target sink` and `-target notifier` processes
serve `/metrics` on their own `-listen` with the series that apply to
them.

```sh
curl -s 127.0.0.1:8080/metrics | grep jaque_
```

The same `-listen` address also answers `/healthz` with a plain `200
ok`: nothing more than "the process is up", no read against the event
log or the projection behind it. It takes no authentication, on any
target. A liveness or readiness probe wants this, not `/metrics` or the
dashboard: it should time out on those before either has finished
serving something interesting to fail on.

## 2. The dashboard

A Grafana dashboard graphing every series on this page ships with jaque.
Download it from https://pkgs.jaque.sh/grafana/jaque.json and import it
(Dashboards > New > Import > Upload), then pick the Prometheus datasource
in the `DS` variable. The `job` variable lists every job that exposes
`jaque_build_info`, so one dashboard covers a whole fleet. It has one row
per section of this page. The contrib compose stack provisions it
already, no import needed there.

## 3. Event log followers

Every consumer of the event log, from the engine to the UI projection,
the Livestatus server, the notifier, each sink and the `-events-out`
dump, is registered as a named follower. Lag is the number of entries it
has not yet applied.

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_eventlog_last_seq` | Gauge | none | Sequence of the most recently appended event log entry. |
| `jaque_eventlog_consumer_lag` | Gauge | `follower` | Entries the named follower has not yet applied. Follower names are `engine`, `ui`, `livestatus`, `livestatus-log`, `notifier`, `eventhandler`, `eventdump` and `sink/<name>`. |
| `jaque_eventlog_consumer_last_applied_timestamp_seconds` | Gauge | `follower` | Unix time of the event the named follower most recently applied. Emitted only for followers that report it: today, the `sink/<name>` followers. Seconds behind while the log is active: `time() - jaque_eventlog_consumer_last_applied_timestamp_seconds`. |
| `jaque_eventlog_follower_applied_seq` | Gauge | `follower` | Seq of the event the named follower most recently applied, reported by the follower itself. The lag row above is the collector's view of the same cursor. |
| `jaque_build_info` | Gauge | none | Always 1; proves the metrics pipeline end to end. |

Snapshots. The engine writes a new snapshot every 5 minutes or every 50k
events, whichever comes first, and every process restores its starting
state from the latest one at startup before it starts following the log.

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_snapshot_seq` | Gauge | none | Seq of the last snapshot this process wrote. Engine only. |
| `jaque_snapshot_bytes` | Gauge | none | Encoded size in bytes of the last snapshot this process wrote. Engine only. |
| `jaque_snapshot_saves_total` | Counter | `outcome` | Snapshot writes, by outcome (`saved`, `error`). Engine only. |
| `jaque_snapshot_save_duration_seconds` | Histogram | none | Wall time of one snapshot write, success or failure. Engine only. |
| `jaque_replay_start_seq` | Gauge | none | Seq this process restored from at startup; 0 when it replayed from the start. Every target that restores from a snapshot. |
| `jaque_snapshot_refused_total` | Counter | `reason` | Stored snapshots refused at startup, by reason (`version`, `decode`, `beyond_log`). Every target that restores from a snapshot. |

## 4. Engine and scheduling

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_scheduler_drift_seconds` | Histogram | none | Difference between a check's scheduled and actual run time. |
| `jaque_scheduler_queue_depth` | Gauge | none | Checks currently queued by the scheduler. |
| `jaque_engine_not_owned_total` | Counter | none | Inputs dropped because this engine does not own the object (ADR-015 D5). |
| `jaque_commit_batch_decisions` | Histogram | none | Decisions made durable per event log flush; sitting at 1 means the disk keeps up, near the cap means the engine is waiting on it. |
| `jaque_commit_queue_depth` | Gauge | none | Decisions waiting for the next event log flush. |

## 5. Checks and work dispatch

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_check_duration_seconds` | Histogram | `check_type` | Duration of an executed check, by check type. |
| `jaque_check_timeouts_total` | Counter | `check_type` | Checks that timed out, by check type. |
| `jaque_work_dispatched_total` | Counter | `zone` | Checks dispatched to a remote worker, by zone. |
| `jaque_work_timeouts_total` | Counter | `zone` | Dispatched checks that timed out waiting for a result, by zone. |
| `jaque_work_rejected_total` | Counter | `zone` | Checks that failed to dispatch to the work queue, by zone. |
| `jaque_work_late_results_total` | Counter | none | Results received for a dispatch no longer pending. |
| `jaque_work_handled_total` | Counter | `zone`, `outcome` | Requests a `-target worker` process took off its zone's queue, by outcome (`ok`, `runner_error`, `gate_timeout`, `poison`, `wire_version`, `reply_error`). Worker only. |

## 6. Cluster and membership

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_membership_size` | Gauge | none | Members currently in the coordination membership view. |
| `jaque_membership_epoch` | Gauge | none | Epoch of the last membership view observed by this engine. |
| `jaque_config_generation_mismatch` | Gauge | none | 1 while this engine's config differs from the newest reload recorded on the log and it holds its checks; reload this engine with the same config as the last one reloaded to clear it. |
| `jaque_engine_owned_objects` | Gauge | none | Objects this engine currently owns. |
| `jaque_membership_heartbeat_failures_total` | Counter | none | Membership heartbeats that failed. |
| `jaque_engine_stale_epoch_total` | Counter | none | Tail events dropped for carrying a stale membership epoch. |
| `jaque_engine_epoch_behind_total` | Counter | none | Writes held back because the log already showed a newer epoch for the object than this engine's membership view; resolves on the next view refresh. |
| `jaque_engine_commands_forwarded_total` | Counter | `outcome` | Commands forwarded to another engine for an object this engine does not own, by outcome (`sent`, `error`). |

## 7. Notifications

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_notify_suppressed_total` | Counter | `reason` | Notifications suppressed, by reason: `downtime`, `acked`, `unreachable`, `dependency`, `outside_period`, `kind_disabled`, `too_soon`, `no_level`, `program_disabled`, `object_disabled`, `no_contacts`. The order they are checked in is in [Escalations and windows](https://jaque.sh/docs/notifications/escalations-and-windows.md). |
| `jaque_notify_delivery_failures_total` | Counter | `reason` | Notification deliveries that failed. Today `reason` always carries the literal value `adapter`: every delivery failure lands in one bucket instead of one per adapter (webhook, email, command, telegram). |
| `jaque_notifier_pending` | Gauge | none | Notifications currently pending delivery across a notifier's shards. |
| `jaque_notifier_blocked_total` | Counter | none | Times a notifier's log follower blocked because a delivery shard's queue was full (ADR-015 D5: block, never drop). |
| `jaque_notifier_fenced_total` | Counter | none | Times a notifier left an owned open notification unenqueued because its membership view was older than the request's epoch. Climbing while `jaque_membership_epoch` stays flat means this notifier is cut off from coordination and will exit after a TTL. |
| `jaque_eventhandler_runs_total` | Counter | `outcome` | Event handler runs, by the mapped result status in lower case. |
| `jaque_eventhandler_blocked_total` | Counter | none | Times an event handler consumer's log follower blocked because an execution shard's queue was full. |
| `jaque_eventhandler_fenced_total` | Counter | none | Times an event handler consumer left an owned firing unrun because its membership view was older than the event's epoch. |
| `jaque_eventhandler_disabled_total` | Counter | none | Times an event handler consumer left a firing unrun because event handlers are disabled for the program or the object. |

## 8. Sinks

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_sink_dropped_total` | Counter | `sink` | `CheckExecuted` events skipped by a perfdata sink's ownership gate because this process does not own the object (ADR-015 D5), by sink name. With several sinks one event counts once per sink. |
| `jaque_sink_write_duration_seconds` | Histogram | `sink` | Duration of a write attempt for one flushed batch, observed on every attempt, success or failure. |
| `jaque_sink_write_rows` | Histogram | `sink` | Rows (or events) in a successfully written batch. Average batch size is `_sum/_count`. |
| `jaque_sink_write_errors_total` | Counter | `sink` | Batches a perfdata sink failed to write, by sink name. A batch the destination rejected outright (a 4xx other than 429, a rotated token among them) counts here too: it is dropped rather than retried, and this counter is the only trace it leaves. |
| `jaque_sink_transitions_total` | Counter | `sink`, `transition` | Sink lifecycle steps the manager drove, by sink name and step (`started`, `stopped`, `build_error`). A reload that changes a sink's spec shows as one `stopped` and one `started`. |

## 9. UI and API

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `jaque_livestatus_connections_total` | Counter | none | Connections the Livestatus server accepted. |
| `jaque_livestatus_queries_total` | Counter | `outcome` | Livestatus requests served, by outcome (`ok`, `not_found`, `bad_request`, `write_error`). |
| `jaque_livestatus_commands_total` | Counter | `outcome` | Livestatus `COMMAND` requests served, by outcome (`accepted`, `rejected`, `refused`, `unavailable`). |
| `jaque_ui_commands_submitted_total` | Counter | `outcome` | Commands a `-target ui` process submitted to the engine fleet over `jaque.work.cmd.ingress`, by outcome (`sent`, `error`). |
| `jaque_cmdbus_received_total` | Counter | `source`, `outcome` | Command messages taken off the queue, by source (`bus`: forwarded by a peer engine; `ingress`: submitted by a `-target ui` process) and outcome (`decoded`, `decode_error`). |
| `jaque_audit_records_total` | Counter | `source`, `outcome` | Commands written to the audit log, by source (`api`, `extcmd`, `livestatus`) and outcome (`accepted`, `rejected`). |
| `jaque_audit_write_errors_total` | Counter | none | Audit log writes that failed; each one refused the command it was about to record. |
| `jaque_objects` | Gauge | `kind`, `state` | Number of monitored objects by kind and current status, from the UI projection. |

The Prometheus client library's default collectors are exposed as well:
the `go_*` and `process_*` metric families, covering scheduler activity,
GC, memory and file descriptors.

## 10. What to watch

`jaque_eventlog_consumer_lag` climbing on any follower is the earliest
warning this page has. Every downstream of the log, projection, sink,
notifier, Livestatus, is a follower, and a follower that is falling
behind is the first visible symptom of a slow sink endpoint, a blocked
delivery shard or a starved process, before a dashboard goes stale or a
page is late. Alert on it per follower; the `follower` label says which
subsystem to look at.

`jaque_notifier_blocked_total` and `jaque_sink_write_errors_total` are
the two counters that name a receiver outside jaque as the problem.

`jaque_snapshot_seq` not advancing while `jaque_eventlog_last_seq` does
means the engine stopped writing snapshots and the next restart replays
the whole log; `jaque_snapshot_refused_total` incrementing means a
restart just did.

The dashboard's Overview row carries these at a glance.

## 11. Security considerations

`/metrics` shares `-listen` with the dashboard and the command API and is
not protected by `-api-token`. Series carry subsystem labels and names of
sinks, zones and followers, never object output or secrets. Expose the
listener to the scraper and to nothing else; see
[Security](https://jaque.sh/docs/deployment/security.md).