jaque exposes its own vital signs in the Prometheus exposition format on /metrics, so a jaque process is scraped the way every other target is. This page is the reference for every jaque_* series: type, labels and what a change in it means. The metrics are emitted by internal instrumentation covering the engine's scheduling loop and the notifier's delivery path, plus a collector that reports how far each event log follower is behind. This page is exhaustive; a series not listed here does not exist.

1. Where

/metrics is served on the same -listen address as the dashboard and the command API; there is no separate flag or port. An empty -listen turns all three off. -target sink and -target notifier processes serve /metrics on their own -listen with the series that apply to them.

curl -s 127.0.0.1:8080/metrics | grep jaque_

The same -listen address also answers /healthz with a plain 200 ok: nothing more than "the process is up", no read against the event log or the projection behind it. It takes no authentication, on any target. A liveness or readiness probe wants this, not /metrics or the dashboard: it should time out on those before either has finished serving something interesting to fail on.

2. The dashboard

A Grafana dashboard graphing every series on this page ships with jaque. Download it from https://pkgs.jaque.sh/grafana/jaque.json and import it (Dashboards > New > Import > Upload), then pick the Prometheus datasource in the DS variable. The job variable lists every job that exposes jaque_build_info, so one dashboard covers a whole fleet. It has one row per section of this page. The contrib compose stack provisions it already, no import needed there.

3. Event log followers

Every consumer of the event log, from the engine to the UI projection, the Livestatus server, the notifier, each sink and the -events-out dump, is registered as a named follower. Lag is the number of entries it has not yet applied.

Metric Type Labels Meaning
jaque_eventlog_last_seq Gauge none Sequence of the most recently appended event log entry.
jaque_eventlog_consumer_lag Gauge follower Entries the named follower has not yet applied. Follower names are engine, ui, livestatus, livestatus-log, notifier, eventhandler, eventdump and sink/<name>.
jaque_eventlog_consumer_last_applied_timestamp_seconds Gauge follower Unix time of the event the named follower most recently applied. Emitted only for followers that report it: today, the sink/<name> followers. Seconds behind while the log is active: time() - jaque_eventlog_consumer_last_applied_timestamp_seconds.
jaque_eventlog_follower_applied_seq Gauge follower Seq of the event the named follower most recently applied, reported by the follower itself. The lag row above is the collector's view of the same cursor.
jaque_build_info Gauge none Always 1; proves the metrics pipeline end to end.

Snapshots. The engine writes a new snapshot every 5 minutes or every 50k events, whichever comes first, and every process restores its starting state from the latest one at startup before it starts following the log.

Metric Type Labels Meaning
jaque_snapshot_seq Gauge none Seq of the last snapshot this process wrote. Engine only.
jaque_snapshot_bytes Gauge none Encoded size in bytes of the last snapshot this process wrote. Engine only.
jaque_snapshot_saves_total Counter outcome Snapshot writes, by outcome (saved, error). Engine only.
jaque_snapshot_save_duration_seconds Histogram none Wall time of one snapshot write, success or failure. Engine only.
jaque_replay_start_seq Gauge none Seq this process restored from at startup; 0 when it replayed from the start. Every target that restores from a snapshot.
jaque_snapshot_refused_total Counter reason Stored snapshots refused at startup, by reason (version, decode, beyond_log). Every target that restores from a snapshot.

4. Engine and scheduling

Metric Type Labels Meaning
jaque_scheduler_drift_seconds Histogram none Difference between a check's scheduled and actual run time.
jaque_scheduler_queue_depth Gauge none Checks currently queued by the scheduler.
jaque_engine_not_owned_total Counter none Inputs dropped because this engine does not own the object (ADR-015 D5).
jaque_commit_batch_decisions Histogram none Decisions made durable per event log flush; sitting at 1 means the disk keeps up, near the cap means the engine is waiting on it.
jaque_commit_queue_depth Gauge none Decisions waiting for the next event log flush.

5. Checks and work dispatch

Metric Type Labels Meaning
jaque_check_duration_seconds Histogram check_type Duration of an executed check, by check type.
jaque_check_timeouts_total Counter check_type Checks that timed out, by check type.
jaque_work_dispatched_total Counter zone Checks dispatched to a remote worker, by zone.
jaque_work_timeouts_total Counter zone Dispatched checks that timed out waiting for a result, by zone.
jaque_work_rejected_total Counter zone Checks that failed to dispatch to the work queue, by zone.
jaque_work_late_results_total Counter none Results received for a dispatch no longer pending.
jaque_work_handled_total Counter zone, outcome Requests a -target worker process took off its zone's queue, by outcome (ok, runner_error, gate_timeout, poison, wire_version, reply_error). Worker only.

6. Cluster and membership

Metric Type Labels Meaning
jaque_membership_size Gauge none Members currently in the coordination membership view.
jaque_membership_epoch Gauge none Epoch of the last membership view observed by this engine.
jaque_config_generation_mismatch Gauge none 1 while this engine's config differs from the newest reload recorded on the log and it holds its checks; reload this engine with the same config as the last one reloaded to clear it.
jaque_engine_owned_objects Gauge none Objects this engine currently owns.
jaque_membership_heartbeat_failures_total Counter none Membership heartbeats that failed.
jaque_engine_stale_epoch_total Counter none Tail events dropped for carrying a stale membership epoch.
jaque_engine_epoch_behind_total Counter none Writes held back because the log already showed a newer epoch for the object than this engine's membership view; resolves on the next view refresh.
jaque_engine_commands_forwarded_total Counter outcome Commands forwarded to another engine for an object this engine does not own, by outcome (sent, error).

7. Notifications

Metric Type Labels Meaning
jaque_notify_suppressed_total Counter reason Notifications suppressed, by reason: downtime, acked, unreachable, dependency, outside_period, kind_disabled, too_soon, no_level, program_disabled, object_disabled, no_contacts. The order they are checked in is in Escalations and windows.
jaque_notify_delivery_failures_total Counter reason Notification deliveries that failed. Today reason always carries the literal value adapter: every delivery failure lands in one bucket instead of one per adapter (webhook, email, command, telegram).
jaque_notifier_pending Gauge none Notifications currently pending delivery across a notifier's shards.
jaque_notifier_blocked_total Counter none Times a notifier's log follower blocked because a delivery shard's queue was full (ADR-015 D5: block, never drop).
jaque_notifier_fenced_total Counter none Times a notifier left an owned open notification unenqueued because its membership view was older than the request's epoch. Climbing while jaque_membership_epoch stays flat means this notifier is cut off from coordination and will exit after a TTL.
jaque_eventhandler_runs_total Counter outcome Event handler runs, by the mapped result status in lower case.
jaque_eventhandler_blocked_total Counter none Times an event handler consumer's log follower blocked because an execution shard's queue was full.
jaque_eventhandler_fenced_total Counter none Times an event handler consumer left an owned firing unrun because its membership view was older than the event's epoch.
jaque_eventhandler_disabled_total Counter none Times an event handler consumer left a firing unrun because event handlers are disabled for the program or the object.

8. Sinks

Metric Type Labels Meaning
jaque_sink_dropped_total Counter sink CheckExecuted events skipped by a perfdata sink's ownership gate because this process does not own the object (ADR-015 D5), by sink name. With several sinks one event counts once per sink.
jaque_sink_write_duration_seconds Histogram sink Duration of a write attempt for one flushed batch, observed on every attempt, success or failure.
jaque_sink_write_rows Histogram sink Rows (or events) in a successfully written batch. Average batch size is _sum/_count.
jaque_sink_write_errors_total Counter sink Batches a perfdata sink failed to write, by sink name. A batch the destination rejected outright (a 4xx other than 429, a rotated token among them) counts here too: it is dropped rather than retried, and this counter is the only trace it leaves.
jaque_sink_transitions_total Counter sink, transition Sink lifecycle steps the manager drove, by sink name and step (started, stopped, build_error). A reload that changes a sink's spec shows as one stopped and one started.

9. UI and API

Metric Type Labels Meaning
jaque_livestatus_connections_total Counter none Connections the Livestatus server accepted.
jaque_livestatus_queries_total Counter outcome Livestatus requests served, by outcome (ok, not_found, bad_request, write_error).
jaque_livestatus_commands_total Counter outcome Livestatus COMMAND requests served, by outcome (accepted, rejected, refused, unavailable).
jaque_ui_commands_submitted_total Counter outcome Commands a -target ui process submitted to the engine fleet over jaque.work.cmd.ingress, by outcome (sent, error).
jaque_cmdbus_received_total Counter source, outcome Command messages taken off the queue, by source (bus: forwarded by a peer engine; ingress: submitted by a -target ui process) and outcome (decoded, decode_error).
jaque_audit_records_total Counter source, outcome Commands written to the audit log, by source (api, extcmd, livestatus) and outcome (accepted, rejected).
jaque_audit_write_errors_total Counter none Audit log writes that failed; each one refused the command it was about to record.
jaque_objects Gauge kind, state Number of monitored objects by kind and current status, from the UI projection.

The Prometheus client library's default collectors are exposed as well: the go_* and process_* metric families, covering scheduler activity, GC, memory and file descriptors.

10. What to watch

jaque_eventlog_consumer_lag climbing on any follower is the earliest warning this page has. Every downstream of the log, projection, sink, notifier, Livestatus, is a follower, and a follower that is falling behind is the first visible symptom of a slow sink endpoint, a blocked delivery shard or a starved process, before a dashboard goes stale or a page is late. Alert on it per follower; the follower label says which subsystem to look at.

jaque_notifier_blocked_total and jaque_sink_write_errors_total are the two counters that name a receiver outside jaque as the problem.

jaque_snapshot_seq not advancing while jaque_eventlog_last_seq does means the engine stopped writing snapshots and the next restart replays the whole log; jaque_snapshot_refused_total incrementing means a restart just did.

The dashboard's Overview row carries these at a glance.

11. Security considerations

/metrics shares -listen with the dashboard and the command API and is not protected by -api-token. Series carry subsystem labels and names of sinks, zones and followers, never object output or secrets. Expose the listener to the scraper and to nothing else; see Security.