Sinks and perfdata

jaque forwards every check’s perfdata to whichever time series backend you already run, rather than storing it itself.

The sinks: block

Each sink is a named entry under sinks: in the CUE config, typed by type the same way #Checks/#Contacts are:

sinks: {
  perf: {
    type:  "clickhouse"
    url:   "clickhouse://ch:9000/jaque"
    table: "perfdata"
  }
  prom: {
    type: "remote_write"
    url:  "http://victoriametrics:8428/api/v1/write"
  }
  log: {
    type:          "archive"
    url:           "file:///var/lib/jaque/archive"
    segment_bytes: 67108864
  }
  webhook: {
    type:     "http"
    url:      "https://ingest.example.com/perfdata"
    headers: {"Authorization": "Bearer ..."}
    selector: "env=prod"
  }
  pipe: {
    type:    "exec"
    command: ["/usr/local/bin/jaque-sink.sh"]
  }
  disk: {
    type:         "file"
    path:         "/var/log/jaque/perfdata.jsonl"
    rotate_bytes: 67108864
  }
  otel: {
    type: "otlp"
    url:  "http://otel-collector:4318"
  }
}

Every sink shares four common fields (#SinkCommon): input ("metrics" by default, or "events" to see the whole log instead of just perfdata — only archive, http, exec and file accept events, and archive requires it), selector (a labels expression, empty by default — no filter beyond the object’s own ownership), batch_size (1000 by default) and flush_interval (5s by default). url/path/command vary by type; cursor (where the sink remembers the last sequence it sent — without one, LastSeq is always 0 and a persistent log resends its retained window on every restart) exists on remote_write, http, exec, file and otlp. Several sinks of the same type coexist fine — several archive entries pointing at different destinations is valid, unlike when -archive was a single flag.

Delivery: retries and what a 4xx means

Delivery failures are handled by kind: a 429 or 5xx retries the same batch; any other 4xx (malformed labels, say) discards the batch with a logged warning and advances the cursor anyway — retrying forever makes no sense against an endpoint that will never accept that batch.

Unit normalization

Every perfdata point is normalized once, at the forwarder boundary, to a canonical UCUM unit plus a scale factor and a kind (gauge or counter); the raw value and UOM are kept as metadata, never discarded:

Raw UOM	Canonical unit	Factor	Kind
(empty)	1	1	gauge
s	s	1	gauge
ms	s	1e-3	gauge
us	s	1e-6	gauge
ns	s	1e-9	gauge
B, b	By	1	gauge
KB, kb, KiB	By	1024	gauge
MB, mb, MiB	By	2^20	gauge
GB, gb, GiB	By	2^30	gauge
TB, tb, TiB	By	2^40	gauge
PB	By	2^50	gauge
%	%	1	gauge
c	1	1	counter

Lowercase b/kb/mb/gb/tb are aliases of their uppercase counterparts — there’s no lowercase alias for PB, nor for the *iB variants, which already normalize to the same factor as their base-1000 form. An unrecognized UOM passes through untouched, factor 1, kind gauge — the table never rejects a point, it just stops normalizing it.

The series shape

For a remote_write sink, __name__ comes from the kind and normalized unit, not the raw UOM: a counter is always jaque_perfdata_total; a gauge in s is jaque_perfdata_seconds, in By is jaque_perfdata_bytes, in % is jaque_perfdata_percent, in 1 (the generic unit) is jaque_perfdata; any other unit without a dedicated series also falls back to jaque_perfdata, but then the uom label carries the raw UOM so the unit isn’t lost — with a dedicated series that label doesn’t appear, the name already carries it. The other labels — host, service (only for service-level results), label (the perfdata point’s name) and origin — are always present. One Sample per series, no warn/crit/min/max — those live in the check config, not the series.

Choosing sinks per process: -sinks

Which sinks run in a given process is a flag, not part of the config graph: -sinks perf,prom selects by name (empty means all declared sinks). A name that doesn’t exist in the merged sinks: block is a startup error listing the valid ones.

-sink/-archive: deprecated, translated

The old flags keep working, but each one is translated into a synthetic SinkSpec with its own name (sink-0, sink-1, … for each -sink URL; archive for -archive) and merged with the config’s sinks: block — if a name collides with one declared in CUE, startup aborts rather than picking a silent winner. Each translation logs a warning naming the sink. -sink keeps its old shape: clickhouse://host:port/database and remotewrite://host:port/api/v1/write[?cursor=/path] (remotewrites:// for https), comma-separated.

Hot reload, with one exception

SIGHUP on -target all|engine reconciles the set of sinks by name through sink.Manager: a sink new to the config starts, one that disappeared stops, and one whose spec changed (URL, type, selector, …) is rebuilt — all without touching sinks that didn’t change. -target sink doesn’t have that wired up — it doesn’t handle SIGHUP at all (same as -target ui), so a change to which sinks run there means a restart. A sink’s selector is evaluated live against the current config on every event — an object a reload just added, that already matches a still-running sink’s selector, gets picked up without that sink restarting.

Speaking of -target sink: it now requires -config to resolve the sinks: block, though it still runs no engine or projection of its own. A sink with input: "events" is rejected there at startup — -target sink shards the log by object ownership, and an events sink doesn’t look at objects, it looks at the whole log, so it only makes sense under -target all|engine, which runs a single unsharded engine.

Extensibility: nothing in-process

There’s no plugin ABI for sinks — no WASM (WASM support stays scoped to checks), no folder of modules. Want a backend that isn’t listed above? Either route it through OTLP to an OpenTelemetry Collector (its exporters cover almost everything), or write an exec sink that reads JSONL on stdin. Never a jaque fork, never a plugin directory.

Ownership in a clustered deployment

With -target engine|all and replicas > 1 sharing a log, or with several -target sink processes, each CheckExecuted event is written by exactly one owner; skipped writes are counted in jaque_sink_dropped_total. After a membership change, a sink picking the log back up may re-evaluate events it already saw under the new membership view — a transient duplicate that ClickHouse’s ReplacingMergeTree collapses on its own.

Configuring a sink end to end

A sinks: block declaring a remote_write entry (pointed at your Prometheus-compatible backend) or an otlp entry (pointed at an OpenTelemetry Collector) is a complete, working setup on its own — the examples earlier on this page are not a separate mechanism from what you’d run in production, just the same block with real endpoints filled in.