An adapter turns a notification decision into a side effect: an SMTP session, an HTTP POST, a process, a Telegram message. jaque has seven, one per contact type, and none of them is trusted to remember anything. Every decision is written to the event log before an attempt is made and its outcome is written after, so whoever delivers can be killed, restarted or replaced by another process and resume from the log with no duplicate that already landed and no notification still owed. Delivery is at least once, by design, and idempotent by notification ID.
Status: the path is tested end to end against a fake receiver, including
a flaky one, but no deployment has paged a human yet; see the
status page. JAQUE_TELEGRAM_TOKEN is the
environment form of -telegram-token by the general rule that every flag
has a JAQUE_ prefixed variable, documented in CLI flags.
1. The seven adapters
| Adapter | Contact type | Mechanism |
|---|---|---|
email |
net/smtp with STARTTLS and optional PLAIN auth, or the system mail binary |
|
| webhook | webhook |
HTTP POST to the contact's url |
| command | command |
a top-level command template run as a process, the Nagios model |
| telegram | telegram |
the Bot API's sendMessage to the contact's chat_id |
| pagerduty | pagerduty |
the Events API v2 enqueue endpoint, keyed by the contact's routing_key |
| slack | slack |
an HTTP POST to the contact's webhook_url |
| incidentio | incidentio |
the alert events HTTP source named by the contact's alert_source_config_id |
The command adapter is the primitive: it runs a process the way
Nagios's notification commands do, and it is what the email adapter
falls back to when no relay is configured. Anything not listed is
reached through webhook or
command; there is no plugin interface for adapters. The contact side
of this table is specified in
Contacts and policies.
2. Durability through the log
A notification has three events and a stable identity.
Text version
engine decides deliverer attempts log
-------------- ------------------ ---
NotifyDecision -------> NotificationRequested -----> appended BEFORE the attempt
adapter.Send(...)
NotificationSent -----> appended on success
or NotificationFailed -----> appended on permanent failure
NotificationRequested is appended before any adapter is called. The
deliverer, whichever process that is, folds the log on startup to find
every NotificationRequested without a terminal NotificationSent or
NotificationFailed and resumes exactly those. The notification ID is
derived from the object, the kind, the level and the instant the problem
began, never from the current time, so a re-notification of the same
problem and a replayed request after a crash produce the same ID and are
deduplicated against it.
The guarantee is at least once. If ownership of delivery moves between processes while an attempt is in flight, the new owner may repeat a send that the old one had started; it will never skip one. Retries use backoff and stay inside one delivery shard, so two attempts for one ID never run concurrently.
3. Where delivery runs: -notify-delivery
The engine that owns an object always decides whether to notify and writes that decision to the log. Who performs the side effect is a separate switch.
-notify-delivery |
Who delivers |
|---|---|
embedded (default) |
the same process that decided, under -target all or -target engine |
external |
one or more -target notifier processes, each pulling its share of open notifications from the log |
Under external, several notifiers split the work by object ownership
over their own membership bucket; Cluster and
coordination describes the
split. A -target notifier process requires -notify-delivery external
on the engine upstream, or it finds nothing to deliver.
In either mode delivery is sharded by object across -notify-workers
workers (default 4), each with a bounded queue of -notify-queue
notifications (default 64). A slow receiver fills its shard's queue and
then blocks the notifier's log follower; nothing is dropped for being
slow. The block is visible as jaque_notifier_blocked_total counting up
and jaque_eventlog_consumer_lag{follower="notifier"} growing, and
jaque_notifier_pending shows how many notifications sit in the shards.
4. Webhook body
The JSON body posted to a webhook contact's url:
| Field | Notes |
|---|---|
notification_id |
stable identity, also sent as the Idempotency-Key header |
object_id |
the object the notification is about, e.g. web1/http |
kind |
PROBLEM, RECOVERY, ACKNOWLEDGEMENT, ... |
level |
the escalation level this send belongs to |
contact |
the contact ID |
at |
RFC3339Nano, when the decision to notify was made |
status |
the object's status at decision time: OK, WARNING, CRITICAL or UNKNOWN |
output |
the check's plugin output |
perfdata |
the check's performance data; the key is omitted when there is none |
problem_since |
RFC3339Nano, when the open problem began; "" when the object has no open problem |
5. Runtime macros
A command-adapter template (and the mail fallback SMTP uses) can
reference these macros at dispatch time, alongside the identity macros
already available everywhere: $NOTIFICATIONTYPE$, $NOTIFICATIONNUMBER$,
$CONTACTNAME$, $CONTACTEMAIL$. Every other Nagios runtime macro still
fails template expansion.
A service or a process contact gets the service vocabulary; a host contact gets the host vocabulary. A process has no macro family of its own, since a process is not part of Nagios's object model and jaque adds no third one for it.
| Macro | Value |
|---|---|
$SERVICESTATE$ |
OK, WARNING, CRITICAL or UNKNOWN |
$SERVICEOUTPUT$ |
the check's plugin output |
$SERVICEPERFDATA$ |
the check's performance data |
$SERVICEDURATION$ |
time since the open problem began, XdYhZmWs |
$LASTSERVICESTATECHANGE$ |
unix seconds when the open problem began |
| Macro | Value |
|---|---|
$HOSTSTATE$ |
UP when the status is OK, otherwise DOWN |
$HOSTOUTPUT$ |
the check's plugin output |
$HOSTPERFDATA$ |
the check's performance data |
$HOSTDURATION$ |
time since the open problem began, XdYhZmWs |
$LASTHOSTSTATECHANGE$ |
unix seconds when the open problem began |
$HOSTSTATE$ never renders UNREACHABLE: reachability tracking is not
part of a notification, so a host's state macro is always one of UP or
DOWN. Nagios has a third host state, UNREACHABLE; this macro does
not carry it.
Both families also get $TIMET$ (unix seconds), $DATE$ (YYYY-MM-DD),
$TIME$ (HH:MM:SS), $SHORTDATETIME$ and $LONGDATETIME$, all as of
the notification decision.
6. SMTP
-smtp-host, -smtp-port (default 25), -smtp-user, -smtp-pass, -smtp-from
With -smtp-host set, email contacts are delivered over SMTP with
STARTTLS; -smtp-user empty disables authentication. With -smtp-host
empty, email contacts are delivered by running the system mail binary
through the command adapter, which suits a host that already has local
mail delivery configured and nothing else.
7. Telegram
A telegram contact carries only a chat_id. The bot token is one per
jaque instance and is a process secret, set once via -telegram-token or
JAQUE_TELEGRAM_TOKEN, the same posture as SMTP credentials. With the
token empty, every telegram contact fails permanently rather than
silently.
To obtain a token, talk to @BotFather, send
/newbot and keep the token it returns. To find a chat_id, message the
bot (or add it to a group), then read
https://api.telegram.org/bot<token>/getUpdates: the chat.id on the
message you sent is the value for the contact.
8. Message templates
An email is plain text, CRLF-terminated, with a fixed set of headers in a
fixed order: From, To, Subject, Date, Message-ID, MIME-Version,
Content-Type. Only Subject and the body are templated; the rest are
computed from the delivery and the SMTP configuration.
The default subject and body:
[jaque] {{.Kind}} {{.Object}} is {{.Status}}
Object: {{.Object}}
Kind: {{.Kind}}
Status: {{.Status}}
Output: {{.Output}}
Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}})
Level: {{.Level}}
Contact: {{.Contact}}
At: {{.At.Format "2006-01-02 15:04:05 MST"}}
The default Telegram text:
[{{.Kind}}] {{.Object}} is {{.Status}}
{{.Output}}
Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}})
Level {{.Level}}
An email contact overrides either or both with subject_template and
body_template; a telegram contact overrides its text with template.
Each defaults to "", meaning the default above:
alice: {
type: "email"
address: "[email protected]"
subject_template: "[{{.Level}}] {{.Object}} {{.Status}}"
body_template: "{{.Output}}\nSince {{.ProblemSince.Format \"15:04:05\"}}\n"
}
Every template renders against the same fields:
| Field | Meaning |
|---|---|
Object |
the object the notification is about |
Kind |
PROBLEM, RECOVERY, ACKNOWLEDGEMENT, ... |
Status |
the object's status at decision time |
Level |
the escalation level this send belongs to |
Contact |
the contact ID |
Output |
the check's plugin output |
Perfdata |
the check's performance data |
At |
when the decision to notify was made |
ProblemSince |
when the open problem began |
Duration |
time since the open problem began, XdYhZmWs |
At and ProblemSince are timestamps with a .Format "layout" method, so
a template can call {{.At.Format "2006-01-02 15:04:05 MST"}} or any other
layout rather than just printing the zero-argument value.
Every template, default or overridden, is parsed and executed once at startup against a fixed probe value. A broken template or one that names a field not in the table above fails the process at load time rather than at the moment a notification is due.
HTML mail is not supported: the body is always text/plain, and there is
no HTML alternative part.
9. Incident platforms: PagerDuty, Slack, incident.io
PagerDuty and incident.io both group a problem's notifications into one open incident that resolves itself when jaque says the problem is over. What makes that possible is an alert key: one key per problem episode, shared by the PROBLEM notification, every re-notification, every escalation and the RECOVERY. The key is derived from the object ID and the instant the problem began, never from the notification ID, which varies by kind and level. An incident platform keyed on the notification ID would open a new incident on every re-notification instead of updating the one already open.
A PROBLEM or STATUS_CHANGE maps to PagerDuty's trigger event action
and to incident.io's firing status, a RECOVERY to resolve and
resolved, and an ACKNOWLEDGEMENT to PagerDuty's acknowledge
(incident.io has no equivalent and receives nothing). The flapping and
downtime kinds are not incidents: both adapters report them delivered
without posting, so a contact that should not receive them at all lists
them in the level's disabled_kinds.
PagerDuty additionally wants a severity, derived from the object's status at decision time:
| Status | PagerDuty severity |
|---|---|
CRITICAL |
critical |
WARNING |
warning |
UNKNOWN |
error |
OK |
info |
The incident.io API token is a process secret the same way the Telegram
bot token is: -incidentio-token or JAQUE_INCIDENTIO_TOKEN. With the
token empty, every incidentio contact fails permanently rather than
silently.
Slack is the exception among the three: a slack contact's
webhook_url is itself the credential, since posting to it is the whole
authorization Slack asks for. Unlike the PagerDuty routing key and the
incident.io token, it is not held back as a process secret. It lives in
the config, on the contact.
10. Grouping
Dependency suppression already handles the case where a failure is
declared to depend on another: nothing is sent for the child while the
parent is the one that is broken. What it cannot handle is the residue:
an undeclared dependency nobody wrote down, the race window before a
failing parent has itself gone hard DOWN, or a correlated failure with
no dependency edge at all, like a bad deploy that breaks forty unrelated
checks at once. Grouping is the safety net for that residue: instead of
forty messages, a contact gets one digest. It is not a general-purpose
alerting router: there is no group_by, no group_interval, no
re-notify inside an open group, one window per contact and nothing more.
A digestable contact, one of email, webhook, telegram or slack,
may set group_wait, a duration. The first notification buffered for
that contact opens a window of that length; every further notification
for the same contact arriving before the window closes joins it. When
the window closes, everything buffered is delivered as one message,
oldest first. group_wait is scoped per contact rather than globally:
two contacts with different settings, or one grouped and one not, are
unaffected by each other.
command, pagerduty and incidentio contacts do not take
group_wait; the config schema rejects it there. The command adapter's
contract is one macro line per object: a digest has nowhere to put a
second object's macros. PagerDuty and incident.io already group on the
platform's own terms (section 9): an alert key ties every notification
for one problem episode into a single incident that opens once and
resolves once, which is a different and stronger guarantee than a time
window, so a second, weaker grouping mechanism underneath it would only
compete with it. slack, unlike those two, has no alert-key grouping of
its own: it is a plain HTTP POST to a webhook, section 9's exception, so
it benefits from group_wait exactly as email and webhook do.
Grouping does not change what lands on the event log. Every object still
gets exactly one NotificationRequested, and every notification and
contact pair still gets exactly one terminal NotificationSent or
NotificationFailed, precisely as section 2 describes. What differs is
delivery-side only: a buffered notification is an ordinary open log entry,
just viewed from memory until its window closes. A crash mid-window loses
nothing but time: the open entries re-fold on restart, re-buffer, and the
window restarts from zero, at most delaying delivery by another
group_wait. The window itself is not persisted anywhere; buying back
that delay would mean writing the window to the log, and the log's
job is delivery record-keeping rather than scheduling.
A window that closes with exactly one notification in it is delivered through the ordinary single-message path, never as a one-item digest. Grouping only changes anything once there is something to group. A window with a PROBLEM and its RECOVERY both buffered does not cancel them against each other the way some alerting systems do: both are delivered, in the order they happened, inside the same digest.
The default email digest subject and body:
[jaque] {{.Count}} notifications
{{.Count}} notifications for {{.Contact}}:
{{range .Items}}[{{.Kind}}] {{.Object}} is {{.Status}}: {{.Output}}
Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}}), level {{.Level}}
{{end}}
The email digest's headers follow section 8's shape with three fields
computed differently: Subject is the rendered digest subject above,
Date is the last member's At (members are always ordered oldest
first), and Message-ID hashes every member's notification ID joined by
| together with the contact, rather than a single notification ID.
The default Telegram digest text, also used verbatim for Slack's digest message since a single Slack message is already plain text:
{{.Count}} notifications:
{{range .Items}}[{{.Kind}}] {{.Object}} is {{.Status}}: {{.Output}}
{{end}}
A webhook digest POSTs the same per-notification body section 4 documents, once per member, wrapped in an envelope:
{"notifications": [ <body>, <body>, ... ]}
Digest templates are fixed defaults in this version: there is no CUE field
to override a digest subject, body or text the way subject_template,
body_template and template override the single-message templates in
section 8. That override, along with group_by/label-based grouping,
group_interval, and re-notifying inside an already-open group, is
out of scope for now.
11. Security considerations
Every credential an adapter needs, the SMTP password, the Telegram bot
token, the incident.io token, is a process-level flag or environment
variable and never part of the CUE config, so a configuration can be
reviewed and shared without carrying a secret. The one exception is a
slack contact's webhook_url: it is both address and credential, so a
config carrying one is not shareable the way the rest of contacts: is.
The command adapter runs processes on the delivering host with the
arguments the commands map specifies; that map is trusted configuration.
Webhook, Telegram, PagerDuty, Slack and incident.io deliveries are
outbound HTTPS to operator-chosen or vendor endpoints; the notifier opens
no listener of its own beyond /metrics on -listen.