> Section: [5. Notifications](https://jaque.sh/docs/notifications/contacts-and-policies.md)
> Next: ingest/passive-checks
> Index: https://jaque.sh/llms.txt


An adapter turns a notification decision into a side effect: an SMTP
session, an HTTP POST, a process, a Telegram message. jaque has seven,
one per contact type, and none of them is trusted to remember anything.
Every decision is written to the event log before an attempt is made and
its outcome is written after, so whoever delivers can be killed, restarted
or replaced by another process and resume from the log with no duplicate
that already landed and no notification still owed. Delivery is at least
once, by design, and idempotent by notification ID.

Status: the path is tested end to end against a fake receiver, including
a flaky one, but no deployment has paged a human yet; see the
[status page](https://jaque.sh/docs/project/status.md). `JAQUE_TELEGRAM_TOKEN` is the
environment form of `-telegram-token` by the general rule that every flag
has a `JAQUE_` prefixed variable, documented in [CLI flags](https://jaque.sh/docs/reference/cli-flags.md).

## 1. The seven adapters

| Adapter | Contact type | Mechanism |
|---|---|---|
| email | `email` | `net/smtp` with STARTTLS and optional PLAIN auth, or the system `mail` binary |
| webhook | `webhook` | HTTP POST to the contact's `url` |
| command | `command` | a top-level command template run as a process, the Nagios model |
| telegram | `telegram` | the Bot API's `sendMessage` to the contact's `chat_id` |
| pagerduty | `pagerduty` | the Events API v2 enqueue endpoint, keyed by the contact's `routing_key` |
| slack | `slack` | an HTTP POST to the contact's `webhook_url` |
| incidentio | `incidentio` | the alert events HTTP source named by the contact's `alert_source_config_id` |

The command adapter is the primitive: it runs a process the way
Nagios's notification commands do, and it is what the email adapter
falls back to when no relay is configured. Anything not listed is
reached through `webhook` or
`command`; there is no plugin interface for adapters. The contact side
of this table is specified in
[Contacts and policies](https://jaque.sh/docs/notifications/contacts-and-policies.md).

## 2. Durability through the log

A notification has three events and a stable identity.


```text
engine decides            deliverer attempts            log
--------------            ------------------            ---
NotifyDecision  ------->  NotificationRequested  ----->  appended BEFORE the attempt
                          adapter.Send(...)
                          NotificationSent       ----->  appended on success
                      or  NotificationFailed     ----->  appended on permanent failure
```

`NotificationRequested` is appended before any adapter is called. The
deliverer, whichever process that is, folds the log on startup to find
every `NotificationRequested` without a terminal `NotificationSent` or
`NotificationFailed` and resumes exactly those. The notification ID is
derived from the object, the kind, the level and the instant the problem
began, never from the current time, so a re-notification of the same
problem and a replayed request after a crash produce the same ID and are
deduplicated against it.

The guarantee is at least once. If ownership of delivery moves between
processes while an attempt is in flight, the new owner may repeat a send
that the old one had started; it will never skip one. Retries use
backoff and stay inside one delivery shard, so two attempts for one ID
never run concurrently.

## 3. Where delivery runs: `-notify-delivery`

The engine that owns an object always decides whether to notify and
writes that decision to the log. Who performs the side effect is a
separate switch.

| `-notify-delivery` | Who delivers |
|---|---|
| `embedded` (default) | the same process that decided, under `-target all` or `-target engine` |
| `external` | one or more `-target notifier` processes, each pulling its share of open notifications from the log |

Under `external`, several notifiers split the work by object ownership
over their own membership bucket; [Cluster and
coordination](../deployment/cluster-and-coordination.md) describes the
split. A `-target notifier` process requires `-notify-delivery external`
on the engine upstream, or it finds nothing to deliver.

In either mode delivery is sharded by object across `-notify-workers`
workers (default 4), each with a bounded queue of `-notify-queue`
notifications (default 64). A slow receiver fills its shard's queue and
then blocks the notifier's log follower; nothing is dropped for being
slow. The block is visible as `jaque_notifier_blocked_total` counting up
and `jaque_eventlog_consumer_lag{follower="notifier"}` growing, and
`jaque_notifier_pending` shows how many notifications sit in the shards.

## 4. Webhook body

The JSON body posted to a `webhook` contact's `url`:

| Field | Notes |
|---|---|
| `notification_id` | stable identity, also sent as the `Idempotency-Key` header |
| `object_id` | the object the notification is about, e.g. `web1/http` |
| `kind` | `PROBLEM`, `RECOVERY`, `ACKNOWLEDGEMENT`, ... |
| `level` | the escalation level this send belongs to |
| `contact` | the contact ID |
| `at` | RFC3339Nano, when the decision to notify was made |
| `status` | the object's status at decision time: `OK`, `WARNING`, `CRITICAL` or `UNKNOWN` |
| `output` | the check's plugin output |
| `perfdata` | the check's performance data; the key is omitted when there is none |
| `problem_since` | RFC3339Nano, when the open problem began; `""` when the object has no open problem |

## 5. Runtime macros

A command-adapter template (and the `mail` fallback SMTP uses) can
reference these macros at dispatch time, alongside the identity macros
already available everywhere: `$NOTIFICATIONTYPE$`, `$NOTIFICATIONNUMBER$`,
`$CONTACTNAME$`, `$CONTACTEMAIL$`. Every other Nagios runtime macro still
fails template expansion.

A service or a process contact gets the service vocabulary; a host
contact gets the host vocabulary. A process has no macro family of its
own, since a process is not part of Nagios's object model and jaque adds
no third one for it.

| Macro | Value |
|---|---|
| `$SERVICESTATE$` | `OK`, `WARNING`, `CRITICAL` or `UNKNOWN` |
| `$SERVICEOUTPUT$` | the check's plugin output |
| `$SERVICEPERFDATA$` | the check's performance data |
| `$SERVICEDURATION$` | time since the open problem began, `XdYhZmWs` |
| `$LASTSERVICESTATECHANGE$` | unix seconds when the open problem began |

| Macro | Value |
|---|---|
| `$HOSTSTATE$` | `UP` when the status is OK, otherwise `DOWN` |
| `$HOSTOUTPUT$` | the check's plugin output |
| `$HOSTPERFDATA$` | the check's performance data |
| `$HOSTDURATION$` | time since the open problem began, `XdYhZmWs` |
| `$LASTHOSTSTATECHANGE$` | unix seconds when the open problem began |

`$HOSTSTATE$` never renders `UNREACHABLE`: reachability tracking is not
part of a notification, so a host's state macro is always one of `UP` or
`DOWN`. Nagios has a third host state, `UNREACHABLE`; this macro does
not carry it.

Both families also get `$TIMET$` (unix seconds), `$DATE$` (`YYYY-MM-DD`),
`$TIME$` (`HH:MM:SS`), `$SHORTDATETIME$` and `$LONGDATETIME$`, all as of
the notification decision.

## 6. SMTP

```text
-smtp-host, -smtp-port (default 25), -smtp-user, -smtp-pass, -smtp-from
```

With `-smtp-host` set, email contacts are delivered over SMTP with
STARTTLS; `-smtp-user` empty disables authentication. With `-smtp-host`
empty, email contacts are delivered by running the system `mail` binary
through the command adapter, which suits a host that already has local
mail delivery configured and nothing else.

## 7. Telegram

A `telegram` contact carries only a `chat_id`. The bot token is one per
jaque instance and is a process secret, set once via `-telegram-token` or
`JAQUE_TELEGRAM_TOKEN`, the same posture as SMTP credentials. With the
token empty, every `telegram` contact fails permanently rather than
silently.

To obtain a token, talk to [@BotFather](https://t.me/BotFather), send
`/newbot` and keep the token it returns. To find a `chat_id`, message the
bot (or add it to a group), then read
`https://api.telegram.org/bot<token>/getUpdates`: the `chat.id` on the
message you sent is the value for the contact.

## 8. Message templates

An email is plain text, CRLF-terminated, with a fixed set of headers in a
fixed order: `From`, `To`, `Subject`, `Date`, `Message-ID`, `MIME-Version`,
`Content-Type`. Only `Subject` and the body are templated; the rest are
computed from the delivery and the SMTP configuration.

The default subject and body:

```text
[jaque] {{.Kind}} {{.Object}} is {{.Status}}
```

```text
Object: {{.Object}}
Kind: {{.Kind}}
Status: {{.Status}}
Output: {{.Output}}
Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}})
Level: {{.Level}}
Contact: {{.Contact}}
At: {{.At.Format "2006-01-02 15:04:05 MST"}}
```

The default Telegram text:

```text
[{{.Kind}}] {{.Object}} is {{.Status}}
{{.Output}}
Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}})
Level {{.Level}}
```

An `email` contact overrides either or both with `subject_template` and
`body_template`; a `telegram` contact overrides its text with `template`.
Each defaults to `""`, meaning the default above:

```cue
alice: {
	type:             "email"
	address:          "alice@example.com"
	subject_template: "[{{.Level}}] {{.Object}} {{.Status}}"
	body_template:    "{{.Output}}\nSince {{.ProblemSince.Format \"15:04:05\"}}\n"
}
```

Every template renders against the same fields:

| Field | Meaning |
|---|---|
| `Object` | the object the notification is about |
| `Kind` | `PROBLEM`, `RECOVERY`, `ACKNOWLEDGEMENT`, ... |
| `Status` | the object's status at decision time |
| `Level` | the escalation level this send belongs to |
| `Contact` | the contact ID |
| `Output` | the check's plugin output |
| `Perfdata` | the check's performance data |
| `At` | when the decision to notify was made |
| `ProblemSince` | when the open problem began |
| `Duration` | time since the open problem began, `XdYhZmWs` |

`At` and `ProblemSince` are timestamps with a `.Format "layout"` method, so
a template can call `{{.At.Format "2006-01-02 15:04:05 MST"}}` or any other
layout rather than just printing the zero-argument value.

Every template, default or overridden, is parsed and executed once at
startup against a fixed probe value. A broken template or one that names a
field not in the table above fails the process at load time rather than at
the moment a notification is due.

HTML mail is not supported: the body is always `text/plain`, and there is
no HTML alternative part.

## 9. Incident platforms: PagerDuty, Slack, incident.io

PagerDuty and incident.io both group a problem's notifications into one
open incident that resolves itself when jaque says the problem is over.
What makes that possible is an alert key: one key per problem episode,
shared by the PROBLEM notification, every re-notification, every
escalation and the RECOVERY. The key is derived from the object ID and
the instant the problem began, never from the notification ID, which
varies by kind and level. An incident platform keyed on the
notification ID would open a new incident on every re-notification
instead of updating the one already open.

A `PROBLEM` or `STATUS_CHANGE` maps to PagerDuty's `trigger` event action
and to incident.io's `firing` status, a `RECOVERY` to `resolve` and
`resolved`, and an `ACKNOWLEDGEMENT` to PagerDuty's `acknowledge`
(incident.io has no equivalent and receives nothing). The flapping and
downtime kinds are not incidents: both adapters report them delivered
without posting, so a contact that should not receive them at all lists
them in the level's `disabled_kinds`.

PagerDuty additionally wants a severity, derived from the object's status
at decision time:

| Status | PagerDuty severity |
|---|---|
| `CRITICAL` | `critical` |
| `WARNING` | `warning` |
| `UNKNOWN` | `error` |
| `OK` | `info` |

The incident.io API token is a process secret the same way the Telegram
bot token is: `-incidentio-token` or `JAQUE_INCIDENTIO_TOKEN`. With the
token empty, every `incidentio` contact fails permanently rather than
silently.

Slack is the exception among the three: a `slack` contact's
`webhook_url` is itself the credential, since posting to it is the whole
authorization Slack asks for. Unlike the PagerDuty routing key and the
incident.io token, it is not held back as a process secret. It lives in
the config, on the contact.

## 10. Grouping

Dependency suppression already handles the case where a failure is
declared to depend on another: nothing is sent for the child while the
parent is the one that is broken. What it cannot handle is the residue:
an undeclared dependency nobody wrote down, the race window before a
failing parent has itself gone hard `DOWN`, or a correlated failure with
no dependency edge at all, like a bad deploy that breaks forty unrelated
checks at once. Grouping is the safety net for that residue: instead of
forty messages, a contact gets one digest. It is not a general-purpose
alerting router: there is no `group_by`, no `group_interval`, no
re-notify inside an open group, one window per contact and nothing more.

A digestable contact, one of `email`, `webhook`, `telegram` or `slack`,
may set `group_wait`, a duration. The first notification buffered for
that contact opens a window of that length; every further notification
for the same contact arriving before the window closes joins it. When
the window closes, everything buffered is delivered as one message,
oldest first. `group_wait` is scoped per contact rather than globally:
two contacts with different settings, or one grouped and one not, are
unaffected by each other.

`command`, `pagerduty` and `incidentio` contacts do not take
`group_wait`; the config schema rejects it there. The command adapter's
contract is one macro line per object: a digest has nowhere to put a
second object's macros. PagerDuty and incident.io already group on the
platform's own terms (section 9): an alert key ties every notification
for one problem episode into a single incident that opens once and
resolves once, which is a different and stronger guarantee than a time
window, so a second, weaker grouping mechanism underneath it would only
compete with it. `slack`, unlike those two, has no alert-key grouping of
its own: it is a plain HTTP POST to a webhook, section 9's exception, so
it benefits from `group_wait` exactly as `email` and `webhook` do.

Grouping does not change what lands on the event log. Every object still
gets exactly one `NotificationRequested`, and every notification and
contact pair still gets exactly one terminal `NotificationSent` or
`NotificationFailed`, precisely as section 2 describes. What differs is
delivery-side only: a buffered notification is an ordinary open log entry,
just viewed from memory until its window closes. A crash mid-window loses
nothing but time: the open entries re-fold on restart, re-buffer, and the
window restarts from zero, at most delaying delivery by another
`group_wait`. The window itself is not persisted anywhere; buying back
that delay would mean writing the window to the log, and the log's
job is delivery record-keeping rather than scheduling.

A window that closes with exactly one notification in it is delivered
through the ordinary single-message path, never as a one-item digest.
Grouping only changes anything once there is something to group. A window
with a PROBLEM and its RECOVERY both buffered does not cancel them
against each other the way some alerting systems do: both are delivered,
in the order they happened, inside the same digest.

The default email digest subject and body:

```text
[jaque] {{.Count}} notifications
```

```text
{{.Count}} notifications for {{.Contact}}:

{{range .Items}}[{{.Kind}}] {{.Object}} is {{.Status}}: {{.Output}}
  Since: {{.ProblemSince.Format "2006-01-02 15:04:05 MST"}} ({{.Duration}}), level {{.Level}}
{{end}}
```

The email digest's headers follow section 8's shape with three fields
computed differently: `Subject` is the rendered digest subject above,
`Date` is the last member's `At` (members are always ordered oldest
first), and `Message-ID` hashes every member's notification ID joined by
`|` together with the contact, rather than a single notification ID.

The default Telegram digest text, also used verbatim for Slack's digest
message since a single Slack message is already plain text:

```text
{{.Count}} notifications:
{{range .Items}}[{{.Kind}}] {{.Object}} is {{.Status}}: {{.Output}}
{{end}}
```

A webhook digest POSTs the same per-notification body section 4
documents, once per member, wrapped in an envelope:

```text
{"notifications": [ <body>, <body>, ... ]}
```

Digest templates are fixed defaults in this version: there is no CUE field
to override a digest subject, body or text the way `subject_template`,
`body_template` and `template` override the single-message templates in
section 8. That override, along with `group_by`/label-based grouping,
`group_interval`, and re-notifying inside an already-open group, is
out of scope for now.

## 11. Security considerations

Every credential an adapter needs, the SMTP password, the Telegram bot
token, the incident.io token, is a process-level flag or environment
variable and never part of the CUE config, so a configuration can be
reviewed and shared without carrying a secret. The one exception is a
`slack` contact's `webhook_url`: it is both address and credential, so a
config carrying one is not shareable the way the rest of `contacts:` is.
The command adapter runs processes on the delivering host with the
arguments the `commands` map specifies; that map is trusted configuration.
Webhook, Telegram, PagerDuty, Slack and incident.io deliveries are
outbound HTTPS to operator-chosen or vendor endpoints; the notifier opens
no listener of its own beyond `/metrics` on `-listen`.