Deciding that an object is broken and telling someone are two different
problems, and jaque keeps them in two different places. The state machine
decides; a notification policy names who is told and under what rule. A
policy is a named entry under notifications:, built from contacts, a
set of time windows and a ladder of escalation levels, and it is attached
to a host or service by name. An object with no policy never notifies,
and that is the default. The ladder's threshold and renotify_interval
are evaluated by the notify package on every notifiable transition; see
Escalations and windows
for the exact mechanics.
1. Contact types
A contact is one of seven discriminated variants of #Contact, told
apart by type. Credentials live outside the contact, with one
exception: a slack contact's webhook_url is both address and
credential.
contacts: {
alice: {type: "email", address: "[email protected]"}
oncall_hook: {type: "webhook", url: "https://hooks.example.com/oncall"}
bob: {type: "command", command: "page", args: ["-q"], address: "[email protected]"}
dave: {type: "telegram", chat_id: "555444333"}
pd: {type: "pagerduty", routing_key: "R0UT1NGK3Y"}
chat: {type: "slack", webhook_url: "https://hooks.slack.com/services/T0/B0/XXXX"}
inc: {type: "incidentio", alert_source_config_id: "01ABCDEF"}
}
| Type | Required field | Delivered by |
|---|---|---|
email |
address |
SMTP, or the system mail binary when no relay is configured |
webhook |
url |
an HTTP POST |
command |
command |
a top-level command template, run as a process |
telegram |
chat_id |
the Telegram Bot API |
pagerduty |
routing_key |
the PagerDuty Events API v2 |
slack |
webhook_url |
an HTTP POST to the webhook |
incidentio |
alert_source_config_id |
the incident.io alert events HTTP source |
An email contact also takes subject_template and body_template; a
telegram contact also takes template. Each defaults to "", meaning
"use the default template". See Adapters
for the fields available to a template and the default text.
email, webhook, telegram and slack contacts also take
group_wait, a duration string defaulting to "0s", meaning no
grouping. A non-zero group_wait buffers this contact's notifications
into one digest per window instead of sending each as it arrives:
alice: {type: "email", address: "[email protected]", group_wait: "30s"}
command, pagerduty and incidentio contacts do not take
group_wait. See Adapters for why, and
for the digest formats.
A command contact names a key in the top-level commands map, the same
indirection a command check uses. args
is passed through; address is what $CONTACTEMAIL$ expands to for that
contact and defaults to the empty string. webhook and command are how
anything without a native adapter is reached: a pager service, a chat
incoming webhook, a script that speaks whatever the receiver speaks. If
it accepts HTTP or command-line arguments, it is a contact.
A telegram contact carries only chat_id. The bot token is a
process-level secret, configured once per jaque instance; see
Adapters.
Any contact may also carry notification_period, the name of a
timeperiod outside which it is not reachable; see
Timeperiods.
2. A policy
timeperiods: business_hours: {
ranges: {
monday: "09:00-17:00"
tuesday: "09:00-17:00"
}
}
notifications: business_hours: {
period: "business_hours"
levels: [
{contacts: ["alice"], threshold: 3, renotify_interval: "30m"},
{contacts: ["oncall_hook", "bob", "dave"], renotify_interval: "10m"},
]
}
period names a timeperiod from the top-level timeperiods map; empty,
the default, means the policy is always open. See
Timeperiods for the range syntax. levels is
a non-empty list of escalation levels evaluated in order, each naming the
contacts it reaches. A policy with no levels is a config error rather
than a silent policy. The semantics of threshold, renotify_interval
and the periods are specified in
Escalations and windows.
3. Wiring a policy to an object
A host or service refers to a policy by name through its notification
field.
hosts: gw: {
address: "192.168.1.1"
check: {type: "icmp", host: "192.168.1.1"}
notification: "business_hours"
services: ssh: {
check: {type: "tcp", address: "192.168.1.1:22"}
notification: "business_hours"
}
}
notification defaults to "", and an empty name means the object never
notifies. That is exactly the behaviour every configuration written before
contacts and policies existed already had, so adding notifications is
opt-in per object and never a silent change to an existing config. A
service does not inherit its host's policy; each object names its own.
4. What a policy is not
A policy does not decide that something is broken. Problem-class notifications come only from hard transitions of the state model; a single failed check never reaches a policy. A policy also does not deliver: it produces a decision that the engine appends to the event log, and delivery is a separate concern with its own durability rules, described in Adapters.
5. Switching notifications off at runtime
DISABLE_NOTIFICATIONS and ENABLE_NOTIFICATIONS switch every
notification off or on for the whole program; the SetProgramNotifications
call does the same.
DISABLE_HOST_NOTIFICATIONS/ENABLE_HOST_NOTIFICATIONS and
DISABLE_SVC_NOTIFICATIONS/ENABLE_SVC_NOTIFICATIONS switch one object
off or on; the SetObjectNotifications call does the same, for one
object or every object a label selector matches.
DISABLE_CONTACT_HOST_NOTIFICATIONS/ENABLE_CONTACT_HOST_NOTIFICATIONS
and DISABLE_CONTACT_SVC_NOTIFICATIONS/ENABLE_CONTACT_SVC_NOTIFICATIONS
switch one contact off or on for one object kind; the
SetContactNotifications call does the same. Every one of these toggles
is a fact on the event log, so it survives a restart.
6. Security considerations
A contact is a destination. SMTP credentials, the Telegram bot token and
the incident.io token are process-level flags or environment variables and
are not expressible in contacts:, so a config file can be shared without
leaking those. Two contact types carry a value that is itself a secret:
the slack webhook URL and the pagerduty routing key. A config that
declares either has to be treated as a secret; see
Adapters. A command contact runs a process on the host that
delivers; the commands map is part of the trusted configuration, and
args are passed to it verbatim.