A notification policy is evaluated by one pure function: given the object's notification state, the policy, the kind of notification the state machine asked for and the current time, it returns either the contacts to notify or a reason not to. This page specifies that function. It has two independent clocks, the re-notification interval and the escalation threshold, and one gate, the time windows, and the three are easy to conflate. Nothing here performs I/O; the decision is a value the engine appends to the event log.
1. Levels
levels: [
{contacts: ["alice"], threshold: 3, renotify_interval: "30m"},
{
contacts: ["oncall_hook", "bob", "carol", "dave"]
renotify_interval: "10m"
disabled_kinds: ["FLAPPING_START", "FLAPPING_STOP"]
},
]
levels is a non-empty list; a policy without one is a config error. A
problem starts at the first level and only ever moves down the list while
it persists. The level never decreases during a problem and returns to
the first level on recovery.
| Field | Type | Default | Meaning |
|---|---|---|---|
contacts |
non-empty list of contact names | - | who this level notifies |
threshold |
int, >= 0 |
0 |
how many notifications this level sends before the next level takes over; 0 means the ladder never escalates past this level |
renotify_interval |
duration | 1h |
the minimum gap between two PROBLEM notifications from this level |
disabled_kinds |
list of kinds | [] |
kinds this level does not send |
The kinds are PROBLEM, RECOVERY, STATUS_CHANGE, FLAPPING_START,
FLAPPING_STOP, ACKNOWLEDGEMENT, DOWNTIME_START, DOWNTIME_END,
DOWNTIME_CANCELLED and CUSTOM; see Notification kinds. A level that
should only page can turn off the flapping meta-notifications with
disabled_kinds rather than needing a second policy.
2. Re-notify versus escalate
These are two independent counters. renotify_interval throttles
the current level: a PROBLEM evaluation that arrives before the interval
has elapsed since the level's last send is suppressed with reason
TOO_SOON. threshold counts the level's sends: once the current level
has sent threshold notifications, the next evaluation escalates, the
count resets, and the next level's contacts and interval apply from then
on.
A PROBLEM evaluation is not only triggered by a check result: a
still-open, non-flapping hard problem also arms its own re-notify timer at
renotify_interval, so the next evaluation happens on schedule even if no
new result ever arrives. An engine that was down, or has just gained
ownership of the object, re-notifies an open problem once when it comes
back or takes over. It never sends once per interval it missed while it
was gone.
With the example above, whose first level has threshold: 3 and
renotify_interval: 30m, a problem that keeps being evaluated runs like
this:
t+0 level 0 alice PROBLEM count 1/3
t+30m level 0 alice PROBLEM count 2/3 (30m interval elapsed)
t+60m level 0 alice PROBLEM count 3/3 threshold reached
t+90m level 1 oncall_hook PROBLEM count 1 escalated; 10m interval from here
t+100m level 1 oncall_hook PROBLEM count 2
...
recovery level 1 oncall_hook RECOVERY ladder resets to level 0
Two details follow from the counters being separate. The interval that
throttles is the one on the level that sent last, so the ladder slows
down or speeds up as it moves. And the last level's threshold is
irrelevant: with no level after it there is nowhere to escalate to, and
the count keeps growing without effect.
3. Recovery and flapping
A RECOVERY always notifies the level the problem had reached, then
resets the ladder: level and count return to the base, the re-notify
throttle is cleared so the next problem's first notification is never
swallowed by the closed one, and a stale acknowledgement is dropped. A
problem that recovers and immediately returns starts again at level 0.
FLAPPING_START and FLAPPING_STOP notify the current level without
touching the ladder: they neither count toward threshold nor reset the
re-notify throttle. They are meta-notifications about the object's
stability rather than part of the problem's cadence.
ACKNOWLEDGEMENT goes out when an acknowledgement asks for it (the
notify flag of the command, on by default in the dashboard), and the
three DOWNTIME_ kinds when a window starts, ends, or is cancelled while
it is in effect. All four notify the current level without touching the
ladder, carry the author and the comment, and can be turned off per level
with disabled_kinds. CUSTOM is what the SEND_CUSTOM_HOST_NOTIFICATION
and SEND_CUSTOM_SVC_NOTIFICATION commands, or the SendCustomNotification
call, send: the current level gets the author and the comment, and a
downtime silences it.
4. Windows
period: "business_hours"
period names a timeperiod from the top-level timeperiods map; empty,
the default, is always open. See Timeperiods
for the range syntax and how a day resolves.
Outside the period a notification of any kind is suppressed with reason
OUTSIDE_PERIOD. For PROBLEM, that is not the end of it: a still-open,
non-flapping hard problem arms a period-reopen timer, and when the period
opens the level is re-evaluated as PROBLEM on that timer, no new check
result required. Every other kind is dropped rather than queued: a
RECOVERY, STATUS_CHANGE or flapping meta-notification suppressed by
OUTSIDE_PERIOD is not delivered when the period opens, and the next
notifiable transition is evaluated on its own terms. A policy that wants
a recovery announced at 09:00 for a problem that happened at 03:00 does
not get one.
An escalation level may also carry its own period, gated the same way
after the policy's own. A contact may carry notification_period: a
contact outside its period is dropped from the level's recipients the
way a disabled contact is, with nothing queued for its window to
reopen.
5. Order of suppression
Evaluate checks its gates in a fixed order and stops at the first that
fires. Each reason is logged by name and counted in
jaque_notify_suppressed_total, whose reason label carries the same
name in lower case (downtime, too_soon, ...).
| Order | Condition | Reason | Silences |
|---|---|---|---|
| 1 | notifications are switched off for the program | PROGRAM_DISABLED |
every kind |
| 2 | notifications are switched off for the object | OBJECT_DISABLED |
every kind |
| 3 | the object is in a downtime | DOWNTIME |
every kind except ACKNOWLEDGEMENT and the DOWNTIME_ kinds |
| 4 | the object is acknowledged | ACKED |
PROBLEM and STATUS_CHANGE only; RECOVERY still goes out |
| 5 | the object is unreachable | UNREACHABLE |
problem-class kinds only; RECOVERY and flapping meta-notifications still go out |
| 6 | an explicit dependency is failing | DEPENDENCY |
every kind, or problem-class kinds only, per the dependency's own suppress |
| 7 | now is outside the policy's period | OUTSIDE_PERIOD |
every kind |
| 8 | the current level does not exist | NO_LEVEL |
every kind |
| 9 | the current level's period is closed | OUTSIDE_PERIOD |
every kind |
| 10 | the kind is in the level's disabled_kinds |
KIND_DISABLED |
that kind |
| 11 | every contact of the level is switched off or outside its period | NO_CONTACTS |
every kind |
| 12 | a PROBLEM arrives before renotify_interval elapsed |
TOO_SOON |
PROBLEM only |
A toggle arms no timer of its own: switching notifications back on does not replay what was silenced while they were off, and an open problem notifies again only on its next check result.
An acknowledged problem therefore still announces its recovery, and a downtime silences even that; what a downtime does not silence is its own start and end, or an acknowledgement made inside it. Acknowledgements and downtimes are suppression signals rather than states; the state model keeps transitioning underneath them and the event log records every transition whether or not anyone was told.