These rules turn a stream of check results into "should a human be paged", without paging anyone for one dropped ping or a five-minute blip. Five mechanisms do that work: soft and hard states, flap detection, acknowledgements, downtimes and dependency reachability. Each is a rule inside one pure state machine, so the same sequence of results always produces the same decision. The semantics match Nagios's soft/hard model. This page states each rule, why it exists, and what it suppresses.
Everything on this page is shipped and property-tested. See Status.
1. Soft and hard states
A single failed check never notifies anything. The object enters a soft
state and is re-checked at retry_interval rather than check_interval,
so confirmation arrives in seconds instead of minutes. Only after
max_attempts consecutive failures, the same count Nagios calls
max_check_attempts, does the state become hard, and only a hard
transition notifies. Recovery from a hard
problem is itself a hard transition, which is why it notifies too. A flaky
link that recovers on the second attempt wakes nobody; a real outage pages
within a bounded number of retries.
att result state notify
--- -------- ------------------ --------------------
1 OK OK hard -
2 CRITICAL CRITICAL soft 1/3 - retry_interval
3 CRITICAL CRITICAL soft 2/3 - retry_interval
4 CRITICAL CRITICAL hard 3/3 PROBLEM -> oncall
5 CRITICAL CRITICAL hard - check_interval
6 CRITICAL CRITICAL hard - renotify_interval
7 OK OK hard RECOVERY -> oncall
att result state notify
--- -------- ------------------ --------------------
1 OK OK hard -
2 CRITICAL CRITICAL soft 1/3 - retry_interval
3 CRITICAL CRITICAL soft 2/3 - retry_interval
4 CRITICAL CRITICAL hard 3/3 PROBLEM -> oncall
5 CRITICAL CRITICAL hard - check_interval
6 CRITICAL CRITICAL hard - renotify_interval
7 OK OK hard RECOVERY -> oncall
max_attempts lives on every check's schedule (the schema's #Schedule)
and defaults to 3; retry_interval defaults to 15s and check_interval
to 60s. Once hard, the object is re-checked at check_interval again, and
a policy's renotify_interval decides how often the still-open problem is
repeated (section 6).
While a problem stays hard, it keeps re-notifying on its own timer:
renotify_interval on the escalation level arms a wakeup that fires the
next evaluation even if no new check result ever arrives, so a still-open
problem is not silent just because every new result repeats the same
status. A result that does arrive before the interval has elapsed is
still re-evaluated, but it is suppressed as too soon to send.
2. Flap detection
An object oscillating OK, CRITICAL, OK, CRITICAL every other check is not
repeatedly broken; it is flapping, and notifying each transition is spam
nobody reads twice. jaque keeps an exponentially weighted moving average
(EWMA) of the state-change rate per object. When it crosses
high_threshold the object is flagged flapping and further problem and
recovery notifications are suppressed until the rate falls back below
low_threshold. The two thresholds are deliberately different: a single
threshold would itself flap.
Flapping start and stop are notified once each, as meta-events. They are the one exception to "only hard states notify" and may fire from a soft state, because flapping is a property of the check history rather than of the current state's confirmation.
check: {
type: "tcp"
address: "10.0.0.1:443"
flap: {enabled: true, alpha: 0.1, high_threshold: 0.3, low_threshold: 0.15}
}
The result that ends a flapping episode emits FlappingStop and, by
default, nothing else for a problem that persists: the next result
re-notifies it. This is what Nagios does. Setting
flap.notify_on_flap_stop: true on the check emits the problem
notification right after FlappingStop on that same result. A recovery
on that result is always sent.
Flap detection also has a runtime switch, for the whole program or for
one object, that an operator flips without a reload. Switching it off on
an object already flapping does not emit FlappingStop: the flap value
and the flapping flag clear silently on the object's next result, the
same as turning flap.enabled off in config would.
3. Acknowledgements
"I know, I am on it." An ack attaches to the current problem and silences further notifications for it. A sticky ack survives transitions between non-OK states: WARNING to CRITICAL is still acked. A non-sticky ack clears on the first state change. Either way an ack clears on recovery, so nothing stays acked after it has fixed itself and nobody has to remember to un-ack it.
4. Downtimes
A downtime is a scheduled window during which no notification fires for the object and every event is marked as occurring in downtime, so the history still shows what was actually expected at the time.
A downtime can be fixed (the window itself silences the object), flexible (it starts on the first problem inside the window and then lasts a fixed duration) or triggered (it starts and stops with another downtime). A recurring downtime is declared in config against a timeperiod rather than scheduled: the object is in downtime whenever the period is open. See Downtimes for all four.
5. Dependencies and unreachability
Hosts declare parents. When a parent goes hard DOWN, the dependency
graph recomputes reachability incrementally and moves every child to
UNREACHABLE, a state distinct from DOWN that means "unknown, because the
path there is broken": a router going hard DOWN with switch-a and
switch-b behind it turns both switches UNREACHABLE in the same
recomputation, and the router coming back returns them to the state
their own checks last established.
While an object is unreachable, its problem-class notifications are suppressed: the notifier never sends them in the first place. Recoveries and flapping meta-notifications still go out. One page fires for the router; everything behind it stays silent until reachability is recomputed.
parents is not the only dependency jaque knows. An explicit dependency
names a rule over one or more masters and which of the rule's outcomes
count as failed, so a service can depend on another service, or on
anything a business process rule can express,
without a path in the reachability graph. See
Dependencies.
6. Escalations, re-notification and periods
A notification policy is a ladder of levels. Each level names contacts, a
threshold (how many notifications at the previous level before the
problem escalates; 0 means never escalate past it) and a
renotify_interval (how often to repeat while the problem stays open). A
period is a named timeperiod; it restricts
when a policy's levels may fire at all, and outside it notifications are
suppressed. A still-open, non-flapping hard problem's
suppressed re-notification is not lost: it fires again on its own timer
when the period reopens. Escalations and
windows has the full shape.
7. Where this runs
All of the above is one pure state machine. Given the current state, the check's configuration and one result, it returns the new state plus a list of effects as data; it takes no action itself and reads no clock, because the result carries its own timestamp. That purity is what lets jaque property-test thousands of random result sequences against invariants such as "a problem notification never fires from a soft state" and "nothing problem-class fires while flapping" without starting anything. Event sourcing describes how those effects reach the world.