State model

This is the part Nagios got right: the rules that turn a stream of check results into “should a human be paged”, without paging anyone for a single dropped ping or a five-minute network blip.

Soft and hard states

A single failed check never notifies anything. The object enters a soft state and gets re-checked quickly (retry_interval, not check_interval). Only after max_attempts (Nagios’s max_check_attempts) consecutive failures does the state become hard — and only a hard-state transition notifies. A flaky link that recovers on the second attempt never wakes anyone up; a real outage does, within a bounded number of retries.

attempt 1 FAIL --> SOFT (retry fast)
attempt 2 FAIL --> SOFT (retry fast)
attempt 3 FAIL --> HARD (notify)

max_attempts lives on every check’s schedule (the config schema’s #Schedule), default 3.

Flap detection

An object oscillating OK/CRITICAL/OK/CRITICAL every other check is not “repeatedly broken” — it’s flapping, and notifying on every transition is spam nobody reads twice. jaque tracks an exponentially weighted moving average (EWMA) of the state-change rate per object; when it crosses high_threshold the object is flagged flapping and further problem/recovery notifications are suppressed until the rate drops back below low_threshold. Flapping start/stop are themselves notified once, as meta-events — the one exception to “only hard states notify”: they may fire from a soft state too, because flapping is a property of the check history, not of the current state’s confirmation status.

check: {
  type: "tcp"
  address: "10.0.0.1:443"
  flap: {enabled: true, alpha: 0.1, high_threshold: 0.3, low_threshold: 0.15}
}

Acknowledgements

“I know, I’m on it.” An ack attaches to the current problem and silences further notifications for it. Sticky acks survive further bad-state transitions (WARNING → CRITICAL is still acked); non-sticky acks clear on the first state change. Either way, an ack is always cleared automatically on recovery — you never have to remember to un-ack something that fixed itself.

Downtimes

Scheduled maintenance windows: while a downtime is active on an object, no notification fires and every event in that window is marked as occurring during a downtime, so history stays honest about what was expected.

Downtimes are fixed-window only. jaque’s config schema and its FIFO parser both require fixed=1 and reject a nonzero trigger_id — Nagios’s flexible downtimes (which start on the next problem inside the window, not immediately) and triggered downtimes are not implemented. A SCHEDULE_HOST_DOWNTIME line asking for either is rejected rather than silently treated as fixed.

Dependencies and unreachability

Hosts can declare parents. When a parent goes hard DOWN, its dependents don’t get forty separate “I can’t reach this either” pages — the dependency graph recomputes reachability incrementally and flips every child to UNREACHABLE, a distinct state from DOWN that says “we don’t actually know, because the path there is broken.” One page for the router, silence for everything behind it.

router (hard DOWN)
  ├── switch-a --> UNREACHABLE (was previously OK)
  └── switch-b --> UNREACHABLE (was previously OK)

Escalations, re-notification and periods

A notification policy is a ladder of levels: each level names contacts, a threshold (how many notifications at the previous level before escalating, 0 means never escalate past it), and a renotify_interval (how often to repeat while the problem stays open). A period restricts when a policy’s levels are allowed to fire at all — outside it, notifications are suppressed, not queued. See Escalations and windows for the full shape.

Where this runs

All of the above — soft/hard, flap, acks, downtimes — is one pure state machine: given a state, an event and a clock, it returns the new state plus a list of effects as data (never actions taken directly). That purity is what lets jaque property-test thousands of random event sequences against invariants like “a problem notification never fires from a soft state” without starting a single goroutine. See Event sourcing for how those effects actually reach the world.