Monitoring and Alerting

01What turns a metric breach into a page — one bad sample, or something steadier?

A rule fires only after its condition holds for a set duration — a breach enters pending first, and only a breach that survives the full for-window ever pages anyone.

02One root cause trips fifty rules — how many notifications go out?

One: firing alerts are deduplicated and grouped by shared labels into a single incident, and inhibition rules suppress downstream alerts entirely while the upstream cause is already firing.

03Who gets paged, and what happens if they sleep through it?

Routing resolves the current on-call from the schedule (handoffs, overrides, timezones included) and escalates on an acknowledgment timeout: primary, then secondary, then manager.

04We deploy tonight — can we mute the expected alerts without editing the rules?

Yes — silences and maintenance windows match alerts by label for a bounded time and expire on their own, so a mute can never quietly outlive the deploy it was created for.

05How does a page physically reach a human?

Per-severity channel fan-out — push, SMS, phone call, chat, email — with delivery confirmation and automatic retry on another channel when one fails.

06Who pages us when the alerting system itself is down?

An always-firing heartbeat alert lands on an independent receiver; when the heartbeat stops arriving, that receiver — not this system — raises the alarm.

01How long from a real breach to a phone buzzing?

Under about two minutes for paging severity beyond the rule’s own for-duration — evaluation interval plus grouping wait plus routing must stay a small, known constant.

02Fifty thousand rules — can evaluation actually keep up?

Every rule is evaluated on schedule even at tens of thousands of rules — evaluation shards by rule group, and a shard falling behind is itself an alert.

03The evaluator crashes mid-cycle — is an alert lost?

No — evaluators run as a redundant pair and notification delivery is at-least-once; a duplicate page is annoying, a swallowed page is an unseen outage.

04What is the real cost of a false page?

The next real one — every needless 3 a.m. page trains on-call to ignore alerts, so pages-per-week is budgeted and reviewed like an SLO (the reliability target you promised), not shrugged off.

05Can the alerting path share infrastructure with what it watches?

As little as possible — the paging path avoids the network, storage, and queues it watches, because a shared dependency fails exactly when the page matters most.