The biggest enemy of an alerting system isn't a missed alert — it's noise. When hundreds of notifications a day train a team to stop reading them, the one that matters gets buried. This post breaks down how to design alert rules in the Whaleal Platform alerting module: from what a single rule should contain, to severity, suppression and escalation, filtering out noise one layer at a time.
Illustrative numbers to show what each layer does · not statistics from any real cluster
Alert fatigue isn't a team that "doesn't care enough" — it's a design problem. When a notification is wrong nine times out of ten, the natural human response is to stop reading it carefully, including the one time it's right.
"Alert when CPU exceeds 80%" is only half a rule. A rule that actually interrupts people less answers five questions at once.
The most commonly skipped — and most effective — noise reducer is the duration. Almost every metric has momentary spikes; requiring it to stay over the line for a short window filters out a large share of notifications nobody needed to act on. Treat any threshold number as a starting point and tune it against your own cluster's baseline.
Two alerts can both be "over threshold" and still differ wildly in urgency — a disk about to fill versus CPU slightly elevated. The point of tiers: the more urgent it is, the louder it should be; the less urgent, the quieter.
Even well-designed rules can't stop "one incident triggers a pile of alerts". Suppression mechanisms make sure one event is announced only once.
While a rule keeps crossing its threshold without recovering, notify once, then send a single "resolved" message — not a new alert every collection cycle.
For planned restarts, upgrades and migrations, define a silence period in advance. Related alerts stay quiet during the window and monitoring resumes automatically afterwards.
When an upstream component is already failing, there's no need to notify separately about everything downstream. Report the root cause first and fold the derived alerts under it.
The most dangerous case isn't having no alert — it's an alert sent to someone who happened to be unreachable. Escalation makes sure a critical alert always has an owner.
A P1 alert fires and the current on-call DBA is notified right away.
No acknowledgement: notify the backup on-call and raise the reminder intensity.
Still no response: escalate to the team lead so it never falls through the cracks.
The tiers and intervals here are common starting examples. For your team, set them according to your own on-call process and how much delay the business can tolerate.
From rule design to suppression and escalation, the goal of the Whaleal Platform alerting module is simple: make every notification worth taking seriously.