Automated alert triage for SRE teams
Automated alert triage gives an SRE team a first reading of every alert. Noci AI triages every alert as it arrives and scores its noise, with the reasons shown. By default paging runs alongside triage; you can have it wait. Noisy alerts can be kept off an escalation policy.
What does alert triage solve for an SRE team?
An SRE team does not suffer from too few alerts. It suffers from too many that look alike. Each page gets the same first questions: is this real, how urgent is it, has it fired before, and is it connected to something else that is firing? A tired person answers them at 3 a.m., one alert at a time, and the answers are different at 3 a.m. than they would be at noon.
Automated triage moves that first reading into software. Every alert is read as it arrives, and the result is a score and the reasons for it. Where triage comes first, the people who are paged start with a reason, and the alerts that were only noise stop competing for their attention. The wider problem, and how a team works through it by hand, is in the 30-day plan for alert fatigue.
How does Noci AI triage an alert?
Noci AI triages every alert as it arrives. What you see for each one:
| What you see | What it tells you |
|---|---|
| A noise score | How noisy the alert is |
| The reasons | Why it scored that way, shown beside the score |
| Whether it is kept off a policy | Whether the alert is above the flakiness score you set for an escalation policy |
The score and the reasons are visible to the person who reads the alert, not hidden in a rule. That is what makes triage something a team can trust: a decision with a reason can be checked, and a decision without one can only be believed.
Should paging wait for triage?
By default the page goes out at once and AI triage runs alongside it from the first alert. If you would rather interrupt people only for alerts triage lets through, you can have paging wait for triage. The choice is yours, not the tool’s.
- Triage alongside paging. The default. AI triage runs alongside paging from the first alert. The page goes out at once, and the triage result arrives while the person is picking up.
- If you choose triage before anyone is paged, paging waits until the alert has been triaged, and the people who are paged start with the score and the reasons.
Waiting for triage suits a team whose pages are mostly noise and whose services can wait a moment. Paging alongside suits a team that would rather be woken for a false alarm than late for a real one.
How are noisy alerts kept off the pager?
A noise score is only useful if something acts on it. In Noci AI you can keep alerts above a flakiness score off an escalation policy. Three things follow from that:
- You set the limit. The score is the tool’s reading. The line above which an alert stays off a policy is yours.
- Look for visibility. An alert kept off a policy should still show its score and reasons, so a wrong limit can be found.
- The policy still belongs to you. Which alerts reach which people is an escalation policy, and policies are yours to write. For rules a team can defend, see suppression by policy.
What stays with your team?
Triage does not decide who is on call, and it does not replace the people who are. On-call schedules and escalation policies still choose who is paged and in what order. Triage adds a first reading beside, or if you choose in front of, that decision, so the person who is paged has context sooner. The score is there to help a person decide.
How do you tell triage is working?
Judge it on your own history, not on a vendor’s claim. Look at:
- Pages that needed no action. Fewer is the goal, and what remains should come with reasons you can read.
- Alerts kept off a policy that later mattered. Each one is a limit or a rule to adjust.
- How often people overrule a score. Frequent overrides mean the reasons are not persuasive.
- Context on arrival. Whether the person has context when they look.
For the plain-language definition, see what automated alert triage is.
Questions
Does triage delay the page?
Only if you choose it to. By default the page goes out at once and AI triage runs alongside paging from the first alert. If you choose, paging waits for triage, which is triage before anyone is paged. Noci AI triages every alert as it arrives either way.
How does an SRE team see why an alert was scored as noise?
Each alert carries its noise score and the reasons beside it, so a person can read the decision and challenge it.
Can triage keep an alert from paging anyone?
You can keep alerts above a flakiness score off an escalation policy. Look for a tool that keeps the score and the reasons visible, so a rule that is wrong can be found and fixed.
Does triage replace the on-call schedule?
No. On-call schedules and escalation policies still decide who is paged. Triage decides what deserves their attention and shows why.
Where should a team start?
Start with the alerts that page most often and need no action. Read their scores and reasons for a week, then decide which to keep off the escalation policy.