Incident correlation for Datadog alerts

How does Noci AI work with Datadog alerts?

Datadog is listed among Noci AI’s alert and telemetry integrations, next to New Relic, Amazon CloudWatch, Prometheus Alertmanager and webhooks. Your monitors keep raising alerts where they do now. Those alerts reach Noci AI, and Noci AI’s agents take it from there.

What does correlation mean for a Datadog alert?

It means asking what else was happening when the alert fired. An alert from a monitor says one signal crossed a line. The useful question is what changed around it: a recent code change, other signals moving at the same time, or the same alert firing before. Noci AI’s agents investigate with your telemetry and your recent code changes and report a hypothesis with the evidence, so the person who picks up the page starts from a reading instead of an empty screen.

Timing is a clue, not a cause. The agents show their evidence, and a person weighs it. For the general version of the approach, see correlating alerts and deploys.

What does it look like for one alert?

An invented example. A Datadog monitor on a service called orders-api fires. The agents read the service’s telemetry around the time it fired and notice that the error rate began to climb shortly after a code change to the same service. They report the hypothesis that the change is responsible, and they list what they read: the telemetry they queried and the change they matched. The on-call engineer opens the evidence, agrees or does not, and decides whether to roll back. The agents did the gathering and offered a reading, and the engineer made the decision.

Can agents act on a Datadog monitor?

Agents can mute or unmute a Datadog monitor. Muting hides a signal, so the choice of whether an agent may do it alone is yours: you choose, step by step, whether an agent asks before it acts. A team new to this might have agents ask first for every mute and let them run on their own only for the steps it has watched go well.

What do you keep in Datadog?

Your monitors and the way you raise alerts. Set up the Datadog integration, keep your monitors as they are, and watch how the first few alerts are worked before you change anything.

What should you check before connecting it?

• Which monitors should page? Decide first, because every alert that reaches Noci AI is worked.
• Where does your code live? The code side comes from GitHub, so that is where recent changes are read.
• Who may let an agent act? Settle which steps ask first, for example muting a monitor, before the first real alert.
• How will you review it? Read the first few investigations against what your team found. The integrations page lists the other tools you can connect.