Incident management for Kubernetes alerts

How does Noci AI relate to Kubernetes?

Noci AI has a Kubernetes connector. You deploy it in the cluster you want Noci AI to watch, and it reports two things: an inventory of the cluster’s workloads, and a feed of the changes made to them. Alerts about the cluster arrive separately, through the tools that raise them.

What does the Kubernetes connector do?

The connector looks at the cluster on a schedule you set and sends Noci AI what it finds.

• Workloads. The inventory covers the cluster’s workloads, so Noci AI knows what runs there.
• Changes. Each change is recorded with the time it was seen, and labelled by who made it: by hand, Argo CD, Flux or CI. The change feed is what agents read when they ask what changed before an alert.
• Scope. One connector watches one cluster, which you name in Noci AI. You choose the namespaces: all except the system ones, only the ones you list, or all except the ones you list.

The connector lists workloads, services, ingresses, autoscalers, config maps and namespaces. It never reads secrets, and it cannot change anything in your cluster.

How do Kubernetes alerts reach Noci AI?

Through the alerting and telemetry tools you already use. These are the listed integrations that fit:

• Prometheus Alertmanager
• Datadog
• New Relic
• Amazon CloudWatch

Any other tool that can send a webhook reaches Noci AI too, with its own URL for each source. The integrations page lists them all under alert sources. Which of them carries your cluster alerts depends on how your clusters are monitored today. Noci AI works from what those tools send, so the quality of an alert’s labels and description is the quality of the starting point.

What does Noci AI do once the alert arrives?

The same as for any alert. Its agents work the alert as it arrives, with your telemetry, your recent code changes and the cluster’s changes from the connector, and report a hypothesis with the evidence behind it. A person reads the evidence and decides what to do.

Then there is the page. Noci AI pages by phone, SMS, Slack channel and Microsoft Teams, as your escalation policy says. The policy is yours, and Noci AI follows it.

What does that look like for one alert?

An invented example. A service called payments-api runs on your cluster, and your Alertmanager rules send an alert when its error rate climbs. The alert reaches Noci AI through the Alertmanager integration. Agents start on it at once, investigate with your telemetry and your recent code changes, and look at what changed in the cluster before the alert. They report a hypothesis with the evidence. Your escalation policy pages the person on call by phone, SMS, Slack channel or Microsoft Teams. The person opens the evidence and decides what to do.

What do cluster alerts look like in practice?

A cluster rarely fails in one place, so a problem can raise several alerts at once. Noci AI reads every alert that reaches it; how it triages them is in alert triage for SRE teams.

Because Noci AI works from the alerts your tools send, the content of an alert matters more than the platform it comes from. An alert that names the service, the symptom and a link to where you look first gives an investigation more to start with than one that says only that a threshold was crossed. Time spent improving your alert text is time spent on the investigation.

What should you check before you start?

• Which tool carries your cluster alerts today? That is the integration to set up.
• Which cluster should Noci AI watch, and which namespaces? The connector watches one cluster, and you choose the namespaces.
• What does an alert say? Labels and descriptions that name the service and the symptom give the investigation more to start with.
• Who is paged, and how? Set the escalation policy and the paging channels before the first real alert.
• Where does your code live? Recent code changes come from the code integration, GitHub.