Building an AI Agent for Production Alert Triage

Published October 22, 2026 • 7 min read

A practical architecture for turning growing alert volume into evidence-backed, auditable triage with LangChain, LangGraph, LangSmith, and MCP.

Alerting has a scaling problem. At first, every notification gets attention. As a system grows, the number of alerts grows faster than the people available to investigate them. Engineers learn which alerts are usually harmless, dashboards become background noise, and the important signal risks being missed.

The answer is not to hand every alert to an unconstrained LLM. The useful pattern is an agent that gathers facts from approved sources, chooses a safe verification path when appropriate, and returns an evidence-backed report for people to act on.

The objective is attention management, not autonomous incident response. The agent reduces the time from alert to useful evidence; people remain responsible for consequential decisions.

The workflow

An alert starts the workflow with its identifier, name, description, timestamps, and measurements. LangGraph is a good fit because this is a stateful process with parallel work, conditional routing, and potentially long-running verification.

ALERT RECEIVED └─ Alert: id, name, description, timestamps, measurements CONSULT CONTEXT — parallel ├─ Monitoring platform, via MCP ├─ Approved test catalogue ├─ Change and deployment records ├─ Active incidents and maintenance windows └─ Service ownership and dependency information ASSESS ├─ Input: alert + context findings └─ Output: category, verification need, affected scope SELECT TESTS — only when verification is needed ├─ Input: alert + assessment + test catalogue └─ Output: smallest suitable approved test group CONSULT EVIDENCE — parallel ├─ CI/CD platform runs selected checks ├─ Test results and failure details ├─ Fresh monitoring signals └─ Optional logs, traces, and dependency checks JUDGE └─ Output: real problem | no problem | inconclusive COMPOSE RESULT └─ Evidence, verdict, impact, and recommended next steps

Start with facts, not conclusions

The context stage should return source data, not an early model opinion. It can collect current measurements, related alerts, recent changes, maintenance records, dependencies, ownership details, and available checks. Each finding should include its source and timestamp so the final report remains auditable.

MCP provides a useful boundary here. Each external system exposes a narrowly scoped capability, and the agent accesses it through a defined interface rather than arbitrary network access. The model gets relevant context without receiving broad operational permissions.

Use LangChain for constrained decisions

The assessment step classifies the alert using the alert and the context bundle. LangChain structured output turns this into a validated contract instead of free-form prose.

Assessment ├─ category: availability | latency | functional regression | data issue | unknown ├─ needs_verification: true | false ├─ affected_scope: regions, channels, tenants, or services ├─ confidence: low | medium | high └─ rationale: evidence-backed explanation

The prompt should require a clear distinction between observation and hypothesis. “The error rate increased during the last ten minutes” is evidence. “A dependency is failing” is a hypothesis until another source supports it.

Verify safely and narrowly

Not every alert needs a production verification. When the assessment requests one, the agent chooses only from an approved test catalogue. The model can select a test group and explain its choice, but it must not invent a command, a pipeline, a credential, or a new production action.

Minimal check

A small, low-risk availability or health check for a tightly scoped hypothesis.

Smoke check

A focused verification of a critical user journey when there is credible impact evidence.

Extended check

A broader suite reserved for high-confidence, potentially wide impact.

Policy guardrails

Allow-lists, rate limits, idempotency, concurrency controls, and least-privilege identities enforce the decision.

The governing rule is simple: choose the smallest approved check that can confirm or challenge the suspected impact.

Why LangGraph belongs in the middle

This flow is more than a chain. Context sources should run in parallel, verification is conditional, and test execution may outlive an individual request. LangGraph makes the stages and their state transitions explicit.

receive_alert │ ▼ consult_context ──► assess ──► verification needed? │ │ │ no │ yes ▼ ▼ judge select_tests ▲ │ └──── consult_evidence │ ▼ judge │ ▼ compose_report

Persisted graph state matters in practice. A context source can be unavailable, a check can take several minutes, or an escalation can pause for human approval. A durable workflow resumes from the state it already has rather than repeating calls or losing its evidence.

Judge with a bounded verdict

The final judgement should have only three outcomes:

“Inconclusive” is a safety feature. The agent should not hide uncertainty behind a reassuring conclusion, and it should not manufacture an incident because one source is temporarily unavailable.

Make the agent observable with LangSmith

The triage agent needs observability too. LangSmith can trace each workflow run using the alert identifier or fingerprint as a correlation key. Capture the sources consulted, the freshness of findings, the assessment, the selected checks, policy decisions, verification results, final verdict, and any notification outcome.

Before enabling automated notifications, evaluate the workflow against historical alerts. Measure classification accuracy, missed real incidents, unnecessary verification runs, evidence quality, and agreement with experienced responders. Start in shadow mode: produce reports without triggering external actions, then compare them with real outcomes.

The report people actually need

The output should be readable in under a minute and link back to the evidence:

Verdict: Real problem Confidence: High Suspected impact: Elevated failures in a critical user journey Evidence: - Monitoring shows an error-rate increase beginning at 12:04 UTC - A related dependency signal is active - A focused verification check failed consistently Recommended next steps: 1. Assign the responsible on-call team 2. Inspect the implicated dependency 3. Review recent changes and configuration updates

Make alerts earn attention

A production triage agent does not replace responders. It gives them the context and corroborated evidence needed to act before alert volume makes critical signals invisible.