AI for DevOps & SRE

DevOps and SRE are promising ground for AI — logs and metrics are text-heavy, incidents follow patterns, and investigation is tedious. They are also exactly where an agent's mistakes are most costly. This tutorial covers useful AI applications in ops, all governed by the read-only-first, human-approved discipline that production demands. The DevAgentic project is the full worked example.

The incident investigation agent#

When an alert fires, an agent can gather evidence far faster than a human paging through dashboards — provided it can only look, not touch:

private static final String SYSTEM = """
        You are an SRE investigating a production incident.
        Method: check recent deployments first (most incidents follow a change),
        then pod health, then logs, then metrics. Form a hypothesis only when
        the evidence supports it.
 
        You have READ-ONLY access. You cannot change anything, and must not imply
        that you have. Report a hypothesis and a recommended action for a human.
 
        Never claim to have checked something you did not check. If the evidence
        is inconclusive, say so.
        """;
 
// All tools are read-only by construction — the agent investigates, a human acts.
var agent = agentBuilder.defaultSystem(SYSTEM)
        .defaultTools(logTools, metricTools, deploymentTools, k8sReadTools)
        .build();

Log analysis#

LLMs add correlation and explanation on top of existing log tooling:

// Existing tools find the errors; the LLM correlates them and explains the
// story — "these 200 connection errors started 3 minutes after the deploy of
// service X, which changed the connection pool size" — in plain language.
LogAnalysis analysis = logAnalyst.correlate(errorCluster, deploymentContext);

The value is not replacing your log platform — it is synthesising across sources into a hypothesis a human can act on faster. Statistical alerting stays; the LLM adds the narrative. See AI for data engineering for the complement-not-replace principle.

Alert correlation#

Alert storms — dozens of alerts from one root cause — are noise that AI can compress:

// Group related alerts by likely common cause, so on-call sees "one incident:
// database saturation, causing these 15 downstream alerts" instead of 15 pages.
CorrelatedIncident incident = correlator.group(activeAlerts);

This turns a wall of pages into a single actionable incident with its probable root cause surfaced — directly reducing alert fatigue.

Runbook automation#

AI can help execute and generate runbooks, within the propose-approve boundary:

public Proposal proposeRunbook(Incident incident) {
    // The agent matches the incident to a runbook and drafts the steps.
    Runbook runbook = runbookMatcher.match(incident);
    RemediationPlan plan = agent.planExecution(runbook, incident);
 
    // It PROPOSES the plan. A human approves before any step runs.
    return approvalGate.propose(plan, agent.reasoning());
}

It can also generate runbooks from incident post-mortems — turning "here is what we did to fix it" into a reusable, documented procedure, which is genuinely useful and low-risk.

Guardrails, with extra force#

Everything from productionizing agentic systems applies, and ops raises the stakes on each:

• Read-only investigation, approved remediation — the core boundary.
• Scoped access — only the systems the task needs, never blanket production access.
• Hard budgets — a runaway agent during an incident is a second incident.
• Trajectory logging — for the post-mortem and for debugging the agent.
• A kill switch — disable the agent instantly if it misbehaves.

Starting safely#

Begin where risk is lowest and value is clear: the read-only investigation agent and alert correlation. Both help immediately and cannot damage anything. Add runbook generation (documentation, low-risk), then, much later and cautiously, runbook execution for a small set of reversible actions behind the approval gate. Autonomous production changes are the last thing you add, if ever. See human-in-the-loop systems.

Next#

• GraphRAG & knowledge graphs
• Enterprise AI agents
• DevAgentic project