LLM Agents Academy

Overview

In Lesson 03, you explored knowledge and support agents that help users find information and resolve issues. This lesson covers DevOps and software operations agents—where agent autonomy is most consequential because a wrong action can take down production systems.

Module 13 introduced computer use agents and coding agents for general automation. This lesson goes deeper into enterprise DevOps, where agents must balance operational efficiency with safety controls. You'll learn how incident response, code review, CI/CD, and runbook automation agents operate within safety tiers that match the blast radius of their actions.

DevOps agents operate in production environments where errors have real consequences: downtime, data loss, security breaches, and revenue impact.

Incident Response Agents

Incident response agents automate the initial triage and remediation steps when alerts fire. Their goal is to reduce mean time to resolution (MTTR) by handling routine incidents autonomously and escalating complex issues to human operators with full context.

The Incident Response Flow

When an alert fires, the incident response agent follows this sequence:

Alert Classification — Determine severity (P1 critical outage, P2 major degradation, P3 minor issue, P4 informational) based on service impact, affected users, and business hours.
Context Gathering — Collect logs from affected services, metrics (CPU, memory, request latency, error rates), recent deployment history, and known issues.
Runbook Matching — Match the alert type and context to known remediation procedures. For example, "high memory usage on service X" maps to "check for memory leaks, restart if OOM, scale replicas if sustained load."
Remediation Execution — Execute runbook steps within the agent's safety tier. Read-only actions (check logs, list resources) proceed automatically. Reversible actions (scale replicas, toggle feature flags) proceed with logging. Irreversible actions (delete resources, rollback deployments) require human approval.
Escalation — If remediation fails or the issue is unrecognized, escalate to the on-call engineer with all gathered context.

sequenceDiagram
  participant Monitor as Monitoring System
  participant Agent as Incident Response Agent
  participant Runbook as Runbook Database
  participant Infra as Infrastructure
  participant Human as On-Call Engineer

  Monitor->>Agent: Alert fired (high memory usage)
  Agent->>Agent: Classify severity (P2)
  Agent->>Infra: Get logs, metrics, deployment history
  Infra-->>Agent: Context data
  Agent->>Runbook: Match alert to remediation procedure
  Runbook-->>Agent: Runbook: "Check for leaks, restart if OOM, scale if sustained"
  Agent->>Infra: Execute read-only checks (query memory stats)
  Infra-->>Agent: Memory usage sustained for 20 minutes
  Agent->>Infra: Execute reversible action (scale replicas 3→5)
  Infra-->>Agent: Scaled successfully
  Agent->>Monitor: Monitor resolution
  alt Issue resolved
      Monitor-->>Agent: Memory usage normalized
      Agent->>Agent: Close incident, log resolution
  else Issue persists
      Agent->>Human: Escalate with full context
      Human->>Infra: Manual investigation
  end

  style Agent fill:#68d391,stroke:#2f855a,stroke-width:2px
  style Human fill:#fc8181,stroke:#c53030,stroke-width:2px
  style Infra fill:#90cdf4,stroke:#2c5282,stroke-width:2px

Incident response agents reduce MTTR by automating routine steps, gathering context before escalation, and executing safe remediation within defined safety tiers.

Alert Triage and Classification

Alert classification determines how the agent proceeds. High-severity alerts (P1, P2) trigger immediate action and human notification. Lower-severity alerts (P3, P4) may be queued for business hours or handled entirely by the agent.

Classification criteria include:

Service Impact — Is a customer-facing service down? Are internal tools affected?
Affected Users — How many users are impacted? Are they external customers or internal employees?
Business Hours — Is this during peak traffic or off-hours?
Known Issues — Does this match a recent deployment or a known bug?

Code Review Agents

Code review agents augment human reviewers by automating mechanical checks and surfacing issues that require domain expertise. They integrate with version control systems to analyze pull requests before they merge.

Beyond Simple Linters

While linters check syntax and style, enterprise code review agents understand:

Business Context — "This API endpoint handles payment processing, so it must include fraud detection checks per our security policy."
Internal API Conventions — "This service uses our internal RPC framework, which requires request IDs for tracing. This call is missing one."
Organization-Specific Security Policies — "This function logs user data, which violates our PII retention policy. Use the sanitized logging wrapper instead."

Code review agents also perform static analysis (null pointer risks, resource leaks), security scanning (dependency vulnerabilities, hardcoded secrets), and style enforcement (naming conventions, documentation requirements).

Integration with Human Review

Code review agents do not replace human reviewers. They handle mechanical checks so humans can focus on design decisions, algorithmic correctness, and maintainability. The typical flow:

Developer opens pull request.
Agent runs automated checks (tests, linters, security scans, policy validation).
Agent posts inline comments on specific lines that violate policies or conventions.
Developer addresses agent feedback.
Human reviewer evaluates design, logic, and context that agents cannot assess.

CI/CD Agents

Continuous Integration and Continuous Deployment (CI/CD) agents automate build, test, and deployment workflows. They triage failures, detect flaky tests, and manage deployment gates.

Build Failure Triage

When a build fails, the CI/CD agent:

Identifies the Failing Test — Parse build logs to locate the test name, failure message, and stack trace.
Analyzes the Failure Pattern — Is this a new failure or a known flaky test? Did it fail on one platform but pass on others?
Searches for Related Code — Locate the test file, the code under test, and recent changes that might have caused the failure.
Suggests Fixes — Propose a fix based on the failure type (assertion mismatch, timeout, dependency issue).

This reduces developer context-switching: instead of manually digging through logs, the developer receives a summary with suggested next steps.

Flaky Test Detection

Flaky tests pass or fail non-deterministically, often due to race conditions, timing dependencies, or environmental state. CI/CD agents detect flaky tests by tracking test results across multiple runs:

A test that fails 30% of the time is flaky.
A test that fails consistently after a specific commit is a real regression.

Once detected, flaky tests are flagged, quarantined (run separately to avoid blocking CI), or automatically retried.

Deployment Gate Management

Before deploying to production, CI/CD agents check deployment gates:

Test Coverage — Does the new code meet the required coverage threshold (e.g., 80% line coverage)?
Performance Benchmarks — Do response times stay within acceptable bounds?
Security Scans — Are there known vulnerabilities in dependencies?
Manual Approval — For high-risk changes, does a human need to approve?

The agent makes automated go/no-go decisions for routine deployments and escalates risky deployments to human operators.

Runbook Automation with Safety Tiers

Runbooks are documented procedures for handling operational tasks: restarting a service, scaling infrastructure, rolling back a deployment. DevOps agents automate runbook execution, but not all runbook actions are equally safe.

Three Safety Tiers

DevOps actions fall into three tiers based on reversibility and blast radius:

Read-Only — Querying logs, checking metrics, listing pods, inspecting configurations. These actions observe state without modifying it. Auto-execute.
Reversible — Scaling replicas (can scale back down), toggling feature flags (can toggle back), restarting services (service recovers). These actions modify state but can be undone. Execute with logging.
Irreversible — Deleting resources (cannot restore deleted data), database schema changes (migrations are hard to reverse), rollback deployments (can cause data inconsistency). These actions cannot be undone or have severe consequences. Require human approval.

This is the blast radius pattern from Module 12 (Production, Deployment and Safety): match autonomy to the reversibility of the action.

Safety tiers ensure that DevOps agents operate autonomously within safe boundaries and escalate risky actions to humans.

Code Example: CI/CD Triage Agent

This example demonstrates a CI/CD triage agent that analyzes build failures and suggests fixes. The agent uses four tools: get_build_logs, get_test_results, search_code, and suggest_fix.

cicd_triage_agent.py

This agent demonstrates:

Safety tier compliance — The agent can read logs and suggest fixes but cannot modify code or restart builds.
Multi-step reasoning — The agent retrieves logs, identifies failures, searches code, and proposes fixes across multiple tool calls.
Actionable output — The final response tells the developer what failed, where to look, and what to fix.

Enterprise Controls Used in This Vertical

DevOps agents operate in high-stakes environments where errors can cause outages, data loss, or security breaches. Enterprise controls ensure agents operate safely:

Blast-Radius Limits — Actions are categorized by safety tier (read-only, reversible, irreversible). High-risk actions require human approval.
Change Management Approval Gates — Deployments to production, schema changes, and infrastructure modifications go through approval workflows.
Incident Audit Trail — Every action the agent takes is logged with timestamp, user context, and outcome. Logs are immutable and retained for compliance.
Rollback Authorization — Only authorized users (e.g., on-call engineers, release managers) can approve rollbacks or irreversible operations.

These controls mirror the production safety patterns from Module 12: start with restrictive defaults, expand autonomy incrementally, and always maintain an audit trail.

Summary

DevOps and software operations agents automate incident response, code review, CI/CD workflows, and runbook execution. They reduce toil and improve reliability, but they operate in environments where mistakes have real consequences.

The key to safe DevOps automation is safety tiers: read-only actions proceed automatically, reversible actions execute with logging, and irreversible actions require human approval. This ensures agents stay within safe boundaries while maximizing operational efficiency.

In Lesson 05, you'll explore workflow and back-office agents—the engines behind document processing, approval chains, and structured business processes.

DevOps and Software Operations Agents