CI/CD for Agent Systems

Overview

In the previous lesson, we containerized our agents and deployed them to cloud infrastructure. But deploying once is easy -- the hard part is deploying continuously without breaking things. Traditional web applications have mature CI/CD pipelines: push code, run tests, deploy if green. Agent systems break this model because their behavior depends on three moving parts -- code, prompts, and models -- and because their outputs are non-deterministic. A unit test that passes today might fail tomorrow with the same code if the model provider updates their weights.

CI/CD for agent systems extends traditional continuous integration and delivery with evaluation gates, prompt versioning, model pinning, and canary deployments. This lesson builds the pipeline that connects your evaluation harness from Module 11 to your deployment infrastructure from the previous lesson, creating an automated path from commit to production that catches regressions before users do.

Why Traditional CI/CD Falls Short

A standard CI/CD pipeline assumes deterministic behavior: the same input always produces the same output. Run the tests, check the exit code, deploy if zero. Agent systems violate this assumption in three ways:

• Non-deterministic outputs -- the same prompt can produce different responses across runs, even with temperature set to zero (model providers do not guarantee bit-exact reproducibility)
• Multi-artifact deployments -- a single agent change might involve updated code, a revised prompt, and a different model version, each needing independent versioning
• Behavioral regressions -- a prompt change that improves performance on one task category can silently degrade another, and unit tests will not catch this

This means agent pipelines need an additional layer between "tests pass" and "deploy": an evaluation gate that runs your eval harness against a representative benchmark and blocks deployment if scores drop below a threshold.

graph TD
  A["Developer Commits"] --> B["Lint & Static Analysis"]
  B --> C["Unit Tests"]
  C --> D["Integration Tests"]
  D --> E["Evaluation Suite"]
  E -->|Score >= Threshold| F["Canary Deploy<br/>5% traffic"]
  E -->|Score < Threshold| X["Block Deploy<br/>Alert Team"]
  F --> G{"Monitor Canary<br/>15-30 min"}
  G -->|Metrics Healthy| H["Full Rollout<br/>100% traffic"]
  G -->|Metrics Degraded| I["Auto-Rollback"]
  H --> J["Production Monitoring"]
  I --> J
  style A fill:#ebf8ff,stroke:#3182ce
  style B fill:#ebf8ff,stroke:#3182ce
  style C fill:#ebf8ff,stroke:#3182ce
  style D fill:#ebf8ff,stroke:#3182ce
  style E fill:#fefcbf,stroke:#d69e2e
  style F fill:#e6fffa,stroke:#38b2ac
  style G fill:#e6fffa,stroke:#38b2ac
  style H fill:#f0fff4,stroke:#68d391
  style I fill:#fff5f5,stroke:#fc8181
  style J fill:#f0fff4,stroke:#68d391
  style X fill:#fff5f5,stroke:#fc8181

Notice the two decision points that do not exist in traditional pipelines: the evaluation gate (yellow) that blocks bad changes before deployment, and the canary monitor (teal) that catches regressions that only appear under real traffic. Together, these form a two-layer safety net.

The Three Versioning Axes

Agent behavior is a function of three independent artifacts. Changing any one of them changes the agent's behavior. A robust CI/CD pipeline must version and track all three.

Prompt versioning treats prompts as first-class source code. Store them in your Git repository, review them in pull requests, and tag releases. When a prompt change causes a regression, you need to know exactly which version introduced it.

Model pinning locks the exact model identifier (e.g., claude-sonnet-4-20250514, not just claude-sonnet-4) in your deployment configuration. Model providers update weights periodically, and an unpinned model reference means your agent's behavior can change without any code or prompt changes on your side.

Tool versioning tracks the versions of external tools and APIs your agent depends on. If your agent calls a search API that changes its response format, your agent breaks -- even though nothing in your repository changed.

graph LR
  subgraph VersionControl["Version Control System"]
      P["Prompt Versions<br/>v1.0, v1.1, v1.2"]
      M["Model Pins<br/>claude-sonnet-4-20250514"]
      T["Tool Versions<br/>search-api: 2.3.1<br/>db-client: 1.8.0"]
  end
  subgraph Deployment["Deployment Manifest"]
      D["agent-config.yaml<br/>prompt: prompts/main@v1.2<br/>model: claude-sonnet-4-20250514<br/>tools: tools.lock"]
  end
  subgraph Registry["Artifact Registry"]
      R1["Prompt Registry<br/>Tagged, immutable snapshots"]
      R2["Container Registry<br/>Agent images with<br/>baked-in configs"]
  end
  P --> D
  M --> D
  T --> D
  D --> R1
  D --> R2
  style VersionControl fill:#ebf8ff,stroke:#3182ce
  style Deployment fill:#e6fffa,stroke:#38b2ac
  style Registry fill:#fefcbf,stroke:#d69e2e

The deployment manifest is the single source of truth. It pins every moving part to an exact version, making deployments reproducible and rollbacks trivial -- just redeploy the previous manifest.

Prompt Versioning in Practice

Prompts deserve the same rigor as application code. Store them in a dedicated directory, track them in Git, and enforce review before merging.

The content hash serves as an automatic version identifier. If someone changes a single character in the prompt, the hash changes, and your CI/CD pipeline can detect the drift. This is especially useful for catching unreviewed prompt edits in production environments.

Evaluation Gates: The Core Innovation

The evaluation gate is what makes agent CI/CD fundamentally different from traditional pipelines. Instead of a binary pass/fail from unit tests, the eval gate runs your agent against a benchmark dataset (the same harness you built in Module 11) and compares the scores to a threshold.

The key design decisions for eval gates are:

• Threshold selection -- set it based on your current production baseline, not an arbitrary number. If production scores 82% accuracy, set the gate at 78% (allowing some variance) rather than 95% (which would block everything).
• Statistical significance -- a single eval run is noisy. Run the benchmark multiple times or use a large enough dataset to get stable scores. A common practice is to require the new version to be "not significantly worse" rather than "better."
• Category-level checks -- an overall score can hide regressions. If your agent handles five task types, check each category independently. A change that improves math tasks but breaks coding tasks might have the same overall score.
• Timeout budgets -- eval suites can be slow and expensive. Run a fast "smoke" eval on every commit and a comprehensive eval on merge to main.

Connection to Module 11: The evaluation harness you built in Module 11 -- with its benchmark datasets, scoring functions, and comparison reports -- is exactly what plugs into this gate. The CI/CD pipeline invokes the same run_eval.py script, but instead of printing a report, it checks the scores against thresholds and returns an exit code.

The Complete CI/CD Pipeline

Here is a production-grade GitHub Actions workflow that implements the full pipeline: lint, test, evaluate, canary deploy, and promote.

There are several details worth highlighting in this workflow. The eval gate runs the benchmark three times (--runs 3) to reduce noise from non-deterministic outputs. The prompt hash is computed and attached to the deployment as an annotation, making it easy to trace which prompt version is running in production. And the canary step includes an automatic rollback on failure -- if the monitoring script detects elevated error rates or latency, the deployment reverts without human intervention.

Canary Deployments for Agents

Canary deployments route a small percentage of traffic to the new version while the rest continues hitting the stable version. For agents, this is especially valuable because evaluation benchmarks cannot catch every failure mode -- some issues only appear under real user traffic with real data.

The canary monitoring period should check three dimensions:

• Error rate -- are more requests failing or timing out compared to the stable version?
• Latency -- is the new version slower, possibly due to a longer prompt or a different model?
• Behavioral metrics -- are tool call patterns different? Is the agent making more API calls per request, or using tools it should not be using?

A typical canary progression looks like: 5% for 15 minutes, then 25% for 15 minutes, then 100%. At each stage, compare metrics against the stable version. If any metric degrades beyond the threshold, roll back immediately.

Rollback Strategies

When something goes wrong in production, you need to roll back fast. Agent systems have three rollback dimensions corresponding to the three versioning axes:

• Code rollback -- revert the container image to the previous tag. This is standard Kubernetes rollback.
• Prompt rollback -- if prompts are baked into the container, this happens with the code rollback. If prompts are loaded from a registry at startup, update the prompt pointer to the previous version hash.
• Model rollback -- if you updated the pinned model version and behavior degraded, revert the model ID in your configuration. This is why model pinning matters -- without it, you cannot "roll back" a model change you did not explicitly make.

The fastest rollback strategy is to keep the previous deployment manifest (the file that pins all three versions) and redeploy it. This is why the manifest should be an immutable, versioned artifact -- not a mutable configuration that gets edited in place.

A/B Testing Agents

Sometimes you do not want to replace the old agent -- you want to run both versions simultaneously and compare their performance on real traffic. A/B testing for agents works the same way as for traditional applications, with one critical addition: you need to measure behavioral quality, not just click-through rates.

An agent A/B test requires:

• Traffic splitting -- route a percentage of users to variant A and the rest to variant B, ensuring consistent routing per user session
• Evaluation metrics -- define what "better" means: task completion rate, user satisfaction scores, average number of tool calls, error rate, latency
• Statistical rigor -- run the test long enough to reach statistical significance before declaring a winner. Agent behavior is noisy, so you typically need more samples than a UI A/B test
• Safety bounds -- set automatic kill switches that end the test early if either variant shows unacceptable error rates

Practical tip: Start with canary deployments before attempting A/B tests. Canary is simpler (one decision: promote or rollback) and catches the most dangerous regressions. A/B testing is valuable for comparing two viable approaches, not for validating basic correctness.

Summary

CI/CD for agent systems extends traditional pipelines with three innovations that address the unique challenges of non-deterministic, multi-artifact systems:

• Three versioning axes -- prompts, models, and tools must all be independently versioned and pinned. The deployment manifest combines them into a single reproducible configuration.
• Evaluation gates connect your eval harness (Module 11) to your deployment pipeline, blocking changes that degrade benchmark scores below defined thresholds.
• Canary deployments catch regressions that benchmarks miss by routing a small percentage of real traffic to the new version and monitoring error rates, latency, and behavioral metrics.
• Rollback strategies must cover all three axes -- code, prompt, and model -- and the fastest path is redeploying a previous immutable deployment manifest.
• A/B testing enables comparing two viable agent versions on real traffic, but requires statistical rigor and safety bounds to be meaningful.

In the next lesson, we will explore reliability patterns -- retries, fallbacks, circuit breakers, and graceful degradation -- that keep your deployed agents running smoothly even when individual components fail.