Testing Strategies for Agents

Overview

In the previous two lessons, you explored why evaluation matters for non-deterministic systems and how benchmarks like SWE-Bench and GAIA measure agent capabilities at scale. But benchmarks answer a different question than testing. Benchmarks tell you how your agent compares to others on standardized tasks. Testing tells you whether your specific agent still works correctly after you changed something.

Traditional software has well-established testing practices: unit tests, integration tests, end-to-end tests. Agent systems need the same layers, but each one requires adaptation. You cannot assert that an LLM produces an exact string. You cannot replay a conversation and expect identical tool calls. You cannot run a full agent loop in milliseconds. These constraints demand new strategies.

This lesson introduces a testing pyramid for agents -- a layered approach where each level trades speed for realism. You will learn how to unit test tools with mocked LLM responses, integration test the agent loop with deterministic scenarios, run end-to-end evaluations against golden datasets, catch regressions with snapshot testing, and explore properties your agent should always satisfy. By the end, you will have a practical pytest test suite for an agent system.

The Agent Testing Pyramid

The classic testing pyramid applies to agents, but the layers map differently. At the base, fast and cheap unit tests verify that individual tools behave correctly in isolation. In the middle, integration tests exercise the agent loop -- the cycle of LLM reasoning, tool selection, and response assembly -- with deterministic (mocked) LLM responses. At the top, slower end-to-end tests run the full agent against real or near-real LLM calls and compare outputs to golden datasets.

graph TD
  E2E["End-to-End Tests<br/>Full agent + real/near-real LLM<br/>Golden datasets<br/>Slow — minutes per case"] --> INT["Integration Tests<br/>Agent loop + mocked LLM<br/>Deterministic scenarios<br/>Medium — seconds per case"]
  INT --> UNIT["Unit Tests<br/>Individual tools + assertions<br/>Mocked dependencies<br/>Fast — milliseconds per case"]

  SNAP["Snapshot Tests<br/>Regression detection<br/>Stored baselines"] -.-> INT
  SNAP -.-> E2E

  PROP["Property-Based Tests<br/>Invariant checking<br/>Random inputs"] -.-> UNIT
  PROP -.-> INT

  subgraph Pyramid["Agent Testing Pyramid"]
      E2E
      INT
      UNIT
  end

  subgraph CrossCutting["Cross-Cutting Strategies"]
      SNAP
      PROP
  end

  style Pyramid fill:#ebf8ff,stroke:#3182ce
  style CrossCutting fill:#fefcbf,stroke:#d69e2e
  style E2E fill:#fed7d7,stroke:#e53e3e
  style INT fill:#fefcbf,stroke:#d69e2e
  style UNIT fill:#c6f6d5,stroke:#38a169
  style SNAP fill:#e9d8fd,stroke:#805ad5
  style PROP fill:#e9d8fd,stroke:#805ad5

The pyramid shape is intentional: you should have many unit tests, some integration tests, and few end-to-end tests. Unit tests run in milliseconds and catch most bugs. End-to-end tests take minutes and cost real money (LLM API calls), so you run them less frequently -- typically in CI before a release, not on every code change.

Two cross-cutting strategies span multiple layers. Snapshot testing can apply at any level: store an approved output, and fail the test if the current output diverges. Property-based testing generates random inputs and checks that your agent satisfies invariants -- properties that should always hold regardless of input.

Test Execution Pipeline

Before diving into each layer, let's see how these tests fit into a development workflow. The following flowchart shows a typical test execution pipeline for an agent system.

flowchart LR
  DEV["Developer<br/>changes code"] --> UNIT_RUN["Run Unit Tests<br/>pytest -m unit<br/>~5 seconds"]
  UNIT_RUN -->|Pass| INT_RUN["Run Integration Tests<br/>pytest -m integration<br/>~30 seconds"]
  UNIT_RUN -->|Fail| FIX1["Fix and retry"]
  FIX1 --> UNIT_RUN

  INT_RUN -->|Pass| SNAP_RUN["Run Snapshot Tests<br/>pytest -m snapshot<br/>~10 seconds"]
  INT_RUN -->|Fail| FIX2["Fix and retry"]
  FIX2 --> INT_RUN

  SNAP_RUN -->|Pass| CI["Push to CI"]
  SNAP_RUN -->|Drift detected| REVIEW["Review snapshots<br/>Accept or reject"]
  REVIEW -->|Accept| UPDATE["Update baselines"]
  UPDATE --> CI
  REVIEW -->|Reject| FIX3["Fix regression"]
  FIX3 --> UNIT_RUN

  CI --> E2E_RUN["Run E2E Tests<br/>pytest -m e2e<br/>~5 minutes"]
  E2E_RUN -->|Pass| DEPLOY["Deploy"]
  E2E_RUN -->|Fail| INVESTIGATE["Investigate<br/>failure"]

  subgraph Local["Local Development"]
      DEV
      UNIT_RUN
      INT_RUN
      SNAP_RUN
      FIX1
      FIX2
      FIX3
      REVIEW
      UPDATE
  end

  subgraph CIPipeline["CI Pipeline"]
      CI
      E2E_RUN
      DEPLOY
      INVESTIGATE
  end

  style Local fill:#ebf8ff,stroke:#3182ce
  style CIPipeline fill:#f0fff4,stroke:#38a169
  style DEPLOY fill:#c6f6d5,stroke:#38a169
  style INVESTIGATE fill:#fed7d7,stroke:#e53e3e

Developers run unit, integration, and snapshot tests locally -- they are fast enough to run on every save. End-to-end tests run in CI because they require LLM API calls and take minutes. Snapshot drift triggers a human review step: you decide whether the change is intentional (update the baseline) or a regression (fix the code).

Layer 1: Unit Testing Tools

The base of the pyramid is unit testing individual tools. A tool is just a function: given inputs, it produces outputs. You test it the same way you test any function -- call it with known arguments, assert on the result.

The challenge with agent tools is that they often depend on external services (APIs, databases, file systems) and on the LLM itself. The solution is mocking: replace the real dependencies with controlled substitutes that return predetermined responses.

Notice the pattern: tools that have no external dependencies (like calculate) need no mocking at all. Tools that call external services get their dependencies replaced with MagicMock objects that return predetermined data. The tests are fast (milliseconds), deterministic (same result every run), and isolated (no network calls, no database).

Key insight: If you designed your tools using the middleware pattern from Module 6, Lesson 3, your tools are already decoupled from cross-cutting concerns. This makes them inherently easier to test -- you can test the tool logic without the middleware, and test the middleware without the tools.

Layer 2: Integration Testing the Agent Loop

Unit tests verify tools in isolation, but they do not tell you whether the agent uses tools correctly. Integration tests exercise the full agent loop -- the cycle where the LLM reasons about a task, selects a tool, processes the result, and decides what to do next.

The key technique is mocking the LLM itself. Instead of calling a real model, you replace the LLM with a function that returns predetermined responses. This makes the test deterministic: you know exactly what the agent will "think" and can assert on the tool calls it makes and the final answer it produces.

These tests verify behavior, not exact outputs. The assertions check that the agent called the right tools, handled errors gracefully, and terminated within bounds. This is the right level of specificity for integration tests -- precise enough to catch real bugs, loose enough to survive minor prompt changes.

Layer 3: End-to-End Evaluation with Golden Datasets

At the top of the pyramid, end-to-end tests run the full agent with a real (or near-real) LLM and compare outputs against a golden dataset -- a curated set of inputs and their expected outputs, reviewed and approved by humans.

Golden dataset tests are expensive -- each case calls a real LLM and may take seconds. Mark them with @pytest.mark.e2e so you can skip them during local development (pytest -m "not e2e") and run them only in CI. Build your dataset incrementally: every time you find a bug in production, add the failing case to the golden dataset so it never regresses.

Snapshot Testing for Regression Detection

Snapshot testing captures the agent's output for a given input and stores it as a baseline. On subsequent runs, the test compares the current output to the stored snapshot. If they differ, the test fails -- forcing you to either fix the regression or explicitly approve the new behavior.

This is especially valuable for agents because small changes to prompts, tool descriptions, or model versions can subtly alter behavior. Snapshot tests make these changes visible.

The normalize_output function is critical. Without normalization, trivial differences in whitespace or capitalization would cause false failures. The hash-based comparison catches meaningful content changes while ignoring formatting noise. When a snapshot test fails, you review the diff, decide whether the change is intentional, and either fix the code or run with update=True to accept the new baseline.

Property-Based Testing for Agents

Traditional tests check specific cases: "given this input, expect that output." Property-based testing takes a different approach: define properties (invariants) that should hold for all inputs, then let a testing framework generate hundreds of random inputs to find violations.

For agents, useful properties include:

• Termination -- the agent always finishes within max_steps, regardless of input
• Tool safety -- the agent never calls a tool with arguments that violate the tool's schema
• Consistency -- semantically equivalent inputs produce semantically equivalent outputs
• Monotonic progress -- each agent step moves closer to a solution (no infinite loops)

Property-based tests with the Hypothesis library are powerful for finding edge cases you would never think to write manually. The test_calculate_never_crashes test generates random arithmetic expressions and verifies the tool always returns structured data rather than raising an exception. The test_agent_always_terminates test sends random text as queries and verifies the agent always produces an answer -- catching potential infinite loops or crash-on-input bugs.

Connecting Middleware to Testing

In Module 6, Lesson 3, you learned how the middleware and hooks pattern creates an interception layer around every tool call. That pattern is not just useful in production -- it is a powerful testing enabler.

Middleware hooks give you natural injection points for test instrumentation:

• Tool call recording -- A test-only middleware can record every tool call the agent makes during a test, building the tool_tracker dictionary used in the examples above. In production, you remove this middleware. In tests, you add it.
• Deterministic stubbing -- Instead of mocking the LLM globally, a middleware can intercept specific tool calls and return canned responses, letting the rest of the pipeline run normally.
• Latency simulation -- A middleware can add artificial delays to tool calls during integration tests, verifying that your agent handles timeouts correctly.
• Fault injection -- A middleware can randomly fail tool calls with configurable probability, stress-testing the agent's error recovery.

This is the payoff of investing in middleware architecture early. Systems that use middleware are inherently more testable because the interception points already exist. You do not need to refactor your tools to accept mock dependencies -- you add a test middleware and the pipeline handles the rest.

Practical Testing Checklist

When building a test suite for your agent, work through these levels in order:

Unit tests (write first, run fastest):

• Every tool function has at least one happy-path and one error-path test
• External dependencies are mocked -- tests make zero network calls
• Edge cases: empty inputs, maximum-length inputs, special characters

Integration tests (write second, run on every commit):

• The agent loop completes within max_steps for representative queries
• The agent calls the expected tool(s) for each query type
• Error recovery works: agent handles tool failures without crashing
• Unknown tool calls are caught and reported gracefully

Snapshot tests (add incrementally):

• Capture baselines for your most important use cases
• Normalize outputs before comparison to avoid false failures
• Review snapshot diffs on every model upgrade or prompt change

End-to-end tests (write selectively, run in CI):

• Golden dataset covers your top 10-20 most critical scenarios
• Each production bug becomes a new golden dataset case
• Assertions check for key facts in the output, not exact string matches

Property-based tests (add for invariants):

• Agent always terminates (never hangs)
• Tools never crash on valid inputs
• Mathematical properties hold (commutativity, idempotency where expected)

Summary

Testing agents requires adapting traditional strategies to handle non-determinism, multi-step reasoning, and external dependencies. The agent testing pyramid provides a framework: many fast unit tests at the base, fewer integration tests in the middle with mocked LLMs for deterministic scenarios, and selective end-to-end tests at the top using golden datasets with real model calls.

Two cross-cutting strategies complement the pyramid. Snapshot testing catches regressions by comparing current outputs against approved baselines -- essential when prompt changes or model upgrades silently alter behavior. Property-based testing with tools like Hypothesis generates random inputs to verify invariants your agent should always satisfy, finding edge cases you would never write by hand.

The middleware pattern from Module 6 is a natural testing enabler. The same hooks that provide logging and validation in production can record tool calls, inject faults, and simulate latency in tests -- no refactoring needed.

In the next lesson, you will learn about LLM-as-Judge evaluation -- using one LLM to evaluate the outputs of another. This technique unlocks scalable evaluation for open-ended tasks where there is no single correct answer, such as summarization quality, instruction following, and conversational coherence.