Evaluation Lab
Hands-on: build an evaluation harness with automated scoring and regression detection
Overview
You have spent five lessons building a conceptual toolkit for agent evaluation. You understand why evaluation matters, how public benchmarks measure capability, what testing strategies apply at each level, how LLM-as-judge scoring works, and how tracing reveals what happens inside a running agent. Each of those lessons gave you a piece of the puzzle. This lesson assembles them into a single, working system.
The evaluation harness you will build in this lesson is the engine that ties everything together. It takes a set of test cases with expected outcomes, runs your agent against each one, scores the results using an LLM judge, aggregates those scores into a report, and compares them against a saved baseline to detect regressions. When you change your agent's prompts, swap models, update tools, or refactor orchestration logic, the harness tells you whether the change helped, hurt, or had no effect -- before that change reaches users.
This is a hands-on lesson. You will build the harness incrementally across four stages, with each stage producing working code that feeds into the next.
Architecture of the Evaluation Harness
Before writing any code, look at the full system you are building. The architecture has five components connected in a pipeline. Test cases flow in one end, and a regression report comes out the other.
graph TD
subgraph Pipeline["Evaluation Harness Pipeline"]
direction TB
DS["Evaluation Dataset<br/>──────────────────<br/>Test cases with inputs,<br/>golden answers, metadata,<br/>and difficulty labels"]
AR["Agent Runner<br/>──────────────────<br/>Executes the agent per test case,<br/>captures full trace: tool calls,<br/>reasoning steps, final output"]
JG["LLM Judge<br/>──────────────────<br/>Scores each output against<br/>its golden answer using<br/>a rubric (0.0 – 1.0)"]
SA["Score Aggregation<br/>──────────────────<br/>Computes per-category and<br/>overall metrics, generates<br/>the eval report"]
RD["Regression Detector<br/>──────────────────<br/>Compares current scores<br/>against saved baseline,<br/>flags degradations"]
DS --> AR
AR --> JG
JG --> SA
SA --> RD
end
RD --> PASS["✅ Pass — No Regressions"]
RD --> FAIL["❌ Fail — Regressions Detected"]
style Pipeline fill:#ebf8ff,stroke:#3182ce
style PASS fill:#c6f6d5,stroke:#38a169
style FAIL fill:#fed7d7,stroke:#e53e3e
Each component is deliberately simple. The power comes from their composition: the dataset defines what to test, the runner executes it, the judge scores it, the aggregator summarizes it, and the regression detector decides whether you should merge the change or investigate further.
Stage 1: Define the Evaluation Dataset
Every evaluation starts with data. An evaluation dataset is a collection of test cases, where each test case specifies an input to give the agent and the expected output you want back. Without golden answers, you have no ground truth. Without ground truth, you have no measurement.
A well-structured test case includes more than just input and output. It carries metadata -- a category label for grouping, a difficulty rating for stratified analysis, and a unique identifier for tracking results across runs.
Notice that the golden answers are not rigid templates. For factual questions, the golden answer states the expected fact. For reasoning questions, it captures the key points the response should cover. For tool-use questions, it describes the expected behavior sequence. The LLM judge you build in Stage 3 will compare agent output against these references using semantic understanding, not string matching.
Versioning your datasets is critical. When you add new test cases or refine golden answers, bump the version string. This lets you correlate evaluation results with the exact dataset that produced them. Without versioning, you cannot tell whether a score change came from a better agent or a harder dataset.
Stage 2: Build the Agent Runner
The agent runner is the component that executes your agent on each test case and captures the results. It needs to record not just the final output, but the full execution trace -- every tool call, every reasoning step, every intermediate decision. The trace is what lets you debug failures after the evaluation finishes.
sequenceDiagram participant R as Agent Runner participant A as Agent participant T as Tools participant S as Trace Store R->>A: Submit test case input activate A A->>A: Reasoning step 1 A->>T: Tool call (e.g., search) T-->>A: Tool result A->>A: Reasoning step 2 A->>T: Tool call (e.g., calculate) T-->>A: Tool result A->>A: Final reasoning A-->>R: Final output deactivate A R->>S: Store trace + output + timing
Two design decisions deserve explanation. First, the runner wraps every execution in a try/except block. Agents fail -- they hit rate limits, produce malformed tool calls, or enter infinite loops. A crashed evaluation is worse than a low score because it produces no data at all. Catching errors and recording them as failed traces lets the rest of the pipeline continue.
Second, the runner captures timing data. Latency is an evaluation dimension that purely accuracy-focused systems overlook. An agent that produces perfect answers in thirty seconds may be worse for your users than one that produces good answers in two seconds. Recording duration per test case lets you track performance regressions alongside quality regressions.
Stage 3: Implement LLM-as-Judge Scoring
You built the conceptual foundation for LLM-as-judge evaluation in Lesson 04. Now you implement it. The scoring function takes each evaluation result -- the agent's output paired with the golden answer -- and asks a judge LLM to rate it on a 0.0 to 1.0 scale using a structured rubric.
The rubric is the most important part of this code. A vague rubric produces noisy scores. A specific rubric produces consistent ones. Notice how each score level has concrete criteria: "fully addresses," "mostly correct with minor omissions," "partially correct with significant gaps." These criteria give the judge LLM clear decision boundaries.
Three defensive measures protect against judge failures. First, agent errors bypass the judge entirely and receive an automatic zero -- there is no point asking a judge to evaluate an empty output. Second, the score is clamped to the 0.0-1.0 range in case the judge returns an out-of-bounds value. Third, JSON parsing failures produce a zero with an explanatory message rather than crashing the pipeline.
Key insight from Lesson 04: Use a different model for the judge than for the agent under evaluation. If your agent runs on Claude Sonnet, judge with Claude Opus. This avoids the self-evaluation bias where a model rates its own outputs too favorably.
Stage 4: Add Regression Detection
Scores from a single run are useful, but the real value of an evaluation harness is regression detection -- the ability to compare the current run against a known baseline and flag any degradation. Without regression detection, you are flying blind every time you change your agent.
The regression detector loads a saved baseline (the scores from your last known-good run), compares it category-by-category against the current run, and flags any category where the score dropped by more than a configurable threshold.
graph LR
subgraph Current["Current Eval Run"]
C1["factual: 0.92"]
C2["reasoning: 0.78"]
C3["tool_use: 0.85"]
end
subgraph Baseline["Saved Baseline"]
B1["factual: 0.90"]
B2["reasoning: 0.85"]
B3["tool_use: 0.88"]
end
C1 --- CMP1["Δ +0.02 ✅"]
B1 --- CMP1
C2 --- CMP2["Δ -0.07 ❌"]
B2 --- CMP2
C3 --- CMP3["Δ -0.03 ⚠️"]
B3 --- CMP3
style CMP1 fill:#c6f6d5,stroke:#38a169
style CMP2 fill:#fed7d7,stroke:#e53e3e
style CMP3 fill:#fefcbf,stroke:#d69e2e
The two-threshold system is intentional. A drop of 0.02 earns a warning -- it might be noise or it might be a real degradation that needs watching. A drop of 0.05 earns a failure -- it almost certainly reflects a meaningful quality loss. You should calibrate these thresholds for your domain. An agent handling medical queries might use a fail threshold of 0.01. A casual chatbot might tolerate 0.10.
Putting It All Together
With all four stages built, you can assemble them into a single evaluation pipeline that runs end-to-end. This is the script you would invoke in CI/CD, triggered by every pull request that changes agent prompts, model configuration, or tool definitions.
The run_evaluation function supports two modes. In baseline mode (save_baseline=True), it runs the full pipeline and saves the resulting scores as the baseline for future comparisons. You do this once when your agent reaches a quality level you are satisfied with. In regression mode (baseline_path="baseline.json"), it runs the pipeline, compares against the saved baseline, and returns False if any category dropped below the failure threshold. In CI/CD, a False return triggers sys.exit(1), blocking the merge.
Domain-Specific Evaluation Challenges: Data Agents
Data-agent evaluation is one of the hardest evaluation domains for LLM agents. Unlike general question-answering, data agent evaluation requires execution-based testing: run the generated SQL query against a test database and compare the results to known-good answers. This introduces evaluation dimensions that general-purpose benchmarks do not cover:
Permission compliance testing — Did the agent only access permitted tables and schemas? A correct answer derived from unauthorized data is a failure, not a success. Your evaluation dataset needs to include test cases where the correct answer requires data the agent should not access, and verify that the agent refuses rather than proceeds.
Hallucinated-column detection — Did the agent reference columns that do not exist in the schema? SQL agents frequently hallucinate plausible-sounding column names (customer_name instead of cust_full_name) that fail at execution time. This is a structural correctness failure distinct from semantic correctness.
Metric-definition correctness — Did "revenue" mean gross revenue or net revenue after refunds? Did "active users" mean users who logged in this month or users with non-cancelled subscriptions? The same natural-language question can map to multiple correct SQL queries depending on business definitions. Your golden answers must encode which definition applies, and your judge must understand metric semantics, not just SQL syntax.
The 2026 Data Agent Benchmark (arXiv:2603.20576) found that the best tested frontier model reached only 38% pass@1 on complex enterprise data-agent tasks — a reminder that specialized domains expose capability gaps that general benchmarks miss.
Module 14, Lesson 2 covers data-agent evaluation in depth, including the semantic layer, evidence trail quality, and write-action autonomy tiers.
Integrating the Harness Into Your Workflow
Building the harness is the first step. Making it part of your development workflow is what makes it valuable. The following diagram shows where the evaluation harness fits in a typical agent development cycle.
graph TD
subgraph Dev["Development Cycle"]
direction TB
CHANGE["Make a Change<br/>Prompt / Model / Tools / Logic"]
LOCAL["Run Eval Locally<br/>Quick feedback loop"]
FIX["Investigate & Fix<br/>Review traces for failures"]
PUSH["Push to Branch"]
CI["CI Runs Full Eval<br/>Against saved baseline"]
REVIEW["Code Review<br/>Eval report attached to PR"]
MERGE["Merge & Update Baseline"]
end
CHANGE --> LOCAL
LOCAL -->|"Regressions"| FIX
FIX --> CHANGE
LOCAL -->|"No regressions"| PUSH
PUSH --> CI
CI -->|"Regressions"| FIX
CI -->|"Pass"| REVIEW
REVIEW --> MERGE
MERGE -->|"Save new baseline"| BASELINE["baseline.json<br/>Committed to repo"]
style Dev fill:#ebf8ff,stroke:#3182ce
style BASELINE fill:#fefcbf,stroke:#d69e2e
Three practices make this workflow effective:
Run locally before pushing. The full evaluation suite might take minutes, but a subset of critical test cases can run in seconds. Create a "smoke" dataset with your five most important test cases and run it after every prompt change. Save the full suite for CI.
Commit your baseline. The baseline.json file belongs in version control alongside your agent code. When you merge a change that improves scores, update the baseline in the same commit. This ensures every developer compares against the same known-good state.
Grow your dataset continuously. Every bug report is a potential test case. When a user reports that the agent gave a wrong answer, add that input and the correct answer to your evaluation dataset. Over time, your dataset becomes a comprehensive record of every failure mode your agent has encountered -- and a guarantee that each one stays fixed.
Summary
You built a complete evaluation harness across four stages. The evaluation dataset defines what to test and what correctness looks like. The agent runner executes each test case and captures traces for debugging. The LLM judge scores each output against its golden answer using a structured rubric. The regression detector compares current scores to a saved baseline and flags any degradation.
This harness turns agent development from guesswork into engineering. Instead of wondering whether a prompt change helped, you measure it. Instead of discovering regressions from user complaints, you catch them before merge. Instead of debating quality subjectively, you point to a number and a trend.
This lesson also concludes Module 11. You now have the full evaluation toolkit: the theory from Lessons 01 through 05, and the working code from this lab. Your agents are tested and evaluated -- Module 12 takes them to production with deployment, monitoring, and safety.