Measuring Agent Effectiveness

Overview

As we discussed in the previous lesson, running an agent session reveals qualitative differences between prepared and unprepared repositories. But feelings and impressions are not enough to guide systematic improvement. This lesson introduces concrete metrics for measuring agent effectiveness -- giving you the tools to quantify how agent-friendly your codebase is and track improvements over time.

Why Measurement Matters

Without measurement, agent readiness becomes a matter of opinion. One developer thinks the codebase works great with agents; another finds it frustrating. Measurement resolves these disagreements with data.

More importantly, measurement tells you where to invest. If your agent's task completion rate is already 85%, spending a week improving documentation might yield marginal gains. But if your iteration count (how many tries the agent needs) is high, speeding up your test suite could have outsized impact.

What gets measured gets improved. Track your agent's performance the same way you track your test coverage or deployment frequency.

The Core Metrics

There are five metrics that together paint a complete picture of agent effectiveness in your codebase:

Five agent effectiveness metrics: Task Completion Rate (does the agent finish the job?), Iteration Count (how many tries?), Human Intervention Rate (how often must a human step in?), Convention Adherence (does output match team standards?), and Time to Completion (how long does it take?).

Metric 1: Task Completion Rate

Task completion rate is the most fundamental metric. It answers: what percentage of tasks does the agent complete successfully without human rework?

A "successful completion" means:

• The code works (tests pass, no runtime errors)
• The code matches the intent of the request
• The code does not introduce regressions

How to track it: After each agent session, record whether the task was completed, partially completed, or failed. Over time, calculate the ratio of completed tasks to total tasks attempted.

Benchmarks by maturity level:

Metric 2: Iteration Count

Iteration count measures how many attempts the agent needs to complete a task. An iteration is one cycle of the plan-execute-observe-adjust loop we discussed in the first lesson.

Low iteration counts indicate:

• Clear conventions that the agent follows correctly on the first attempt
• Fast feedback loops that quickly surface issues
• Well-structured code that the agent can navigate efficiently

High iteration counts indicate:

• Ambiguous conventions that lead to trial-and-error
• Slow test suites that delay the feedback loop
• Complex or undocumented architecture that confuses the agent

How to track it: Most agent tools show you the agent's actions in real time. Count the number of times the agent writes code, runs tests, and revises. A single successful write-test cycle is 1 iteration. A write-test-fail-fix-test cycle is 2 iterations.

Target: Well-prepared codebases see most tasks complete in 1-3 iterations.

Metric 3: Human Intervention Rate

Human intervention rate measures how often a human must step in during or after an agent session. Interventions include:

• Mid-task corrections -- "No, use the other database client"
• Post-task fixes -- manually adjusting code the agent produced
• Clarification requests -- the agent asks a question that a good instruction file would have answered
• Outright rejections -- discarding the agent's output and writing the code manually

How to track it: For each task, note whether you intervened and classify the type of intervention. Over time, categorize the interventions to identify patterns.

Four categories of human intervention mapped to their fixes: Convention Mismatch is fixed by documenting conventions in CLAUDE.md; Missing Context is fixed by adding an architecture overview; Wrong Approach is fixed by adding task-specific workflow guides; Scope Creep is fixed by adding constraints and boundaries.

The intervention categories directly map to agent-readiness improvements. Each category of intervention tells you exactly what to fix in your preparation.

Metric 4: Convention Adherence

Convention adherence measures whether the agent's output matches your team's coding standards. This is more nuanced than "does it work" -- code can be functionally correct but stylistically wrong.

Check for:

• Naming conventions -- does the agent use camelCase, snake_case, or PascalCase where your team expects it?
• File organization -- does the agent put new files in the right directories?
• Error handling patterns -- does the agent use your team's error handling approach?
• Test structure -- do the agent's tests follow your team's testing patterns?

How to track it: During code review, note convention violations separately from functional issues. Track the ratio of "convention-conforming" reviews to total reviews.

Metric 5: Time to Completion

Time to completion measures the wall-clock time from when you give the agent a task to when the task is done and verified. This encompasses:

• Time the agent spends reading and understanding the codebase
• Time spent writing code
• Time waiting for tests and builds to complete
• Time spent iterating on failures

Why it matters: An agent that completes a task in 2 minutes saves your team time. An agent that takes 20 minutes on a task you could have done in 5 is a net negative.

How to track it: Time your agent sessions. Most agent tools provide session duration in their output.

Building a Measurement Practice

You do not need a complex tracking system to start measuring. Here is a lightweight approach:

Step 1: Create a simple log. After each agent session, jot down:

• Task description
• Completion status (completed / partial / failed)
• Number of iterations
• Whether you intervened (and why)
• Time taken

Step 2: Establish a baseline. Run 10-15 tasks across different areas of your codebase before making any agent-readiness improvements. This baseline tells you where you are starting from.

Step 3: Make one improvement and remeasure. Add a CLAUDE.md, document your conventions, or speed up your tests. Then run another 10-15 tasks and compare against your baseline.

Step 4: Iterate. Identify the metric that improved the least and target your next improvement there.

The Effectiveness Dashboard

As you collect data, visualize it to spot trends and communicate progress to your team. A simple dashboard tracks the five metrics over time:

An effectiveness dashboard comparing two periods. Current period: 68% completion, 2.4 avg iterations, 35% intervention, 78% convention adherence, 4.2 min avg time. Previous period: 42% completion, 4.1 avg iterations, 62% intervention, 51% convention adherence, 8.7 min avg time — showing improvement across all metrics.

This example shows a team that added a CLAUDE.md and documented their conventions between the two measurement periods. Every metric improved -- completion rate up 26 points, iterations nearly halved, intervention rate cut by nearly half, convention adherence up 27 points, and average time cut in half.

What Good Looks Like

Here are practical targets to aim for as you improve your codebase's agent readiness:

• Task completion rate above 70% -- the agent successfully handles the majority of routine tasks
• Average iteration count under 3 -- the agent gets it right quickly, without excessive trial-and-error
• Human intervention rate below 30% -- you step in for complex judgment calls, not for routine guidance
• Convention adherence above 85% -- the agent's code passes your style review without changes
• Time to completion consistently under 5 minutes for standard tasks

These are targets for routine development tasks (adding features, fixing bugs, writing tests). Complex architectural decisions and novel feature design will always require more human involvement.

Connecting Metrics to the Maturity Model

As we covered in the maturity model lesson, each maturity level corresponds to a range of agent effectiveness. Use your metrics to validate your self-assessment:

• If your completion rate is below 30%, you are likely at Level 0 or early Level 1
• If your completion rate is 50-70% but convention adherence is low, you are at Level 1 heading toward Level 2
• If both completion and convention adherence are above 70%, you are solidly at Level 2 or Level 3
• If all five metrics are consistently strong, you are approaching Level 4

The metrics do not just tell you where you are -- they tell you what to improve next. Low completion rate? Focus on feedback loops and test reliability. High completion but low convention adherence? Invest in documenting your standards.

Summary

Measuring agent effectiveness transforms agent readiness from a subjective impression into a data-driven practice. The five core metrics -- task completion rate, iteration count, human intervention rate, convention adherence, and time to completion -- give you a complete picture of how well your codebase supports AI agents. By establishing a baseline, making targeted improvements, and remeasuring, you create a virtuous cycle that continuously improves both agent performance and overall codebase quality. With this foundation, you are ready to move into Module 2, where we will build the most important artifact for agent readiness: the project instruction file.