How to build an agent evaluation framework

Short answer. An agent evaluation framework has four parts: tasks that mirror real work, environments that start from a verified state, graders that read what actually changed, and statistics that account for non-determinism. Get any one of them wrong and the score stops meaning what you think it means.

What an agent evaluation framework is

An agent evaluation framework is the system that decides, repeatably, whether an agent did a job right. It is broader than an LLM eval harness, which scores text. Agents act, so the framework has to control the world they act in and read back what they changed.

The lessons below come from building environments for agents and from running our own Cross-App Agent Benchmark: 600 scored runs across six model configurations.

1. Tasks: realistic and verifiable

A good task mirrors real work and has an answer you can check in state. A few patterns make tasks far more informative:

• Record the exact instruction. Store the verbatim string the agent received, so results can be read against it later.
• Same words, different state. Give two tasks the same instruction but different starting state, so there are two different correct answers. It separates agents that read the world from agents that pattern-match the prompt.
• Decoys. Put a plausible wrong answer in the environment, such as a backup reminder time next to the real shift start.
• Terse instructions on purpose. Some real requests leave the hard part unsaid. Keep a few of those, and note it when reading their scores.

2. Environments: start from a verified state

Every run should start from a known, identical state. Seed the environment, then read the seed back to confirm it landed. If it can’t be confirmed, fail the run before the agent starts, so no model is charged for a world nobody set up.

After the run, restore the environment and verify the restore. An environment that can’t be returned to baseline can contaminate the next run on it; quarantine that run rather than overwrite it.

Keep environments isolated between runs and teams, and add failure conditions on top of the starting state: expired tokens, missing records, duplicates, outages.

3. Graders: read what changed

• Deterministic checks on state. Read the end state of every system and its change from the starting point.
• Checks on the call log. Every request and response in order, including refused calls.
• Forbidden actions. A Never rule broken is a violation, even when the task itself got done.
• Self-report, kept apart. Record what the agent says it did, but never grade from it.
• Model review, advisory only. Useful for open-ended quality, never allowed to change the score.

4. Outcome and validity: keep them separate

Every run should carry two fields. The outcome belongs to the agent: pass, fail, partial, or violation. The validity belongs to the instrument: valid, degraded (the verdict stands but a corroborating signal is missing), or invalid (the evidence is compromised).

Collapsing the two hides real results or counts compromised ones. A run can break a rule and leave evidence that couldn’t be restored; it should be scored as a violation and flagged invalid at the same time. Report both, with explicit denominators, such as a pass rate over all runs and a validity-filtered rate beside it.

5. Statistics: trials, pass rates and what not to pool

• Repeat every task. Several trials per task and model, from the same starting state.
• Report denominators. 49 of 50 means more than 98%.
• Read gaps of several runs. At 50 runs per model, a one-run difference carries no signal.
• Don’t pool different suites. In our benchmark, two suites scored 72.3% and 28.7%. The pooled 50.5% moved with the mix of tasks, not with any model’s ability.
• Expect rankings to move. One model fell 60 points between suites while another fell 20 and overtook it. A ranking that reorders when the task set changes is measuring model and task family together.

6. Reporting: keep every attempt

Keep retries, failures, violations and invalid runs in the evidence, not just the final score. Pin model versions and decoding settings, and record them per run. When a measurement is unavailable, report it as unavailable, never as zero.

Common mistakes

• Grading from the agent’s own summary
• Judging a version from a single run
• Starting runs from a state nobody verified
• Ignoring side effects when the task succeeded
• Averaging different task suites into one headline number
• Deleting failed and invalid runs from the record

FAQ

What is the difference between agent evaluation and LLM evaluation?

LLM evaluation scores a model’s text. Agent evaluation scores what an agent does to other systems across several steps, so it needs a controlled environment, state-based graders and repeated runs.

What metrics should an agent evaluation report?

A pass rate with its denominator per task and per model, a validity-filtered rate, the number of violations of forbidden actions, and how consistent results are across repeated runs.

How many tasks does an agent evaluation need?

Fewer, harder tasks beat many easy ones. Ten well-built tasks per suite, each run several times per model, already produce hundreds of scored runs.