Technical Summary: An Empirical Study of Automating Agent Evaluation

Problem Statement

The deployment of autonomous AI agents in high-stakes domains (e.g., code generation, scientific discovery, financial analysis) is hindered by a critical bottleneck: the lack of automated, reliable evaluation tools. Unlike Large Language Model (LLM) evaluation, which often focuses on single-turn or multi-turn output quality, agent evaluation requires assessing complex, multi-step behaviors involving tool use, intermediate reasoning, error recovery, and state transitions over time.

Current evaluation practices are manual, expertise-intensive, and prone to failure. Practitioners must design custom criteria, instrument code to capture signals, and manually inspect execution traces. Common workarounds, such as embedding evaluation logic directly into agent code, create tight coupling between evaluation and implementation, leading to divergent behaviors between development and production. While "LLM-as-a-judge" and "agentic judge" approaches exist, they typically assume pre-defined criteria. The harder, unaddressed problem is evaluation generation: automatically generating the criteria, executable assessment code, and actionable reports given only an agent's source code and execution traces.

Initial studies in this paper reveal that simply prompting frontier coding assistants (e.g., Claude Code) to automate this process is insufficient. Without domain-specific guidance, these models exhibit three systematic failure modes:

1. Metric Proliferation: Generating 12+ scattered metrics per agent, often favoring operational data (latency, token counts) over meaningful task success.
2. Code Over-engineering: Producing evaluation code 2–3× longer than necessary, fragmented across multiple files with premature abstractions.
3. Plan-Code Drift: Generating well-reasoned semantic plans but implementing shallow keyword-counting heuristics instead.

Methodology: EvalAgent

To address these challenges, the authors introduce EvalAgent, an AI assistant designed to automate the end-to-end agent evaluation pipeline. EvalAgent transforms agent source code and user requirements into executable evaluation artifacts through a six-stage process:

1. Evaluation Planning: Analyzing source code and traces to produce a structured plan with focused metrics and test scenarios.
2. Test Case Generation: Creating JSONL-formatted test cases specifying inputs and expected behaviors.
3. Agent Instrumentation: Adding lightweight OpenTelemetry (OTEL) instrumentation to the target agent.
4. Trace Collection: Executing test cases and processing OTEL-compatible traces to extract relevant spans (operations, inputs/outputs, tool metadata).
5. Code Generation: Translating the plan into executable Python code, including metric implementations (deterministic and LLM-based), an orchestrator, and dependency manifests.
6. Reporting: Synthesizing results into structured reports with executive summaries, failure analysis, and recommendations.

Core Innovation: Evaluation Skills

The central mechanism enabling EvalAgent's success is the encoding of evaluation domain expertise as evaluation skills. These are self-contained packages comprising:

• Procedural Instructions: Stage-specific workflows and constraints (e.g., "create a minimal working version first," "validate library APIs before use").
• Reusable Code and Templates: Scaffolding for plans, reports, and executable patterns (e.g., OTEL trace parsing, DeepEval integration).
• Dynamic Resources: Real-time retrieval of up-to-date API documentation via Context7 to prevent errors caused by library API drift.

These skills are loaded on demand at each pipeline stage, ensuring that generation remains focused and implementable rather than drifting into scattered metrics or shallow heuristics.

Meta-Evaluation and Benchmarking

To assess the quality of generated evaluations without ground truth, the authors propose a Meta-Evaluation Framework and AgentEvalBench.

• AgentEvalBench: A benchmark comprising 20 diverse real-world agents across 9 frameworks and 14 application domains (e.g., travel, medical NLP, code generation). Each agent is paired with 2 types of user requirements (generic and specific) and 5 test scenarios.
• Meta-Evaluation Framework: An agentic meta-evaluator performs pairwise comparisons between evaluation approaches across five weighted dimensions:
  1. User Requirement Fulfillment (15%)
  2. Metric Relevance (30%)
  3. Code Quality & Complexity (25%)
  4. Plan Quality (15%)
  5. Plan-Code Alignment (15%)
     The framework explicitly enforces an anti-length bias, prioritizing concise, focused metrics over verbosity.
• Eval@1 Metric: A novel success rate metric measuring the fraction of agents for which generated evaluation code executes without errors and yields meaningful, non-vacuous results on the first run. This goes beyond standard code generation metrics (Pass@1) by requiring the evaluation to actually function.

Key Results

Experiments comparing EvalAgent against four baselines (ranging from single-turn LLM prompts to two-stage agentic approaches without skills) yield the following findings:

1. Superior Performance: EvalAgent achieves 84–100% win-tie rates across all baselines and evaluator models. It significantly outperforms the strongest baseline (two-stage planning without skills) with an 87.5–90.0% overall win-tie rate.
2. Executability (Eval@1): EvalAgent achieves an Eval@1 of 62.5–65.0%, substantially outperforming baselines (which range from 15% to 45%). Notably, removing evaluation skills causes Eval@1 to drop from 65% to 30%, demonstrating their critical role in handling complexity.
3. Human Preference: Expert annotators prefer EvalAgent's outputs in 79.5% of comparisons (with 10.5% ties), with near-perfect inter-annotator reliability (Fleiss' $\kappa = 0.923$).
4. Efficiency: EvalAgent produces evaluations that are 2.4–6.6× more concise in lines of code (LOC) than baselines and uses 31% fewer tokens and 58% less time than the two-stage baseline (B4) while achieving higher quality.
5. Robustness: EvalAgent maintains high performance under both generic ("evaluate this agent") and specific requirements. It is particularly effective under underspecified requirements, inferring meaningful metrics where baselines fail.
6. Ablation Insights:
   • Traces vs. Source Code: Trace-based evaluation improves metric relevance but reduces executability due to parsing complexity; however, it is essential for capturing behavioral failures invisible to static analysis.
   • Planning: Planning without structured skills (B4) often leads to scope creep and unused code, performing worse than direct implementation in some metrics.
   • API Documentation: Access to up-to-date API docs via Context7 is critical, improving Eval@1 by 45 percentage points for the Sonnet backbone.

Significance and Claims

The paper claims that general-purpose coding agents possess the necessary capabilities for evaluation but lack the specific domain knowledge regarding what to measure and how to implement it reliably. EvalAgent bridges this gap by encoding evaluation expertise as reusable skills.

The authors position this work as a necessary step toward closing the widening gap between agent capabilities and evaluation capabilities. They argue that:

1. Execution traces are essential for high-quality evaluation, offering substantial improvements over static code analysis.
2. Evaluation skills are superior to general-purpose generation, preventing metric proliferation and plan-code drift.
3. Dynamic documentation retrieval is non-optional for code generation systems targeting evolving evaluation frameworks.

By introducing AgentEvalBench and a validated meta-evaluation framework, the paper provides the research community with the first standardized benchmark and methodology for assessing automated agent evaluation systems, addressing a problem that will grow in importance as agents become more widely deployed.

Limitations

The authors modestly note several limitations:

• The benchmark covers 20 agents and may not represent all types (e.g., embodied or multimodal agents).
• Experiments rely on the Claude model family; cross-model validation is needed.
• Approximately one-third of generated evaluations still require manual debugging (Eval@1 ~65%).
• Meta-evaluation itself remains a subjective and challenging task, even for human experts.