VeRO: An Evaluation Harness for Agents to Optimize Agents

1. Problem Statement

The paper addresses the emerging challenge of Agent Optimization: the iterative improvement of a target LLM agent through code modifications (edit–execute–evaluate cycles). While significant progress has been made in optimizing LLM weights (via RL) or prompts (via frameworks like DSPy), the automated optimization of the broader agent program structure (control flow, tool logic, orchestration) remains underexplored.

Current challenges in this domain include:

• Lack of Standardization: There is no standardized benchmark framing agent optimization as a code generation task.
• Unreliable Evaluation: Without strict infrastructure, coding agents often fail to complete optimization runs, produce unreproducible results, or exceed computational budgets.
• Stochastic Complexity: Target agents interleave deterministic code with stochastic LLM completions, requiring structured capture of both intermediate reasoning and execution outcomes to evaluate performance accurately.

2. Methodology: The VeRO Framework

The authors introduce VeRO (Versioning, Rewards, and Observations), a comprehensive evaluation harness designed to make agent optimization reproducible, fair, and interpretable.

A. Formal Problem Definition

The task is defined as finding an optimal agent implementation $A^*$ that maximizes expected performance on a held-out test set, subject to a strict evaluation budget ($B$).

• Target Agent ($A$): A Python program invoking LLMs via tool-calling APIs.
• Optimizer ($S$): A coding agent that iteratively modifies $A$.
• Constraints: The search space is restricted (e.g., no changing model checkpoints, no access to test data) to ensure fair comparison and prevent trivial solutions.

B. Core Abstractions (The VeRO Harness)

VeRO provides six key requirements through five core abstractions:

1. Versioning (Git Worktree): All modifications are captured as discrete Git commits. This enables rollback, diff inspection, and trajectory analysis.
2. Budget Enforcement: An ExperimentRunner tool gates evaluation requests, ensuring the total number of evaluations ($n_E$) does not exceed the budget $B$.
3. Permission Control: Filesystem and dataset abstractions enforce strict access controls (e.g., preventing access to ground-truth test data or modifying evaluation infrastructure).
4. Reproducible Execution: Agents are packaged as uv-managed Python projects with lockfiles, ensuring consistent dependencies and environments across runs.
5. Structured Tracing: An ExperimentDatabase stores detailed traces (inputs, outputs, scores, intermediate steps) to provide directional feedback to the optimizer.
6. Standardized Observation Interface: A consistent API (ExperimentViewer, DatasetViewer) exposes history and traces to all optimizers, ensuring no privileged access.

C. Optimization Loop

The framework does not prescribe a search strategy. The optimizer ($S$) operates in a loop:

1. Inspect: Read code, view dataset samples, and analyze past evaluation traces.
2. Hypothesize & Implement: Propose code edits (prompts, tools, logic).
3. Evaluate: Run the modified agent on the training/validation set via the gated ExperimentRunner.
4. Iterate: If performance improves, keep the commit; otherwise, optionally rollback.

3. Key Contributions

1. The VeRO Harness: A reproducible infrastructure that enforces versioning, budget control, and structured feedback, solving the reliability issues of previous ad-hoc optimization attempts.
2. The Agent Optimization Benchmark: A standardized suite of 5 tasks (Math, Tool Use, Multi-step QA, Factual QA, Science QA) with reference target agents and evaluation procedures.
3. Empirical Findings: A large-scale study comparing optimizer configurations, revealing critical insights into what drives successful agent optimization.

4. Experimental Results

The authors conducted three studies using VeRO:

A. Benchmark Study (Optimizer Configurations)

• Infrastructure Matters: Optimizers using the full VeRO harness significantly outperformed those without it (e.g., "Claude Code Pure" vs. "VeRO Default"). VeRO enabled an 8% average lift in performance, whereas unstructured approaches showed minimal gains.
• Task Dependency: Optimizers showed consistent gains on tool-use tasks (GAIA, TAU-Bench Retail, SimpleQA) but struggled with reasoning-heavy tasks (GPQA, MATH), suggesting current agents lack the ability to structurally improve deep reasoning capabilities.
• Scaffold Bias: The "Default" scaffold (full tools + sub-agent delegation) outperformed "Orchestrator" (delegation only) and "Resources Only" (prompt/param changes only), indicating that structural changes (tools/workflow) are more impactful than prompt tuning alone.

B. Robustness Study (Generalization)

• Model Family Transfer: Performance gains generally persisted when the target agent model was swapped within the same model family (e.g., GPT-4.1 to GPT-4.1 variants).
• Cross-Model Failure: Gains often failed to transfer to different model families (e.g., GPT to Gemini or Qwen), and in some cases, led to performance regressions. This highlights that optimizations are often overfit to specific model behaviors.

C. Case Study (Complexity & Instructions)

Using two target agents—Pawn (minimal) and Knight (sophisticated)—the study analyzed optimization dynamics:

• Optimization Headroom: Simpler agents (Pawn) admitted larger performance lifts (+11.5% on GAIA) compared to sophisticated agents (Knight, +6.9%), suggesting simpler agents have more "low-hanging fruit" for structural improvements.
• Instruction Sensitivity:
  • Pawn performed best with Cookbook+Reasoning (detailed guidance), as it needed structural direction.
  • Knight performed best with Minimal instructions (removing cookbook access), suggesting that prescriptive guidance can constrain already capable agents.
• Variance vs. Performance: High-variance instruction templates yielded higher peak performance but lower stability. Low-variance templates (e.g., "Evidence-Based") offered stability but capped potential gains.
• Generalization Gap: Optimizations targeting one task (e.g., GAIA) often degraded performance on related tasks (e.g., SimpleQA) due to overfitting specific verification tools.

5. Significance and Future Directions

• Paradigm Shift: The paper establishes that agent optimization is a viable but distinct capability from prompt engineering, requiring code-level structural changes.
• Current Limitations: The study reveals that current optimizers default to prompt modifications (>50% of changes) and lack diversity in structural exploration. They also struggle with cross-task generalization.
• Community Impact: By releasing VeRO, the authors provide the necessary infrastructure for the community to benchmark progress, moving beyond "black-box" optimization to transparent, traceable agent evolution.
• Future Work: The authors suggest exploring tree-search methods for program space exploration, principled construction of optimizer context (cookbooks), and integrating reinforcement learning to train optimizers on these structured traces.

In conclusion, VeRO demonstrates that while automated agent optimization is possible and yields significant gains on tool-use tasks, it requires rigorous infrastructure to be reproducible. The field is currently limited by a lack of diversity in modification strategies and poor generalization across model families and tasks.