Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation

1. Problem Statement

The paper addresses two critical bottlenecks in evaluating Large Language Model (LLM) based Code Agents at the project level:

• High Annotation Costs and Limited Diversity: Creating high-quality, project-level benchmarks (e.g., SWE-Bench, PaperBench) traditionally requires extensive human expertise (PhD-level or domain experts) to write unit tests, verify logic, and create reference solutions. This is prohibitively expensive, time-consuming, and limits the diversity of evaluation domains.
• Inaccuracy of General LLM Judges: While "Agent-as-a-Judge" paradigms offer flexibility beyond rigid unit tests, relying on general-purpose LLMs (via In-Context Learning) for scoring often leads to hallucinations and misalignment with human standards. General models struggle to interpret complex execution logs, subtle file differences, and intricate project requirements, resulting in low human alignment rates (often <85%).

2. Methodology

The authors propose a comprehensive framework consisting of an automated data generation pipeline and a specialized evaluation model.

A. PRDBench: Agent-Driven Benchmark Construction

Instead of manual creation, the authors introduce a human-supervised agent-driven pipeline to generate benchmarks efficiently:

1. Initialization: Seed tasks are selected from real-world requests, academic projects, and thesis work.
2. Agent Generation: A state-of-the-art code agent generates:
   • A Product Requirement Document (PRD) detailing functional and data requirements.
   • A Code Scaffold (project structure and interfaces).
   • A Criteria Scheme (test plan) using the Arrange-Act-Assert (AAA) methodology.
3. Human Inspection: Human annotators (requiring only undergraduate-level CS knowledge) verify that the generated criteria align with the PRD and that the scaffold functions correctly. They do not write code or tests manually.
4. Iterative Refinement: Agents fix issues based on human feedback. Only tasks passing at least 5 rounds of refinement are retained.
5. Scaffold Removal: The final benchmark removes the scaffold, forcing evaluated agents to build the project from scratch based solely on the PRD.

Dataset Statistics:

• 50 Projects across 20 domains (e.g., AI, Web Security, Data Analysis).
• 1,258 Evaluation Metrics categorized into three types:
  • Unit Tests (408): Functional correctness via pytest.
  • Shell Interaction (732): Command-line and system operation verification.
  • File Comparison (118): Verification of generated artifacts and directory structures.

B. PRDJudge: Specialized Evaluation Agent

To overcome the instability of general LLM judges, the authors developed PRDJudge:

• Architecture: A lightweight agent built on Qwen3-Coder-30B equipped with six core tools (file I/O, shell execution, graph/image analysis, etc.).
• Training Strategy:
  • Data Construction: Trajectories were generated by a powerful teacher model (Qwen3-Coder-480B) scoring code from 11 different agents.
  • Strict Filtering: Only trajectories where the model's score matched human ground truth (Score 0, 1, or 2) were retained. A secondary rule-based filter removed trajectories with excessive tool spamming or invalid usage.
  • Fine-tuning: The model was fine-tuned using Low-Rank Adaptation (LoRA) on 911 high-quality evaluation trajectories.
• Evaluation Mechanism: PRDJudge executes the criteria scheme, analyzes logs/file differences, and outputs a structured JSON report. It follows a strict "Explore-then-Evaluate" workflow to ensure evidence-based scoring.

3. Key Contributions

1. Agent-Driven Data Pipeline: A novel workflow that reduces annotation complexity, allowing non-expert humans to supervise the creation of complex, diverse project-level benchmarks.
2. PRDBench: A new benchmark covering 20 domains with structured PRDs and multi-modal evaluation metrics (Unit, Shell, File), moving beyond simple unit-test pass rates.
3. PRDJudge: A specialized, fine-tuned evaluation agent that achieves >90% human alignment in fixed-interface scenarios, significantly outperforming general LLMs and zero-shot baselines.
4. Comprehensive Empirical Analysis: Extensive experiments evaluating 12 code agents (both minimal and commercial) on PRDBench, providing insights into their generation vs. debugging capabilities and cost-efficiency.

4. Experimental Results

• PRDJudge Performance:
  • Human Alignment Rate (HAR): Achieved 91.75% on in-domain and 92.69% on out-of-domain test sets, surpassing massive proprietary models like GPT-5.2 (89.76%) and Claude-4.5 (90.64%).
  • Stability: Demonstrated superior consistency with a Unanimous Agreement Rate (UAR) of 94.19%, compared to ~91% for general models.
  • Efficiency: Reduced inference time and token consumption by over 50% compared to its base model (Qwen3-30B) and massive baselines.
• Code Agent Evaluation Insights:
  • Foundation Models: Zero-shot development capability is heavily dictated by the underlying LLM's reasoning power (e.g., Claude-4.5 and GPT-5.2 lead in initial generation).
  • Commercial Frameworks: While commercial agents (e.g., Claude Code) often have lower initial generation scores due to rigid guardrails, they excel in iterative debugging, effectively utilizing feedback to fix issues.
  • Debugging Risks: Unconstrained minimal agents often suffer from "regressions" during long-context debugging, breaking previously functional modules.
  • Test Difficulty: Unit tests were found to be the most challenging to debug, as they require understanding internal test logic, whereas Shell/File tests rely more on output comparison.

5. Significance

This work establishes a scalable, robust, and highly accurate paradigm for evaluating code agents:

• Scalability: By decoupling benchmark creation from deep human expertise, it enables the rapid expansion of evaluation datasets across diverse domains.
• Reliability: PRDJudge solves the "black box" problem of LLM-as-a-Judge by providing a specialized, fine-tuned evaluator that aligns closely with human QA standards.
• Future Direction: It shifts the focus from isolated functional correctness (unit tests) to comprehensive PRD alignment, offering a more realistic assessment of an agent's ability to deliver complete software solutions. The authors identify "Free Development" (unconstrained interface mapping) as the next frontier for automated evaluation.