Evaluating Agentic Optimization on Large Codebases

1. Problem Statement

Large Language Model (LLM) coding agents are evolving from isolated function synthesis to autonomous repository-level optimization. However, existing benchmarks (e.g., SWE-Bench, HumanEval) fail to adequately evaluate this emerging capability due to several limitations:

• Synthetic Nature: Many benchmarks rely on synthetic tasks or LLM-generated code, lacking the complexity of real-world software ecosystems.
• Binary Feedback: Most benchmarks use pass/fail unit tests, which are insufficient for measuring optimization (a continuous spectrum of performance improvement).
• Single-Objective Focus: Real-world optimization often involves trade-offs between multiple workloads (e.g., improving runtime for one function might degrade memory usage or slow down another). Existing benchmarks rarely assess an agent's ability to balance these multi-objective constraints.
• Lack of Granularity: There is a need for fine-grained metrics that capture continuous performance changes rather than discrete correctness.

The paper argues that to advance agentic coding, a new evaluation standard is required that operates at the repository scale, utilizes real-world performance workloads, and employs multi-objective metrics.

2. Methodology: The FORMULACODE Benchmark

The authors introduce FORMULACODE, a novel benchmark designed to evaluate the holistic optimization ability of agents on large, evolving codebases.

A. Dataset Construction

The benchmark is constructed through a four-stage automated pipeline:

1. Repository Scraping: Identified 766 mature Python repositories (e.g., Pandas, Scikit-learn, SciPy) with established performance benchmarking infrastructure (specifically using the Airspeed Velocity (ASV) framework).
2. Attribute Filtering: Scraped ~26,700 Pull Requests (PRs). Applied rule-based filters (merged status, file changes) and LLM-based intent classification to identify PRs primarily aimed at performance optimization.
3. Environment Synthesis: Used a reflexive LLM agent to automatically generate reproducible Docker build scripts for each candidate, ensuring the environment could be built and tested reliably.
4. Statistical Validation: Executed expert patches against baseline code. Used Mann-Whitney U tests ($p < 0.002$) and strict correctness checks (unit tests + snapshot tests) to retain only tasks with statistically significant, reproducible performance improvements.

Final Dataset: 957 tasks across 70 repositories. Each task includes:

• A baseline repository snapshot.
• An expert-authored patch (the "gold" solution).
• 264.6 community-maintained performance workloads per task on average (totaling ~250k workloads).

B. Evaluation Metrics

Unlike binary pass/fail, FORMULACODE uses continuous, multi-objective metrics:

• Speedup: The geometric mean of performance ratios ($Code_{agent} / Code_{0}$) across all workloads.
• Advantage ($Adv$): The difference between the agent's speedup and the human expert's speedup ($Adv = speedup_{agent} - speedup_{expert}$). This measures performance relative to the human baseline.
• Stratified Advantage: Aggregates advantage at different hierarchy levels (Module, Class, Function) to determine if agents excel at local vs. global optimizations.
• Normalized Advantage: Accounts for noise and variance in workload measurements.
• Cost-Weighted Metrics: Evaluates the improvement gained per unit of inference cost (token usage).

3. Key Contributions

1. First Multi-Workload Repository Benchmark: FORMULACODE is the first benchmark to evaluate agents on real-world codebases using hundreds of community-maintained performance workloads per task, capturing the complexity of multi-objective optimization.
2. Holistic Evaluation Framework: It moves beyond binary correctness to measure continuous performance gains, trade-offs, and cost-efficiency.
3. Live, Contamination-Resistant Dataset: The benchmark is continuously updated (monthly) with new tasks, preventing data leakage and ensuring agents are tested on evolving codebases.
4. Comprehensive Agent Analysis: Provides a detailed breakdown of agent capabilities across optimization strategies, repository popularity, and cost-performance trade-offs.

4. Experimental Results

The authors evaluated frontier models (GPT-5, Claude 4.0 Sonnet, Gemini 2.5 Pro, Qwen 3 Coder) using two agent frameworks (Terminus 2 and OpenHands).

Key Findings:

• Agents vs. Humans: While agents can improve runtime performance over the baseline, they consistently underperform human experts. All configurations showed negative "Advantage" scores, indicating a fundamental gap in optimization capability.
• Scope of Optimization: Agents are generally better at local/function-level optimizations but struggle with repository-level (systemic) refactors. They often miss holistic architectural improvements that yield broader gains.
• Strategy Specificity:
  • Agents excel at parallelization and batching strategies.
  • Agents struggle with vectorized operations (e.g., NumPy/C extensions) and delegating to lower-level system implementations.
• Repository Popularity:
  • Agents perform best on mid-popularity repositories (20th–60th percentile), likely due to better alignment with training data and available optimization headroom.
  • Performance drops significantly on tail repositories (low stars) and high-popularity repositories (where most easy optimizations are already exhausted).
• Cost Efficiency: Despite higher per-call costs, frontier models (e.g., Claude 4.0 Sonnet) are often more cost-effective than open-weight models because they require fewer tokens and shorter reasoning chains to achieve similar (or better) results.
• Multi-Workload Trade-offs: Agents struggle to balance competing workloads. While humans can negotiate trade-offs to achieve high global speedups, agents often cause significant regressions in specific workloads while trying to optimize others.
• Data Leakage: Minimal evidence of data leakage was found; performance did not significantly correlate with the temporal proximity of tasks to the model's training cutoff.

5. Significance and Impact

• Benchmarking Standard: FORMULACODE sets a new standard for evaluating agentic coding, shifting the focus from "can the agent fix a bug?" to "can the agent optimize a complex system?"
• Guidance for Future Agents: The results highlight that current agents lack the architectural reasoning required for holistic optimization. Future research must focus on multi-objective reasoning, better handling of system-level trade-offs, and improving vectorization capabilities.
• Practical Utility: The benchmark provides a rigorous, real-world testbed for developers and researchers to assess the viability of LLM agents in production software engineering, particularly for performance-critical scientific computing libraries.
• Open Source: The dataset and evaluation framework are open-sourced to foster community-driven improvement of coding agents.

In conclusion, FORMULACODE reveals that while frontier LLM agents are capable of making incremental performance improvements, they are not yet ready to replace human experts in optimizing large-scale, complex software systems, particularly when balancing multiple conflicting performance constraints.