Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

1. Problem Statement

Current Large Language Model (LLM) agent benchmarks predominantly focus on binary pass/fail tasks (e.g., code generation passing a test suite, or search-based question answering with a definitive correct answer). These benchmarks fail to capture the essence of real-world engineering, which is rarely about finding a single "correct" solution from scratch. Instead, engineering is an iterative optimization process: starting with a feasible but suboptimal design, practitioners systematically refine it under hard constraints (safety, physics, cost) to extract marginal gains that compound into significant economic and practical value.

Existing benchmarks lack:

• Continuous reward signals: They often rely on binary outcomes rather than graded performance improvements.
• Hard feasibility constraints: They rarely enforce strict physical or safety limits that render a solution invalid if violated.
• Budget-aware search: They do not evaluate how effectively an agent finds the best possible solution within a fixed interaction budget (iteration limit).
• Domain diversity: They are often limited to specific domains like pure math or software, lacking the breadth of physical sciences, control systems, and hardware design.

2. Methodology: Frontier-Eng Benchmark

The authors introduce Frontier-Eng, a large-scale benchmark designed to evaluate Generative Optimization—the process where an agent iteratively proposes, executes, evaluates, and revises artifacts to improve performance.

A. Task Formulation

An engineering task is defined as a triple $\tau = (C, x_0, E)$:

• Context ($C$): Read-only problem specifications, constraints, and reference data.
• Initial Solution ($x_0$): A feasible but naive starting artifact (e.g., code, configuration).
• Evaluator ($E$): A fixed, read-only function that returns a feasibility indicator ($v \in \{0,1\}$) and a scalar score ($s \in \mathbb{R}$). The score is only meaningful if the solution is feasible.

The agent operates under a fixed budget $B$, aiming to maximize the best feasible score found: $s^* = \max \{s_t \mid v_t=1, 0 \le t \le B\}$.

B. Benchmark Composition

Frontier-Eng comprises 47 tasks across 5 engineering categories:

1. Computing & Quantum Information: GPU kernel optimization (CUDA/Triton), memory allocators, cryptographic throughput, and quantum circuit routing.
2. Operations Research & Decision Science: Inventory optimization, job-shop scheduling, and portfolio rebalancing.
3. Robotics, Control & Energy Systems: PID tuning, quadruped gait optimization, UAV path planning, and battery fast-charging profiles.
4. Optics & Communication Systems: Adaptive optics control, holographic pattern design, and fiber-optic network allocation.
5. Physical Sciences & Engineering Design: Structural truss optimization, chemical reaction optimization, and aerodynamic sensor placement.

C. Integrity and Anti-Hacking Safeguards

To prevent "reward hacking" (where agents exploit evaluator weaknesses rather than improving the solution), the benchmark enforces:

• Read-only Evaluators: Simulators, solvers (e.g., FEM, SPMe), and reference implementations are frozen and inaccessible to the agent.
• Sandboxed Execution: Candidates run in isolated environments.
• Verifier-Parsed Scoring: Scores are extracted from simulator logs, not self-reported by the agent.
• Feasibility Checks: Hard constraints (e.g., voltage limits, structural stress) must be met for a score to count.

D. Evaluation Protocol

Since tasks have incompatible units (e.g., latency vs. weight), the benchmark uses a Rank-Based Primary Metric:

• Average Rank: Models are ranked on each task; the average rank across all 47 tasks determines performance.
• Performance Profiles: Based on Dolan & Moré (2002), measuring the fraction of tasks where a model is within a factor $\alpha$ of the best score.
• Win Rate: The fraction of tasks where the model improves upon the initial baseline.

3. Key Contributions

1. Formalization of Generative Optimization: Defines a new evaluation scope for AI agents focused on iterative, budget-aware improvement of executable artifacts under hard constraints, distinct from one-shot generation.
2. Comprehensive Benchmark: Introduces 47 diverse, industrial-grade engineering tasks with unified interfaces, grounded in real simulators and solvers.
3. Empirical Analysis of Scaling and Dynamics: Evaluates 9 frontier models and 3 search frameworks, revealing systematic failure modes and scaling laws.

4. Experimental Results

A. Model Performance

• Top Performer: Claude 4.6 Opus achieved the best average rank (3.18), demonstrating robust performance across diverse domains.
• Other Models: GLM-5 (4.02) and DeepSeek-V3.2 (4.41) followed.
• Reliability: While Claude 4.6 Opus was the strongest, GLM-5 and GPT-OSS-120B showed high "Win Rates" (improving over baseline in >95% of tasks), indicating they are reliable improvers even if not always the absolute best.

B. Search Framework Dynamics

The study compared three search frameworks: ABMCTS (conservative), OpenEvolve (aggressive), and ShinkaEvolve (sustained refinement).

• Model-Framework Interaction: Different frameworks exposed different strengths. For example, OpenEvolve favored models capable of "bold" structural rewrites, while ShinkaEvolve helped models that required incremental scaffolding to avoid "capability collapse."
• Depth vs. Width: Under a fixed budget, depth (running a single chain of reasoning for many steps) significantly outperformed width (running many parallel chains with shallow depth). This suggests generative optimization requires sustained context accumulation for structural breakthroughs.

C. Optimization Dynamics (Power Laws)

Analysis of improvement trajectories revealed a dual power-law decay:

1. Frequency: Improvement events decay as $\propto t^{-1}$ (most improvements happen early).
2. Magnitude: The size of the $k$-th improvement decays as $\propto k^{-1}$ (early improvements are large structural changes; later ones are small refinements).

• Implication: Marginal returns diminish rapidly after ~50–100 iterations, suggesting that search strategies must balance early exploration with late-stage refinement.

D. Case Study Insights

• Structural Thresholds (e.g., MallocLab): Some tasks require a "regime shift" (e.g., switching from a bump allocator to a segregated list). Models that could perform "one-shot" architectural overhauls (like Claude + OpenEvolve) succeeded, while others failed without incremental scaffolding.
• Boundary Control (e.g., Battery Charging): Success required "boldness" to push against safety limits. Incremental methods often remained trapped in conservative regimes, while aggressive search found near-optimal boundaries.

5. Significance and Future Directions

• New Standard for AI Engineering: Frontier-Eng shifts the focus from "can the AI solve the problem?" to "how well can the AI optimize a solution?" This aligns AI evaluation with the actual workflow of human engineers.
• Path to AGI in Engineering: The results suggest that achieving Artificial General Intelligence in engineering requires not just larger models, but search strategies that effectively integrate domain knowledge with executable feedback.
• Practical Impact: The benchmark identifies that current agents struggle with multi-objective trade-offs and long-horizon reasoning in complex physical systems, highlighting specific areas for future research in agent design.

In conclusion, Frontier-Eng establishes that while current LLMs show emerging capabilities in iterative optimization, they still face significant challenges in navigating the complex, constrained, and continuous search spaces inherent to real-world engineering. The benchmark provides the necessary infrastructure to drive progress toward AI systems capable of delivering tangible value through systematic engineering optimization.