Technical Summary: Opti-Agent-Bench

Problem Statement

Current Large Language Model (LLM)-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks fail to evaluate their capabilities in real-world scenarios. The primary limitation of current evaluation frameworks is their reliance on pre-structured mathematical formulations (e.g., "minimize total weighted tardiness subject to capacity constraints"). These benchmarks allow agents to succeed through pattern matching against known academic templates (such as TSP or Knapsack) rather than demonstrating genuine business understanding. Consequently, models that achieve high accuracy on synthetic or textbook problems often collapse when faced with industrial-scale tasks where the optimization structure must be inferred from messy, ambiguous business descriptions. Furthermore, conventional benchmarks typically rely on single-metric evaluations (e.g., objective function value or binary pass/fail), which mask critical failure modes such as constraint omission, model-code inconsistency, and report-implementation divergence.

Methodology

The authors introduce Opti-Agent-Bench, an end-to-end benchmark designed to evaluate optimization R&D agents across the complete pipeline: from understanding business-language descriptions to mathematical modeling, algorithm selection, code implementation, and solution report generation.

1. The ORAC Validity Framework

To ensure scientific rigor, the benchmark employs the Optimization R&D Agent Checklist (ORAC), a bi-level validity framework:

• Problem Formulation Validity: Ensures tasks are genuinely challenging and representative of real-world complexity. This is achieved through four design principles:
  • Business-Semantic Authenticity: Tasks use domain-specific terminology (e.g., "changeover downtime," "yield rate") rather than academic jargon.
  • Anti-Template Design: Each task includes structural "traps" (e.g., hidden constraints, non-standard objectives) that cause standard templates to produce infeasible or suboptimal solutions.
  • Multi-Constraint Coupling: Constraints interact non-trivially, requiring holistic system reasoning.
  • Anti-Pattern-Matching: Descriptions avoid keywords that directly reveal the problem type, forcing agents to infer structure from context.
• Solution Validity: Ensures scoring metrics accurately reflect solution quality. It distinguishes between superficial success and genuine capability, enforcing checks for:
  • Feasibility vs. Correctness: Verifying solutions against ground-truth Oracle constraints, not just the agent's stated constraints.
  • Objective Value vs. Model Fidelity: Detecting "coincidental correctness" where a flawed model yields a good objective value.
  • Runnable Code vs. Correct Implementation: Performing "semantic diffs" between mathematical formulations and code.
  • Coherent Report vs. Faithful Report: Cross-referencing reports against actual execution logs to detect hallucinations.

2. Modular Evaluation Architecture

The benchmark decomposes the optimization pipeline into four independently assessable modules, with cross-module consistency checks:

1. Problem Understanding: Identifying objectives, variables, and constraints from natural language.
2. Formal Modeling: Translating understanding into rigorous mathematical formulations.
3. Algorithm Design & Implementation: Selecting and coding solution algorithms.
4. Report Generation: Communicating results and insights.

Consistency Metrics: The framework introduces three cross-module metrics to detect misalignment:

• Understanding–Model Consistency (C12): Does the model faithfully reflect the understood problem?
• Model–Code Consistency (C23): Does the code faithfully implement the stated model?
• Code–Report Consistency (C34): Does the report accurately describe the implementation?

3. Benchmark Tasks and Experimental Setup

The benchmark comprises 12 industrial-scale tasks spanning diverse problem types, including integer programming, robust optimization, stochastic programming, Markov decision processes (MDP), second-order cone programming (SOCP), and non-convex optimization.

• Task Design: Each task includes anti-template traps, multi-scale instances (small, medium, large), and expert-verified reference models.
• Evaluation Agent: An automated evaluator (Claude Opus 4.6) acts as a senior Operations Research expert, applying structured rubrics with severity-weighted checkpoints (Critical, Major, Minor) to score agent outputs.
• Experimental System: The authors evaluate agents using a dedicated optimization agent built on the RD-Agent framework, which iteratively develops solutions through data exploration, code generation, and debugging loops.

Key Contributions

1. First End-to-End Benchmark: Opti-Agent-Bench is the first benchmark to evaluate optimization agents on the full pipeline from business-language requirements to solution reports, covering nine industrial-scale scenarios.
2. Modular Architecture with Consistency Checks: It introduces a modular evaluation system that independently assesses understanding, modeling, implementation, and reporting, while explicitly measuring cross-stage consistency to diagnose specific failure points.
3. ORAC Validity Framework: The development of a bi-level validity framework (ORAC) that ensures both task design rigor (via anti-template traps) and scoring integrity (via semantic verification).
4. Identification of Failure Modes: Through systematic experiments, the authors identify critical failure modes invisible to conventional metrics, including constraint omission under business language, model-code inconsistency, scalability collapse, and report-implementation divergence.

Results and Findings

The paper reports that current LLM-based agents exhibit significant fragility when moving from structured academic problems to real-world business scenarios.

• Template Matching Failure: Agents relying on pattern matching fail catastrophically on tasks with anti-template traps. For instance, in a custom roll-material scheduling task, agents anchoring to the "Cutting Stock Problem" template failed to recognize the problem as a dynamic online scheduling task with time-dependent constraints, leading to suboptimal transportation costs.
• Inconsistency Detection: The modular evaluation revealed that agents often produce mathematically correct models that are not faithfully implemented in code, or generate reports that describe idealized solutions rather than the actual executed code.
• Scalability Issues: Methods that succeed on small instances often fail at scale, highlighting a gap in robustness that single-metric evaluations miss.
• Performance Gaps: Even advanced models struggle with the "formulation gap"—the cognitive distance between business language and mathematical structure—resulting in near-zero performance on industrial-scale instances compared to high accuracy on textbook problems.

Significance

The paper argues that the field of optimization R&D has reached a point where the ability to translate complex, ambiguous business requirements into correct mathematical models is the primary bottleneck, not the ability to solve well-formulated problems. By shifting the evaluation paradigm from "solving a known problem" to "discovering and solving an unknown problem," Opti-Agent-Bench provides a necessary standard for assessing the true readiness of LLM agents for industrial deployment. The work underscores that genuine optimization capability requires deep domain understanding, creative modeling, and rigorous consistency across the entire development pipeline, capabilities that current benchmarks systematically overestimate.