🤖 AI

OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents

The paper introduces OR-Space, a full-lifecycle workspace benchmark designed to evaluate industrial optimization agents on realistic, multi-stage tasks involving model construction, revision, and grounded explanation within persistent, multi-artifact environments, moving beyond traditional one-shot problem formulation assessments.

Original authors: Chenyu Zhou, Xinyun Lu, Jiangyue Zhao, Jianghao Lin, Dongdong Ge, Yinyu Ye

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenyu Zhou, Xinyun Lu, Jiangyue Zhao, Jianghao Lin, Dongdong Ge, Yinyu Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant new assistant to help run a factory. You want to see if they can actually do the job, not just talk about it.

For a long time, the way we tested these AI assistants (called "LLM agents") was like giving them a perfectly written recipe card. The card said: "Here is the problem, here are the ingredients, here is the math formula. Now, write the code to solve it."

If the AI wrote the code correctly, we gave it a gold star. But in the real world, factory managers don't get perfect recipe cards. They get a messy desk with sticky notes, spreadsheets, old code from last year, and emails from the boss saying, "Oh, by the way, we need to change the rules for next month."

OR-Space is a new, much harder test designed to see if AI can handle that messy, real-world desk.

The Three Ways We Test the AI

The paper introduces a benchmark called OR-Space (Operations Research Space). Instead of a single question, it gives the AI a whole "workspace" (a digital folder with files) and asks it to do three different types of jobs that happen in a real factory:

Build (The Architect):
- The Scenario: You hand the AI a folder with a business memo, a spreadsheet of numbers, and an empty code file.
- The Task: The AI has to read the memo, figure out what the numbers mean, and write a brand-new computer program to solve the factory's problem from scratch.
- The Catch: The memo might be vague, and the spreadsheet might have messy columns. The AI has to connect the dots between the text and the data without a perfect guide.
Revise (The Fixer):
- The Scenario: The factory just changed its mind. Maybe they have more trucks now, or a new rule about safety. The AI is given the old program and the new rules.
- The Task: The AI must update the old code to fit the new rules without breaking the parts that were already working.
- The Catch: This is like trying to renovate a house while people are still living in it. If the AI copies a variable name from the old code that doesn't make sense anymore, the whole thing crashes. The paper found that some AIs get confused by the old code and try to copy its mistakes, while smarter AIs know what to keep and what to throw away.
Explain (The Translator):
- The Scenario: The computer has solved the problem, but the factory manager (who isn't a math expert) asks, "Why did you decide to send 5 trucks to the north warehouse instead of the south?"
- The Task: The AI has to look at the solution, the code, and the data logs to give a truthful answer.
- The Catch: The AI can't just make up a story. It has to point to the specific line in the spreadsheet or the specific rule in the code that forced that decision. If it guesses, it loses points.

Why This Test is Different (The "Workspace" vs. The "Prompt")

The authors argue that previous tests were like flashcards. You show the AI a clean, isolated problem, and it answers.

OR-Space is like a real office.

The Files are Separate: The requirements are in a Word doc, the numbers are in a CSV file, and the code is in a Python file. The AI has to open them all, read them, and realize how they fit together.
The "Grounding" Problem: In a flashcard test, the AI might guess the right answer because it recognizes the words. In OR-Space, if the AI doesn't correctly link the word "capacity" in the memo to the column "max_load" in the spreadsheet, it fails. The paper calls this "grounding"—tying the words to the actual data.

What They Found (The Results)

The researchers tested 20 different AI models (the "smartest" ones available) on this new test. Here is what happened:

It's Harder Than It Looks: Even the best AIs struggled. While some could solve the "flashcard" version of the problem, they often failed when they had to navigate the messy folder of files.
Old Code is a Double-Edged Sword: When the AI was given old code to help it revise the model, the smartest AIs got better because they used the old code as a helpful hint. But the weaker AIs got worse because they blindly copied the old code's mistakes (like trying to use a variable that no longer existed).
The "Explain" Gap: Some AIs were great at writing the code but terrible at explaining it. They could solve the math but couldn't tell you why they made that choice based on the files they had.
The "File System" Penalty: The paper found that when AIs had to actually navigate folders and read files (the "Filesystem" mode), they performed worse than when the same information was just pasted into one giant block of text. This proves that finding and organizing information is a skill in itself, not just a formatting issue.

The Bottom Line

The paper concludes that we can't just test AI on how well it writes code from a perfect prompt. To be useful in real industries (like logistics, manufacturing, or finance), AI needs to be tested on its ability to:

Find the right information in a messy pile of documents.
Update old systems without breaking them.
Explain its decisions using evidence from the actual files, not just made-up stories.

OR-Space is the new "driving test" for these AI agents, moving them from the parking lot (perfect prompts) to the busy city streets (real workspaces).

Technical Summary: OR-Space

Problem Statement

Existing benchmarks for Large Language Model (LLM) agents in Operations Research (OR) largely abstract away the complexities of real-world industrial workflows. Current evaluations typically reduce OR tasks to "one-shot" problems where an agent translates a self-contained textual prompt into a mathematical formulation or solver code. This approach fails to capture two defining characteristics of industrial OR:

The Workspace Setting: Real-world agents operate within persistent, multi-artifact environments containing business requirements, structured data (CSV/JSON), code artifacts, solver feedback, and stakeholder interactions distributed across multiple files.
The Lifecycle Setting: Industrial OR is iterative. Engineers do not merely generate a solution once; they build initial models, revise them in response to changing business rules or solver feedback, and explain the resulting decisions to non-expert stakeholders.

Consequently, current benchmarks cannot determine if an agent can perform reliable optimization work across a multi-artifact workspace and through multiple stages of the modeling lifecycle.

Methodology

The authors introduce OR-Space, a full-lifecycle workspace benchmark designed to evaluate industrial optimization agents. The methodology is structured around two core axes: a persistent executable workspace and a lifecycle-oriented task pipeline.

1. OR Workspace Formalization

An OR workspace ( $W$ ) is defined as a structured, persistent, executable environment:
$W = \langle D, P, S, E, M \rangle$

$D$ (Documents): Natural language artifacts (e.g., business_requirement.md) describing goals and requirements.
$P$ (Parameters): Structured data files (CSV, JSON) containing numerical inputs, often requiring cleaning and schema alignment.
$S$ (Code): Code artifacts, including heuristic scripts, partial templates, or legacy implementations.
$E$ (Environment): A Docker-based sandbox supporting file I/O, solver execution (Gurobi, COPT, PuLP, HiGHS), and resource constraints.
$M$ (Metric): The evaluation metric (hidden from the agent) used by the benchmark harness.

Unlike prior benchmarks that serialize all context into a single prompt, OR-Space requires agents to locate, connect, and ground information across these distributed artifacts.

2. Lifecycle-Oriented Task Modes

OR-Space defines three sequential task modes derived from 100 base problems (extended from the IndustryOR benchmark):

Build ( $T_{Build}$ ): The agent constructs a solver-ready optimization model from scratch using business documents and data files, starting from an empty code scaffold.
Revise ( $T_{Revise}$ ): The agent modifies an existing optimization workflow under updated requirements. This includes variants like Revise-code (modifying legacy code), Revise-model (modifying formulation only), and Revise-all. The agent must preserve valid prior logic while integrating new constraints or variables.
Explain ( $T_{Explain}$ ): The agent answers grounded questions about solutions, constraints, and business implications using evidence distributed across documents, data, code, and solver outputs (e.g., dual variables, slack values).

3. Evaluation Protocol

Build & Revise: Evaluated via solver-scored metrics. A submission is correct if it executes without error, returns an "Optimal" status, and achieves a relative objective error $\le 1\%$ against a ground-truth oracle.
Explain: Evaluated using a grounded "LLM-as-judge" rubric. Scores are based on Exact Coverage (35%), Reasoning (35%), Grounding (20%), Answer Quality (10%), and a Hallucination Penalty (up to -12).
Failure Analysis: The framework logs specific failure modes, including execution errors, infeasible formulations, incorrect objectives, and unfaithful explanations.

Key Contributions

Identification of Limitations: The paper identifies that existing OR benchmarks suffer from self-contained workspace settings and single-shot task settings, which underrepresent the multi-artifact and lifecycle-oriented nature of real industrial workflows.
OR-Space Benchmark: A new benchmark built around persistent executable workspaces where requirements, data, code, and solver outputs are distributed across multiple files.
Task Modes: The definition of three lifecycle-oriented modes (Build, Revise, Explain) that capture model construction, maintenance, and communication.
Evaluation Framework: A protocol for analyzing agent capabilities and failure modes in workspace-grounded tasks, including data grounding, revision consistency, and explanation faithfulness.

Results

The authors evaluated 20 models (closed-source frontier, small API-based, and open-source) across the benchmark. Key findings include:

Workspace Grounding is Difficult: Even frontier agents struggle with workspace-grounded model construction. The best Build score was 72.0% (gemini-3.1-pro), with most frontier models in the 53–68% range. This highlights the difficulty of recovering data schemas and business semantics from distributed artifacts compared to clean prompts.
Legacy Code as Signal vs. Noise: In the Revise task, strong models (e.g., gpt-5.4, claude-opus-4-6) often improved performance by leveraging legacy code as a signal (e.g., gpt-5.4 improved from 59% to 79%). Conversely, weaker or "flash" models often suffered performance drops (e.g., gemini-3-flash dropped 23 percentage points), suggesting they treated implementation details as noise or copied non-solver imports.
Explainability is Distinct: Performance in Explain was weakly correlated with Build and Revise scores ( $r \approx 0.16-0.28$ ). Some models that built runnable models failed to explain them faithfully (e.g., gemini-2.5-pro solved 69% of Revise instances but scored only 32.2 on Explain).
Interface Impact: Using a filesystem interface (vs. a flat-prompt serialization) generally reduced performance in code-generation tasks (Build and Revise) by an average of 8.2 and 12.2 percentage points, respectively. This indicates that file discovery and schema inspection are substantive difficulties, not cosmetic ones.
Context Sensitivity: Adding formal formulations to code context generally helped strong models but could hurt lightweight models. Formulation-only context was found to be more brittle than code-based context.

Significance and Claims

The paper positions OR-Space as a necessary shift in evaluation paradigms from isolated text generation to robust, solver-grounded engineering workflows. The authors claim that:

Reliability Gap: Navigating an open, multi-artifact workspace remains highly challenging for current LLM agents, exposing systemic failures in data mapping and constraint grounding that text-only benchmarks largely hide.
Practical Readiness: OR-Space provides a rigorous foundation for measuring the practical readiness of optimization agents by simulating the iterative, multi-step nature of industrial OR.
Failure Modes: The benchmark enables fine-grained attribution of failures, distinguishing between mathematical modeling errors, data grounding errors, and hallucinated constraints, which is critical for deploying agents in real-world settings.

The work concludes that while LLMs show promise in isolated formulation tasks, their ability to perform reliable OR engineering across a persistent workspace and full lifecycle is currently limited, necessitating further research into agent reliability, failure modes, and workspace interaction.