Technical Summary: PEARL – Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

1. Problem Statement

Optimization modeling involves translating real-world decision problems, typically described in natural language (NL), into formal mathematical formulations and executable solver code. While Large Language Models (LLMs) have shown promise in automating this, existing approaches largely treat the task as a one-shot generation problem. Current methods either rely on fixed prompting, hand-designed workflows, or supervised fine-tuning to produce a complete formulation in a single pass.

This paradigm fails to capture the inherently interactive nature of real-world modeling, where human experts engage in repeated solve–debug–revise cycles. Minor errors in constraints, indexing, or data binding often render models infeasible or produce silent runtime errors. Existing automated systems rarely condition on solver feedback, execute partial models to diagnose issues, or learn when to terminate. Consequently, generated outputs frequently hallucinate constraints or fail at runtime. The paper argues that optimization modeling should be reconceptualized not as static text generation, but as a multi-turn, tool-integrated decision process where the agent learns to interact with solvers to verify and refine its outputs.

2. Methodology: PEARL Framework

The authors introduce PEARL (interactive optimization modeling with tool-integrated agentic reinforcement learning), a system that formalizes optimization modeling as an episodic Partially Observable Markov Decision Process (POMDP).

2.1 POMDP Formulation

• State ($S$): The latent environment state includes the true problem semantics, hidden modeling errors, and solver internal states, which are not directly observable.
• Observation ($O$): The agent observes the NL specification, conversation history, intermediate code artifacts, and crucially, tool feedback (runtime logs, solver status like "infeasible" or "optimal," objective values, and validation reports).
• Action Space ($A$): The agent selects between two action types:
  1. Natural Language Actions: Revising or extending the mathematical formulation (variables, objectives, constraints) and implementation plans.
  2. Tool-Invocation Actions: Triggering the Execute tool (to run Python code and invoke solvers) or the Validate tool (to perform deterministic checks).
• Termination ($T$): The episode ends when the agent emits a final answer (formulation + code) or exhausts a turn/time budget.

2.2 Environment and Tools

The environment provides two first-class tools:

• Execute: Runs model-generated Python code in a sandboxed, resource-bounded runtime. It returns structured feedback including solver status (optimal, infeasible, unbounded), objective values, and extracted solutions.
• Validate: Performs deterministic checks on artifacts (e.g., constraint satisfaction) and compares objective values against reference solutions if available.
  Unlike prior work where execution is a post-hoc evaluator, PEARL incorporates these traces and diagnostics directly into the observation stream, enabling the agent to iteratively revise based on feedback.

2.3 Reward Design

The reward function $R(\tau)$ is designed to be verifiable and solver-grounded, avoiding text-based proxies. It consists of two components:

1. Format Reward ($r_{fmt}$): Provides dense incentives for correct tool invocation syntax and adherence to output formats, stabilizing training by discouraging invalid interactions.
2. Correctness Reward ($r_{corr}$): Grounded in solver execution. If a reference optimal value $v_{label}$ exists, the agent receives a positive reward if the generated solution's objective value $v_{pred}$ satisfies $|v_{pred} - v_{label}| \le \epsilon$. This directly rewards solver-correctness rather than heuristic well-formedness.

2.4 Training Objective

PEARL employs Agentic Reinforcement Learning with multi-turn rollouts.

• Policy Optimization: The policy is initialized from a pretrained LLM and optimized using GRPO-style reinforcement learning with decoupled clipping (inspired by DAPO) to handle high-variance returns in tool-augmented environments.
• Credit Assignment: Rewards are sparse and delayed. The system normalizes trajectory-level returns within a batch to compute advantages, enabling long-horizon credit assignment over the entire solve–debug–revise trajectory rather than just the final output.
• Curriculum: Training adapts to optimization-specific challenges, moving from short, fully specified instances to long-context and industrial-style inputs, while treating solver nondeterminism as environment noise.

3. Key Contributions

1. Reconceptualization of Modeling: The paper formalizes optimization modeling as an interactive, multi-turn decision problem where solver execution and diagnostic feedback are treated as observable state components, distinct from traditional one-shot generation.
2. Agentic RL for Optimization: It introduces the first agentic RL approach specifically for optimization modeling, enabling agents to learn when to execute, how to revise based on solver diagnostics, and when to terminate, rather than relying on fixed heuristics.
3. Empirical Superiority with Compact Models: Extensive experiments demonstrate that PEARL substantially improves verified solve rates over strong one-shot and tool-augmented baselines. Notably, PEARL-Qwen3-4B (a 4B parameter model) outperforms significantly larger general-purpose models, including DeepSeek-V3.2-685B, in both macro- and micro-averaged accuracy on optimization tasks.

4. Experimental Results

The authors evaluated PEARL across diverse benchmarks (NL4Opt, MAMO, NLP4LP, OptMATH, IndustryOR, ComplexOR, Resocratic).

• Performance vs. Scale: PEARL-4B achieved a 69.71% macro-average Pass@1, outperforming DeepSeek-V3.2-685B (66.51%) and GPT-5 (71.40%). It showed particularly strong gains on complex, underspecified benchmarks like MAMO ComplexLP and Resocratic.
• Interaction vs. Scaffolding: Ablation studies confirmed that the performance gains stem from the learned interaction policy, not merely the presence of a tool scaffold. A base model using the same tool interface but without agentic RL training achieved significantly lower accuracy.
• Failure Mode Analysis: PEARL training shifted failure modes away from execution and parsing errors (e.g., code/runtime, format extraction) toward more modeling-centric issues (e.g., objective mismatch, infeasibility). This indicates the agent successfully learned to repair implementation errors and interact with solvers, though structural formulation errors remain challenging.
• Training Dynamics: Tool usage behavior converged early in training, while later training refined the detail and structure of the modeling trajectory. Supervised Fine-Tuning (SFT) served as a useful warm-start but was not the primary driver of final performance compared to solver-grounded agentic RL.

5. Significance and Claims

The paper claims that learned interaction and feedback utilization can outweigh raw model scale in structured decision-making problems like optimization. By treating modeling as an interactive process, PEARL enables compact models (e.g., 4B parameters) to achieve performance comparable to or exceeding much larger general-purpose models, making repeated agentic invocation more practical for deployment.

The authors maintain a modest stance regarding the scope of their claims:

• The results demonstrate effectiveness in the current public NL-to-Opt regime.
• The benchmarks do not exhaust real-world complexity; stochastic, robust, and highly domain-specific enterprise settings remain under-represented.
• Solver-based validation is partial; while it improves feasibility, it cannot fully guarantee structural correctness against ambiguous specifications.

Future work is suggested to extend agentic modeling to stochastic and robust settings with richer validators and more adaptive interaction policies.