🤖 AI

Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers

This survey categorizes the use of Large Language Models in mathematical optimization into direct, tool-augmented, and tool-creating paradigms, analyzing their performance frontiers and arguing that while direct methods hold future potential, tool-augmented approaches currently offer superior auditability and that tool-creating strategies may ultimately provide the best operational efficiency for repetitive problems.

Original authors: Roko Peran, Luka Hobor, Mihael Kovac, Mario Brcic

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roko Peran, Luka Hobor, Mihael Kovac, Mario Brcic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the Large Language Model, or LLM) who knows how to solve puzzles. You give them a messy, real-world problem written in plain English, like "Figure out the best way to deliver 50 packages to different houses without running out of gas."

This paper surveys three different ways this librarian can try to solve that problem. Think of them as three different job descriptions for the librarian.

1. The "Guess-and-Check" Librarian (Direct Optimization)

How it works: The librarian tries to solve the puzzle entirely in their head. They write down a solution, check if it's good, realize it's a bit off, and then write a new, slightly better one. They keep doing this loop of "guess, check, improve" over and over.

The Analogy: It's like trying to find the best route through a maze by walking it, hitting a wall, turning back, and trying a different path, all without a map.
The Catch: This works great for small mazes (simple problems). But if the maze gets huge (like a city with thousands of streets), the librarian gets confused. They start making up rules that don't exist or forgetting important details. The paper notes that once the problem gets too complex, this method hits a "reasoning cliff" and stops working well.

2. The "Translator and Project Manager" Librarian (Tool-Augmented Optimization)

How it works: Instead of solving the puzzle themselves, the librarian acts as a translator. They take your messy English description and turn it into a strict, formal mathematical language (like a code or a blueprint). Then, they hand that blueprint to a super-fast, specialized robot (a "solver") that is an expert at crunching numbers. The robot solves it, and if it makes a mistake, the librarian tries to fix the blueprint and asks the robot again.

The Analogy: The librarian is the architect who draws the blueprints, and the robot is the construction crew that actually builds the house. The librarian doesn't lay the bricks; they just make sure the plans are right.
The Catch: This is the most accurate method for big, structured problems. However, the librarian is only as good as their translation. If they write the blueprint wrong (e.g., forgetting a wall or a door), the robot will build a perfect house that doesn't match what you actually wanted. The paper highlights that while the robot is perfect at math, the librarian often makes "silent mistakes" in the translation that are hard to spot.

3. The "Tool-Maker" Librarian (Tool-Creating Optimization)

How it works: Instead of solving one specific puzzle or translating one specific request, the librarian invents a brand-new machine or a reusable rulebook that can solve entire families of similar puzzles. Once they build this machine, they can use it over and over again without needing to ask the librarian for help again.

The Analogy: Instead of walking through the maze every time, the librarian designs a robot that can walk through any maze. They spend a lot of time and energy building the robot once, but then they can send the robot through 1,000 mazes instantly and for free.
The Catch: It takes a lot of effort and computing power to build the tool in the first place. Also, sometimes the tools they invent are brilliant, but other times they are just lucky guesses that can't be repeated by other people. The paper notes that this is the fastest-growing area because, once the tool is built, it's incredibly efficient.

The Big Takeaway: Which is Best?

The paper compares these three approaches like a trade-off menu:

Direct (Guess-and-Check): Good for small, messy problems where you don't have a clear rulebook. But it gets unreliable when things get complicated.
Tool-Augmented (Translator): The most reliable for big, structured problems (like logistics or scheduling) if the translation is perfect. It gives you a "certificate of correctness" from the robot, but only if the librarian didn't mess up the instructions.
Tool-Creating (Tool-Maker): The most efficient for the long run. If you have to solve the same type of problem over and over, it's better to spend the energy once to build a reusable tool than to keep asking the librarian to solve it from scratch every time.

The "Reasoning Gap"
The paper concludes with a warning: Even the smartest librarians (the most advanced AI models) struggle with deep, complex logic. They are really good at recognizing patterns they've seen before, but they often fail when they have to invent a completely new logical path.

The Future Strategy
The authors argue that the smartest move for the future isn't to make the librarian smarter at solving every single puzzle directly. Instead, the future lies in Tool-Making. Even if we have super-smart AI in the future, it will likely be more efficient to have that AI build a specialized tool once, and then let a smaller, cheaper AI use that tool to solve thousands of problems instantly. It's the difference between hiring a master chef to cook every meal for a restaurant versus hiring them to invent a recipe and a machine, then letting a line cook run the machine.

Technical Summary: Large Language Models as Optimizers

Problem Statement

Real-world optimization problems often originate as natural language descriptions (e.g., logistical constraints, experimental goals) that require translation into formal mathematical programs before traditional solvers (e.g., Gurobi, CPLEX, Z3) can process them. While Large Language Models (LLMs) have shown promise in bridging this gap, their application in complex mathematical optimization faces significant hurdles. The core challenge lies in navigating the trade-off between the flexibility of direct LLM reasoning and the mathematical rigor of external solvers, particularly as problem complexity scales. Current architectures struggle with a "reasoning gap," where LLMs rely on pattern matching rather than genuine algorithmic reasoning, leading to formulation errors that persist even when solvers execute code without syntax errors.

Methodology: Three Paradigms

The survey categorizes LLM-based optimization into three distinct paradigms, each with unique mechanisms and performance characteristics:

1. Direct Optimization

In this paradigm, the LLM acts as the optimizer itself, navigating the solution space without external solvers.

Mechanism: The model generates candidate solutions, receives feedback on objective values, and iteratively refines them using prompting strategies like Chain-of-Thought (CoT), Tree-of-Thought (ToT), and iterative prompting (e.g., OPRO, SGE).
Evolution: The approach has evolved from simple prompting to fine-tuning (e.g., E2E CO Solver, LLaMoCo), where models internalize optimization heuristics via supervised fine-tuning (SFT) and reinforcement learning (RL) rather than reasoning from scratch.
Scope: Effective for ill-structured problems and expensive-evaluation settings (e.g., hyperparameter tuning, chemical synthesis) where domain knowledge accelerates search.

2. Tool-Augmented Optimization

Here, the LLM serves as a translator and orchestrator, converting natural language into formal specifications (LP, MILP, SAT) that are dispatched to external solvers.

Mechanism: The workflow follows a pipeline: Natural Language $\rightarrow$ LLM Formulation $\rightarrow$ Solver Code $\rightarrow$ Execution $\rightarrow$ Error Feedback. Advanced systems employ multi-agent architectures (Formulator/Programmer/Evaluator) or data-centric fine-tuning (e.g., ORLM, SIRL) to improve formulation accuracy.
Focus: The primary challenge is "formulation correctness." Recent research (2025–2026) focuses on detecting "silent formulation failures" where code executes cleanly but encodes the wrong problem. Techniques include semantic anchor reconstruction (SAC-Opt), combinatorial search for alternative models (Autoformulation), and behavioral verification (ReLoop).

3. Tool-Creating Optimization

This paradigm uses LLMs to discover reusable algorithms or heuristics that operate within solver or metaheuristic frameworks, rather than solving individual problem instances.

Mechanism: LLMs act as mutation operators in evolutionary search (e.g., FunSearch, EoH) or generate "verbal gradients" to refine heuristics (ReEvo).
Economic Model: While incurring high upfront LLM costs to generate a tool, these approaches achieve effectively zero marginal LLM cost upon deployment, as the reusable tool is amortized over repeated use.

Key Contributions and Results

Performance Frontiers and Benchmarks

The paper synthesizes performance data from recent literature to establish current frontiers:

Direct Optimization: Achieves strong results on small-scale or specific domains (e.g., +50% improvement on BBH via OPRO; chemical yield improvements from 25.2% to 60.7% via LLAMBO). However, performance degrades significantly beyond ~20 variables (e.g., on TSP), where simple heuristics outperform LLMs.
Tool-Augmented Optimization: Achieves near-perfect accuracy on simple benchmarks (e.g., 97.3% on NL4Opt) but faces a "complexity cliff." On real-world industrial problems (IndustryOR), accuracy drops to 57%, and on compositional multi-constraint scenarios (RetailOpt-190), it falls to 31.1%.
Tool-Creating Optimization: Has produced state-of-the-art (SOTA) results in algorithm discovery, such as bin-packing strategies outperforming classical heuristics (FunSearch) and scheduling heuristics deployed on Google's Borg cluster (AlphaEvolve) yielding a 23% speedup.

The Reasoning Gap

A critical contribution of the survey is the identification of the "reasoning gap."

Feasibility vs. Correctness: Models like DeepSeek-V3.2 achieve 91.1% solver feasibility but only 0.5% formulation correctness in compositional scenarios.
Pattern Matching: Evidence suggests LLMs rely on pattern matching rather than algorithmic reasoning. When problem complexity increases, reasoning effort peaks and then declines, leading to "complete accuracy collapse."
Error Types: Systematic errors include missing constraints, incorrect variable types, and objective function mismatches.

Comparative Trade-offs

The survey provides a comparative analysis (Table II) highlighting that:

Tool-Augmented approaches offer the highest auditability and formal guarantees (if the formulation is correct) but are bottlenecked by formulation accuracy.
Direct approaches offer low auditability and no formal guarantees but excel in small-scale or ill-structured contexts.
Tool-Creating approaches offer novel SOTA heuristics with medium auditability, balancing high upfront cost with zero marginal deployment cost.

Significance and Claims

The paper argues that the choice of paradigm depends on the specific problem constraints and model capabilities.

Limitations of Direct Approaches: While direct optimization is expanding, the reasoning gap at higher complexity limits its scalability.
The Case for Tool-Making: The authors posit that "tool-making" (creating reusable solver components or formulation templates) is the most promising direction for the future. Even as models become more powerful, the per-instance opportunity cost of direct solving may outweigh the benefits. Creating a reusable tool amortizes the LLM's reasoning cost across entire families of problems, offering a more operationally efficient path than per-instance reasoning.
Reproducibility Crisis: The survey highlights a crisis in the tool-creating subfield, where SOTA claims (e.g., FunSearch's cap set) are often based on results that are difficult to replicate independently due to proprietary infrastructure or path-dependence. The authors call for community norms to label closed-system claims and report result distributions rather than best-case outcomes.

In conclusion, the survey identifies that while LLMs are increasingly involved in optimization, the field must address the "complexity cliff" and the "formulation correctness" bottleneck. The future trajectory likely favors hybrid or tool-creating strategies that leverage LLMs for algorithm discovery and template generation, reserving formal solvers for execution and verification.