💻 computer science

Better Together, in the Right Order: Classical-then-LLM Optimization for SE

The paper introduces SNAP2, a novel approach that outperforms both standalone large language models and classical-optimizer-led hybrid methods in software engineering configuration tasks by first using a cheap classical learner to seed an LLM, thereby achieving superior results with reduced cost and faster execution.

Original authors: Srinath Srinivasan, Tim Menzies

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Srinath Srinivasan, Tim Menzies

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the single best recipe for a cake out of a cookbook containing 100,000 pages. You have two tools to help you:

The Fast Scout (Classical Optimizer): A robot that can flip through pages incredibly fast, taste a few, and quickly narrow down the search to the "good" section of the book. It's cheap, fast, and doesn't need to think deeply, but it might miss the perfect cake because it's just looking for "good enough."
The Master Chef (Large Language Model): A brilliant human chef who can taste a recipe and imagine exactly how to tweak it to make it perfect. However, this chef is slow, expensive to hire, and gets tired if you ask them to taste every single page in the book.

The Old Way: Asking the Chef First

For a long time, researchers tried to solve this by asking the Master Chef to start the process. They'd say, "Here is the book, guess the first few recipes, and then we'll use the robot to finish the job."

The paper argues this is inefficient. The chef spends a lot of time and money (tokens) guessing blindly at the start, only to have the robot take over later. It's like hiring a world-famous chef to walk into a dark room and guess where the light switch is, before handing the job to a janitor to find the rest.

The New Way: SNAP2 (The Scout First, Then the Chef)

The authors discovered a much better order: Let the Fast Scout do the heavy lifting first, then hand the results to the Master Chef to finish.

They call this method SNAP2. Here is how it works in their experiment:

The Scout (EZR): First, they let the cheap, fast robot scan the first 10 pages. It quickly identifies the "best" section of the cookbook and throws away the terrible recipes.
The Chef (SNAP): Then, they give those 10 "good" recipes to the Master Chef. The chef says, "Ah, I see the pattern now. Based on these good starts, I can imagine the perfect recipe."

The Results: Why the Order Matters

The paper tested this on 105 different real-world software problems (like tuning computer settings to make them faster or cheaper).

The Chef Alone: If you just ask the Master Chef to do the whole job, they get the top result about 75% of the time.
The Scout Alone: If you just use the robot, it gets the top result about 71% of the time.
The Old Combo (Chef first, then Robot): This only got about 70-74% success.
The New Combo (SNAP2): By letting the robot scout first and the chef finish, they hit the top result 85% of the time.

The Analogy: It's like playing a video game.

Old Way: You ask a genius strategist to guess the first move, then you play the rest.
SNAP2 Way: You play the first few levels yourself to get a feel for the controls (the cheap robot work), then you ask the genius strategist to give you the winning strategy for the final boss. The strategist performs much better because they aren't starting from zero; they are building on a solid foundation.

The Bonus: It's Also Cheaper

Because the robot does the "boring" early work, the Master Chef doesn't have to read as many pages or think as hard.

SNAP2 uses 30% fewer "tokens" (the currency used to pay the AI) than using the Chef alone.
It runs 1.4 times faster.

The Bottom Line for Practitioners

The authors conclude that you shouldn't choose between a robot or a human; you should use both, but in the right order.

Start with the cheap, fast method. It's nearly free and gives you a great head start.
Only then, bring in the expensive AI. Use the AI to polish the results the robot found.

If you need the absolute best result for a critical system, use SNAP2. If you are on a tight budget or need an answer instantly, just use the robot (EZR), because it's surprisingly good and runs thousands of times faster. But never let the AI start from scratch if you can help it.

Technical Summary: Better Together, in the Right Order

Problem Statement

Software engineering (SE) configuration is a critical yet increasingly difficult task. Misconfigurations in systems like MySQL, Apache, and Hadoop account for over 40% of failures, with single bad settings causing performance degradations up to 480×. The search space for these configurations has exploded; for instance, PostgreSQL options have grown 300% and MySQL 600% over 15 years. While search-based software engineering (SBSE) has long utilized stochastic search and Bayesian optimization, recent efforts to integrate Large Language Models (LLMs) have yielded conflicting results. Some studies suggest LLMs offer limited advantage, are only useful for small problems, or are too slow for practical use.

The core issue identified is that existing research often treats LLMs and classical optimizers in isolation or combines them in suboptimal ways. Specifically, prior "hybrid" approaches (termed opt2) typically let a classical optimizer drive the search loop while the LLM only assists in subroutines (e.g., warm-starting or proposing mutations). The authors argue that the evidence base is too thin to settle disputes, often relying on fewer than five datasets where variance swamps effect sizes.

Methodology

The study evaluates seven optimization methods across three distinct regimes using the MOOT repository, a collection of 127 real-world SE optimization tasks (reduced to 105 for fair comparison).

The Three Regimes

opt1 (LLM-only): The LLM acts as the sole optimizer, re-prompting itself with the trajectory of tried points (e.g., OPRO, extended here as SNAP).
opt2 (LLM-assisted): A classical optimizer owns the search loop; the LLM assists (e.g., BS_LLM, SYNTHCORE). In these methods, the LLM typically seeds the process first.
opt3 (Classical-only): A classical optimizer runs without LLM involvement (e.g., EZR, RRP, RANDOM).

The Proposed Method: SNAP2

The paper introduces SNAP2, a hybrid approach that reverses the traditional order of operations found in opt2 methods.

Mechanism: SNAP2 first runs a cheap, classical active learner (EZR) for a portion of the labeling budget (the first half). EZR selects informative rows to label, building a high-quality initial trajectory.
Handoff: The results of this classical phase are then used to "warm-start" the LLM.
Optimization: The LLM (using the SNAP algorithm, an extension of OPRO) takes over the remaining budget to refine the search and finish the optimization.
Key Distinction: Unlike prior hybrids where the LLM seeds and the classical optimizer finishes, SNAP2 lets the classical learner do the inexpensive early work and the LLM perform the final optimization.

Experimental Setup

Datasets: 105 tasks from MOOT covering software configuration, performance tuning, project health, and defect prediction.
Budget: A fixed labeling budget of B=20 samples per run to ensure fairness and cost constraints.
Metrics: Performance is measured by distance to heaven (d2h), normalized to a percentage improvement ( $\Delta$ ). Statistical significance is determined via Cliff's delta and Kolmogorov-Smirnov tests to define a "top-tier" of indistinguishable best methods.
Cost: Token usage, wall-clock time, and dollar cost were tracked.

Key Contributions

SNAP2 Algorithm: An optimizer that seeds an LLM with the output of a cheap classical learner, reversing the standard warm-start direction.
Large-Scale Evaluation: A comprehensive benchmark across all three regimes (opt1, opt2, opt3) on 105 real-world SE tasks, addressing the lack of broad evidence in prior literature.
Evidence on Order: Demonstration that the order of combination matters. Combining a classical learner and an LLM beats either alone, but specifically, the "classical-first, LLM-second" ordering (SNAP2) significantly outperforms the "LLM-first, classical-second" ordering used in prior work.
Reproducibility: All code, prompts, and data are released under an open-source license.

Results

The study evaluated 20 repeats per method on 105 datasets.

Top-Tier Performance:
- SNAP2 reached the top tier on 85% of tasks (89/105).
- SNAP (LLM-only) reached 75%.
- EZR (Classical-only) reached 71%.
- BS_LLM and SYNTHCORE (LLM-first hybrids) reached 74% and 70%, respectively.
Significance: SNAP2 is the only method to break the statistical cluster of 70–75% performance, achieving an 11-point lead over the best LLM-first hybrid.
Cost Efficiency:
- Compared to the LLM-only method (SNAP), SNAP2 uses roughly 30% fewer tokens and runs 1.4× faster.
- EZR (Classical-only) is virtually free (zero tokens) and runs three orders of magnitude faster than SNAP2, though it is statistically slightly inferior in top-tier frequency.

Significance and Claims

The paper concludes that it is unwise to study classical learners or LLMs in isolation. The primary significance lies in the finding that order matters.

Reversing the Paradigm: Prior hybrid work assumed the LLM should lead (seed) and the classical optimizer should finish. The authors show that the reverse—letting a cheap classical learner do the "inexpensive early work" and handing the trajectory to the LLM to "finish"—yields superior results.
Practical Guidance:
- For Mission/Safety-Critical Work: Use SNAP2. It offers the highest quality (85% top-tier rate) while being more efficient than using an LLM alone.
- For Resource-Constrained Environments: Use EZR. It provides near-optimal results (71% top-tier) at near-zero cost and extreme speed, making it ideal for edge computing or tight API budgets.
- Avoid: Pure LLM-only optimization (SNAP) is never the right choice as it is dominated by SNAP2 in both quality and cost.

The authors emphasize that their results apply to tabular selection problems where configurations are projected onto known rows. They acknowledge that extending this to generative settings (where configurations are not pre-scored) remains an open question. The study suggests that for many SE optimization tasks, a small budget (B=20) is sufficient when guided by the right combination of classical and LLM reasoning.