Technical Summary: LLaMEA-SAGE

Problem Statement

Automated Algorithm Design (AAD) using Large Language Models (LLMs) has emerged as a promising paradigm for discovering optimization algorithms directly from natural language prompts. However, existing LLM-based approaches, such as LLaMEA, rely almost exclusively on fitness feedback (performance on benchmarks) to guide the evolutionary search. This "black-box" feedback loop leaves a substantial amount of information within the generated code unused. Consequently, search strategies often struggle to balance exploration and exploitation in large, heterogeneous code spaces, leading to premature stagnation or inefficient use of expensive LLM queries. Furthermore, there is limited understanding of how to systematically improve LLM-generated algorithms post-initial generation and how these improvements compare to state-of-the-art (SOTA) methods under controlled benchmarks.

Methodology: LLaMEA-SAGE

The paper proposes LLaMEA-SAGE, a framework that integrates structured, explainable AI (XAI) feedback into the LLaMEA evolutionary algorithm design process. The core mechanism involves extracting structural features from the code of candidate algorithms and using them to guide subsequent mutations.

1. Structural Feature Extraction

Each candidate algorithm is represented as executable Python code. The system parses the code into an Abstract Syntax Tree (AST) to construct a directed graph. From this graph, the system extracts a rich set of features, including:

• Graph-theoretic statistics: Node/edge counts, degree statistics (mean, variance, entropy), tree depth, clustering coefficients, and path-based metrics (diameter, average shortest path).
• Complexity indicators: Cyclomatic complexity, function token counts, and parameter counts aggregated across functions.

2. Archive-Based Surrogate Modeling

During the evolutionary process, all evaluated solutions, their fitness scores, and their extracted code features are stored in an archive. Once a minimum archive size is reached, a gradient-boosted regression tree model (XGBoost) is trained to approximate the relationship between code features and observed fitness. This surrogate model learns non-linear relations between algorithmic structure and performance.

3. Explainable AI (SHAP) for Guidance

To derive actionable instructions, the system applies SHAP (SHapley Additive exPlanations) to the trained surrogate model. SHAP values quantify the contribution of each feature to the predicted fitness. The feature with the largest absolute attribution is selected, and the sign of its SHAP value determines whether the feature value should be increased or decreased to improve performance.

4. Guided Mutation Prompting

The structural guidance is translated into a natural-language instruction that augments the standard mutation prompt provided to the LLM. For example, if the analysis suggests that increasing cyclomatic complexity correlates with better performance, the prompt is modified to: "Based on archive analysis, try to increase the total cyclomatic complexity of the solution."
Crucially, this guidance biases the LLM's generation toward empirically beneficial structural regions without imposing rigid syntactic constraints or predefined templates, preserving the expressivity of the LLM.

Key Contributions

1. LLaMEA-SAGE Framework: The proposal of a novel mechanism that guides AAD using feature-driven feedback derived from AST and complexity analysis, mediated by a surrogate model and SHAP analysis.
2. Systematic Benchmarking: A comprehensive experimental comparison against SOTA LLM-based heuristic design methods, specifically MCTS-AHD (Monte Carlo Tree Search for Algorithm Heuristic Design) and LHNS (LLM-driven Heuristic Neighborhood Search), alongside the vanilla LLaMEA baseline.
3. Empirical Validation: Demonstration that structured guidance improves algorithm quality and robustness on black-box optimization benchmarks, supported by extensive analysis using Code Evolution Graphs (CEGs) and statistical testing.

Experimental Results

The authors evaluated LLaMEA-SAGE across two primary experiments using the IOH-BLADE benchmarking toolbox.

Experiment 1: Controlled Comparison (SBOX-COST)

• Setup: Compared vanilla LLaMEA against LLaMEA-SAGE on a subset of the SBOX-COST suite (10 dimensions, 5 instances).
• Findings: LLaMEA-SAGE achieved better performance faster than vanilla LLaMEA, particularly in the early stages of the evolutionary process. While both methods reached similar final performance within the budget, the guided approach showed less variance across runs.
• Statistical Significance: LLaMEA-SAGE showed an average improvement of 11.1 Area Under the Convergence Curve (AUC) units with a large effect size (Cliff's $\delta$ = 0.60). However, due to the limited number of independent runs (5 seeds), the difference did not reach statistical significance ($p=0.44$).
• Behavioral Analysis: Code Evolution Graphs revealed that the guidance successfully biased structural exploration. Notably, the "refine" prompt led to increased code complexity when guided, whereas the "random new" prompt often ignored the guidance or resulted in lower complexity.

Experiment 2: Large-Scale Benchmark (MA-BBOB)

• Setup: Evaluated on the Many Affine BBOB (MA-BBOB) suite, a standard for GECCO competitions, comparing LLaMEA-SAGE against vanilla LLaMEA, MCTS-AHD, and LHNS.
• Findings: LLaMEA-SAGE consistently outperformed all baselines, including the SOTA MCTS-AHD and LHNS methods. It reached fitness levels that vanilla LLaMEA could not achieve even after the full evaluation budget.
• Efficiency: In terms of token usage, LLaMEA-SAGE was marginally more expensive than vanilla LLaMEA but significantly more efficient than MCTS-AHD (which queries the LLM multiple times per solution).
• Generalization: Final validation on unseen instances (different dimensions and problem instances) showed that the best algorithm discovered by LLaMEA-SAGE outperformed other automatically designed algorithms and the 2025 MA-BBOB competition winner on 10 and 20-dimensional tasks, though it did not surpass the tuned competition winner on the 5-dimensional task it was specifically optimized for.

Significance and Claims

The paper claims that LLaMEA-SAGE successfully bridges the gap between code structure and human-understandable performance feedback in automated algorithm design. By translating structural signals into natural language, the method provides a data-driven inductive bias that steers LLM-driven evolution without restricting the model's creative capacity.

The authors emphasize that even imperfect, noisy correlations derived from code features are sufficient to beneficially bias exploration in the early stages of AAD. The work demonstrates that signals derived from code can effectively improve the efficiency and stability of algorithm discovery, offering a scalable, template-free alternative to existing heuristic-centric approaches.

Limitations and Future Work

The authors modestly note several limitations:

• LLM Dependency: Experiments were conducted with a single LLM backend (GPT-5-mini), though preliminary ablation on Gemini showed consistent trends.
• Scope: Evaluation was limited to continuous black-box optimization at moderate dimensions; generalization to high-dimensional, noisy, or fundamentally different problem classes remains an open question.
• Static Analysis: The current feature set captures only static structural properties, ignoring dynamic execution behavior or interaction effects during optimization runs.

Future work is proposed to combine structural features with runtime descriptors, explore adaptive multi-feature guidance, and integrate the mechanism into other LLM-based AAD frameworks.