🤖 AI

An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

This paper proposes the LLM-driven evolutionary benchmark generator (LLM-EBG), a framework that utilizes large language models as evolutionary operators to automatically generate diverse and structurally complex optimization benchmarks that effectively differentiate the performance of distinct algorithms like genetic algorithms and differential evolution.

Original authors: Yuhiro Ono, Tomohiro Harada, Yukiya Miura

Published 2026-01-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuhiro Ono, Tomohiro Harada, Yukiya Miura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to figure out which of two athletes is better at running. If you only test them on a perfectly flat, straight track, you might think they are equally fast. But in the real world, races happen on muddy hills, winding paths, and slippery surfaces. To truly know who is the better runner, you need a course that specifically challenges one runner's strengths and weaknesses while testing the other's.

This paper introduces a new "coach" for computer algorithms called LLM-EBG. Its job is to automatically design these tricky "racing courses" (math problems) to see which computer algorithm wins.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Fake" Tracks

Usually, scientists test computer optimization algorithms (like Genetic Algorithms or Differential Evolution) on standard, pre-made math problems. Think of these as a gym with only one type of treadmill.

The Issue: These standard tracks are too perfect and predictable. They don't look like the messy, irregular problems found in the real world.
The Alternative: Scientists could use real-world problems (like designing a car engine), but those are expensive, secret, and hard to share.
The Goal: We need a way to automatically build new, unique "tracks" that are just hard enough to tell the difference between two algorithms.

2. The Solution: The "AI Architect"

The authors created a system called LLM-EBG.

The Architect (The LLM): They used a Large Language Model (like the AI you might chat with) not to write stories, but to act as a "genetic operator." Imagine the LLM as a creative architect who can draw new blueprints for math problems.
The Evolution: The system doesn't just draw one blueprint and stop. It runs an "evolutionary" process:
1. Generation: The AI architect draws a batch of new math problems.
2. The Race: Two different algorithms (GA and DE) try to solve these problems.
3. Selection: The system checks who won. If the AI architect made a problem where the "Target" algorithm (say, GA) won easily, that problem is kept. If the "Target" lost, the problem is discarded or changed.
4. Mutation & Crossover: The AI takes the winning problems and "breeds" them. It might swap parts of two problems together (crossover) or tweak a number in a formula (mutation) to make the next generation of problems even better at highlighting the differences.

3. The Experiment: The "GA vs. DE" Duel

The researchers tested this system with two specific algorithms:

GA (Genetic Algorithm): Like a team of explorers spreading out to search a wide area.
DE (Differential Evolution): Like a group of hikers who follow the steepest path down a hill very quickly.

They asked the AI to create two types of courses:

GA-Friendly Courses: Problems where the "explorers" (GA) should win because the terrain is tricky and requires broad searching.
DE-Friendly Courses: Problems where the "hikers" (DE) should win because the path is smooth and steep.

4. The Results: The AI Got It Right

The system was surprisingly successful:

Success Rate: In 8 out of 10 attempts to make a "GA-friendly" course, the GA consistently beat the DE. In 9 out of 10 attempts to make a "DE-friendly" course, the DE beat the GA.
The "Why": The researchers analyzed the math problems the AI created.
- The GA-friendly problems were like jagged, uneven terrain where the size of the steps mattered a lot (variable scaling). The "explorers" were good at handling this mess.
- The DE-friendly problems were smoother and more consistent, allowing the "hikers" to zoom to the finish line.

5. How the AI "Thinks"

The researchers watched how the AI architect changed the problems over time:

For DE-friendly problems: The AI mostly made small, careful tweaks (mutations) to a single line of code, refining it step-by-step.
For GA-friendly problems: The AI mixed and matched parts of different problems (crossover), combining ideas from different "parents" to create something new and complex.

The Bottom Line

This paper shows that we can use a smart AI to automatically invent new, challenging math problems that perfectly expose the strengths and weaknesses of different computer algorithms. Instead of relying on old, static tests, we can now evolve custom "racing tracks" that tell us exactly which algorithm is the best runner for a specific type of terrain.

Technical Summary: An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

Problem Statement
Optimization benchmarks are essential for evaluating algorithm performance, yet existing approaches face significant limitations. Conventional artificial benchmarks (e.g., COCO, CEC suites) rely on predefined mathematical formulations that often fail to capture the heterogeneity and irregularity of real-world problem structures. Conversely, benchmarks derived from real-world applications are frequently costly to construct, difficult to reproduce, and restricted by confidentiality. While automatic benchmark generation methods exist, they typically depend on predefined parametric generators or tree-based symbolic representations (e.g., genetic programming), which constrain the diversity and expressiveness of the generated problem landscapes. There is a need for a framework that can automatically generate diverse, reproducible benchmark instances with specific structural characteristics without relying on rigid, pre-specified templates.

Methodology: LLM-EBG
The authors propose the LLM-driven Evolutionary Benchmark Generator (LLM-EBG), a framework that integrates Evolutionary Algorithms (EAs) with Large Language Models (LLMs) to evolve benchmark problems.

Representation: Benchmark problems are represented as text-based symbolic mathematical expressions (Python code) for unconstrained single-objective continuous minimization.
Evolutionary Process: The framework treats benchmark problems as individuals in an evolutionary population. The LLM serves as the generative operator for initialization, crossover, and mutation, guided by prompt engineering.
- Initialization: An initial population is generated using a seed function and few-shot prompting, where the LLM is conditioned on existing benchmarks to induce structural similarity while introducing new components.
- Crossover: Two parent benchmarks are provided to the LLM as in-context examples, which is instructed to generate a new problem combining structural components of both parents.
- Mutation: A single parent is provided, and the LLM is instructed to modify the expression structurally (e.g., changing constants, substituting functions, or adding terms).
Prompting Strategy: The system uses few-shot prompting with a strict output format (single-line Python code) to ensure syntactic validity. The prompt explicitly defines the target performance relationship (e.g., "Create a problem where Algorithm A1 outperforms Algorithm A2").
Evaluation Function: The fitness of a generated benchmark is determined by the relative performance of two target algorithms (a Genetic Algorithm [GA] and Differential Evolution [DE]) across $T$ trials. The fitness function minimizes the normalized rank of the target algorithm's performance relative to the comparative algorithm. It includes a penalty term to prevent the optimal solution from diverging to negative infinity and penalizes benchmarks that produce invalid (e.g., complex or infinite) values.
Validation: A pre-validation step ensures that generated expressions produce valid real-valued outputs for randomly sampled design variables before being included in the population.

Key Contributions

Novel Framework: The introduction of LLM-EBG, which leverages LLMs as evolutionary operators to search a symbolic expression space directly, moving beyond the constraints of predefined parametric generators.
Algorithm-Specific Benchmark Generation: The successful demonstration of generating benchmarks that induce significant performance disparities between GA and DE. The framework can systematically evolve problems where GA consistently outperforms DE, and vice versa.
Structural Analysis: The application of Exploratory Landscape Analysis (ELA) to characterize the generated benchmarks. The study reveals that GA-favoring benchmarks exhibit high sensitivity to variable scaling (ill-conditioning), while DE-favoring benchmarks show more moderate curvature disparities.
Evolutionary Dynamics: An analysis of the LLM's operator behavior, showing that mutation (particularly the introduction of novel symbolic terms) is crucial for escaping local optima, while crossover facilitates the combination of partial structures.

Experimental Results
The framework was tested in two scenarios: "GA-preferred" and "DE-preferred" searches, using 10 independent trials for each.

Success Rate: LLM-EBG successfully generated benchmarks where the designated target algorithm outperformed the comparative algorithm in more than 80% of trials. Specifically, the theoretical minimum fitness (indicating perfect dominance) was reached in 8 out of 10 GA-preferred trials and 9 out of 10 DE-preferred trials.
Landscape Characteristics:
- GA-preferred benchmarks were found to be highly sensitive to variable scaling, characterized by strong anisotropy and ill-conditioned landscapes (similar to BBOB10 and BBOB11). These landscapes favor GA's ability to maintain population diversity.
- DE-preferred benchmarks exhibited lower sensitivity to variable scaling and smoother landscapes, allowing DE to exploit relative positional differences effectively.
Search Behavior: On GA-preferred benchmarks, DE tended to converge prematurely to local optima, whereas GA maintained robustness. Conversely, on DE-preferred benchmarks, DE converged rapidly to near-optimal solutions, while GA struggled with multiple local optima.
Operator Usage: The DE-preferred search relied more heavily on mutation operations (6.9% lower crossover ratio than GA-preferred), suggesting that local structural modifications are more effective for evolving DE-favoring landscapes. The GA-preferred search utilized crossover and mutation at nearly equal frequencies.

Significance and Claims
The paper claims that LLM-EBG establishes a more principled and reliable methodology for automatic benchmark generation compared to existing approaches. By treating the LLM as an evolutionary operator, the framework can explore a flexible, expressive representation space to generate problems with distinct geometric characteristics that reflect the intrinsic search behaviors of different optimization algorithms.

The authors assert that the framework successfully addresses the limitations of artificial benchmarks (lack of diversity) and real-world benchmarks (cost and reproducibility) by producing reproducible, mathematically defined problems that specifically challenge or favor certain algorithmic mechanisms. The study concludes that LLM-EBG can automatically identify and amplify key geometric properties, such as anisotropy and ill-conditioning, that differentiate the search behaviors of evolutionary algorithms. Future work is proposed to extend the framework to constrained and multi-objective optimization and to a broader range of algorithms.