🤖 AI

Automated Algorithm Design for Auto-Tuning Optimizers

This paper proposes a novel framework that leverages large language models to automatically generate and iteratively refine specialized optimization algorithms for auto-tuning, demonstrating that these LLM-designed optimizers can significantly outperform state-of-the-art human-designed methods across diverse real-world applications and hardware platforms.

Original authors: Floris-Jan Willemsen, Niki van Stein, Ben van Werkhoven

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Floris-Jan Willemsen, Niki van Stein, Ben van Werkhoven

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brand-new, incredibly powerful car engine (like a modern computer chip). To get the most speed and fuel efficiency out of it, you need to tweak thousands of tiny dials: how the fuel mixes, the timing of the spark plugs, the tire pressure, and so on.

The Problem:
There are so many dials (millions of combinations) that a human mechanic trying to find the perfect setting by turning them one by one would take a lifetime. This is the world of Auto-Tuning. Computers have to find the "perfect settings" for their software to run fast on specific hardware, but the search space is a giant, messy maze.

Traditionally, we've used "smart" algorithms (like Genetic Algorithms or Simulated Annealing) to navigate this maze. Think of these as experienced, but rigid, tour guides. They know some tricks, but they are hand-crafted by humans and often get stuck in dead ends or take too long to find the exit.

The New Idea:
This paper asks a bold question: What if we didn't hire a human to design the tour guide, but instead asked an AI (a Large Language Model or LLM) to write a brand-new, custom tour guide for every single maze?

The authors built a system called LLaMEA (Large Language Model Evolutionary Algorithm) to do exactly this.

How It Works: The "AI Chef" Analogy

Think of the LLM as a brilliant but inexperienced chef.

The Order: The computer tells the chef, "We need a recipe (an algorithm) to find the best settings for this specific cooking task (auto-tuning)."
The Ingredients: The chef is given a list of ingredients (the problem description, the search space, and the rules).
The Cooking: The chef writes a "recipe" (code for a new optimization algorithm).
The Taste Test: The computer immediately tries this new recipe on a real problem.
- If the dish tastes terrible (the algorithm is slow or buggy), the recipe is thrown in the trash.
- If the dish tastes good, it's kept.
Evolution: The computer takes the best recipes, mixes them together, and asks the chef to tweak them slightly to make them even better. This happens over and over, like a cooking competition where the winners get to cook again with new ideas.

The Big Discovery

The researchers tested this on four real-world problems (like processing radio signals from space, simulating heat on a processor, and doing complex math). They compared the AI-generated chefs against the human-designed tour guides.

Here is what they found:

The AI Can Outperform Humans: The best AI-generated algorithms were 72% better than the best human-made algorithms currently used in the industry. They found the "perfect settings" much faster and more reliably.
Context is King: When the AI was given extra details about the specific problem (like "this maze has a lot of dead ends" or "this engine runs hot"), it performed even better. It's like telling the chef, "This is a spicy dish, don't use too much sugar." The AI used that info to create a specialized, super-efficient guide.
Self-Healing: If the AI wrote code that crashed (like a recipe that catches fire), the system didn't give up. It showed the error to the AI and said, "Fix this." The AI would rewrite the code, learn from the mistake, and try again.

The Result

The paper proves that we don't need to rely solely on human experts to design the tools that optimize our computers. Instead, we can use AI to automatically design its own optimization tools.

In simple terms:
Instead of hiring a human to design a map for a specific city, we taught an AI to draw a perfect, custom map for every single city it visits. And guess what? The AI drew better maps than the human cartographers ever could.

This opens the door to a future where computer software automatically tunes itself to run perfectly on any device, from your phone to a supercomputer, without needing a team of engineers to manually tweak the settings.

1. Problem Statement

Auto-tuning is critical for optimizing high-performance computing (HPC) applications (e.g., scientific simulations, AI) across diverse hardware architectures. It involves searching vast, irregular, and noisy design spaces to find optimal program configurations (e.g., thread layouts, loop transformations).

The Challenge: Traditional auto-tuners rely on classical optimization algorithms (Simulated Annealing, Genetic Algorithms, Particle Swarm Optimization). However, these algorithms often require careful manual hyperparameter tuning and are not inherently designed for the specific characteristics of auto-tuning search spaces (large, discontinuous, non-convex).
The Gap: Designing robust, near-optimal optimization algorithms manually for every new auto-tuning task is infeasible. The paper asks: Can Large Language Models (LLMs) be leveraged to automatically generate effective, problem-specific optimization algorithms for auto-tuning?

2. Methodology

The authors propose a novel framework that integrates LLaMEA (Large Language Model Evolutionary Algorithm) with Kernel Tuner, an established auto-tuning framework. The core idea is to use LLMs as generative engines to synthesize optimization algorithms, which are then evolved and selected based on empirical performance.

System Architecture

Target Framework (Kernel Tuner):
- An open-source Python framework for tuning computational kernels (CUDA, HIP, OpenCL, etc.).
- It defines a search space $X$ of valid configurations and uses an optimization strategy to select candidates for compilation and execution.
- The authors extended Kernel Tuner to accept externally defined optimization algorithms via an OptAlg wrapper.
Generator (LLaMEA):
- LLaMEA uses an evolutionary loop where an LLM generates candidate optimization algorithms in code.
- Prompting: The LLM is prompted with the task description, code format specifications, and optionally, specific search space characteristics (parameter ranges, constraints).
- Evolutionary Loop:
  - Initialization: The LLM generates an initial population of algorithms (e.g., 4 parents).
  - Evaluation: Algorithms are executed within Kernel Tuner on a training set of search spaces. Performance is measured via a specific metric (see below).
  - Selection & Mutation: High-performing algorithms are selected. The LLM applies mutation operators (e.g., "refine strategy," "generate new algorithm") to create offspring.
  - Self-Debugging: If generated code fails (runtime errors, timeouts), stack traces are fed back to the LLM to repair the code in the next iteration.
Evaluation Metric (Performance Score $P$ ):
- To ensure fair comparison across diverse search spaces, the authors use a community-standard metric that calculates the area under the performance-versus-time curve.
- The score is normalized against a baseline (typically random search) and a known optimum.
- $P = 0$ implies parity with the baseline; $P = 1$ implies finding the global optimum immediately.
- This metric captures both search efficiency (speed) and solution quality.

3. Key Contributions

First Closed-Loop Framework: Introduced the first system that evolves LLM-generated optimizers under formal auto-tuning evaluation metrics.
Novel Paradigm: Demonstrated that LLMs can synthesize entirely new optimization strategies (not just tuning hyperparameters of existing ones) tailored to auto-tuning.
Problem-Specific Synthesis: Showed that providing search space details to the LLM significantly improves algorithm performance.
Superiority over Human Design: Proved that LLM-generated algorithms can outperform state-of-the-art human-designed algorithms (GA, SA, DE) in auto-tuning contexts.
Open Source: All code, data, and the best-performing generated algorithms are integrated into the Kernel Tuner framework and available on GitHub.

4. Experimental Results

Experimental Setup

Benchmarks: 4 real-world applications (Dedispersion, 2D Convolution, Hotspot, GEMM) representing diverse domains (astronomy, image processing, thermal simulation, linear algebra).
Hardware: Evaluated across 6 different GPUs (AMD and Nvidia) from DAS-6 and LUMI supercomputers, creating 24 unique search spaces.
Models: Used GPT-4o-mini as the primary LLM.
Baselines: Compared against Kernel Tuner's best human-designed GA and SA (extensively hyperparameter-tuned) and pyATF's Differential Evolution (DE).

Key Findings

Impact of Search Space Information:
- Providing the LLM with specific search space details (parameter values, constraints) resulted in an average performance improvement of 14.6% over prompts without this info.
- For specific targets like Dedispersion and GEMM, the improvement was substantial (up to +25% in specific cases).
Performance vs. Human-Designed Algorithms:
- The best LLM-generated algorithms (e.g., HybridVNDX for Dedispersion and AdaptiveTabuGreyWolf for GEMM) achieved an average performance improvement of 72.4% over the state-of-the-art human-designed optimizers (GA, SA, DE) across all search spaces.
- The LLM-generated algorithms showed superior robustness and adaptability across different hardware architectures.
Algorithm Characteristics:
- The best-performing generated algorithms combined elements of Variable Neighborhood Descent (VND), Tabu Search, Simulated Annealing, and surrogate modeling (k-NN).
- They shared two key traits: lightweight control logic (treating evaluation time as dominant) and architecture-agnostic neighbor moves that respect constraints.

5. Significance and Conclusion

This paper represents a paradigm shift in auto-tuning: moving from hand-crafted heuristics to automatically generated, adaptive algorithms.

Efficiency: It eliminates the need for domain experts to manually design or tune optimization strategies for every new hardware/application combination.
Scalability: The approach allows for the rapid discovery of optimization strategies that exploit shared structural characteristics across different problems while benefiting from specialization.
Future Impact: The authors suggest this could become a foundational capability in compilers and ML deployment systems, enabling self-adapting software that automatically generates the best search strategy for any given hardware workload.

The work demonstrates that LLMs are not just capable of writing code but can act as meta-optimizers, designing the very algorithms used to optimize high-performance computing systems.