🤖 machine learning

Evolving Executable Pipeline Programs for AutoML with Language Models

LACE is a novel AutoML framework that employs an evolutionary loop guided by a large language model to generate and evolve executable Python pipeline programs, achieving competitive accuracy on 68 OpenML tasks while offering superior transparency, editability, and search space flexibility compared to traditional systems.

Original authors: Sofoklis Kitharidis, Cor J. Veenman, Jan N. van Rijn, Thomas Bäck, Niki van Stein

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sofoklis Kitharidis, Cor J. Veenman, Jan N. van Rijn, Thomas Bäck, Niki van Stein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern data science, machines are increasingly asked to make sense of messy, real-world information: predicting whether a loan applicant will repay, diagnosing a disease from a spreadsheet of symptoms, or forecasting sales based on past trends. To do this, researchers build "pipelines," which are step-by-step instructions that clean the data, choose a mathematical model, and tune that model until it performs well. For years, the most powerful tools for this job have been automated systems that act like a master chef with a fixed set of ingredients. These systems can mix and match known pre-processing steps and learning algorithms, adjusting the quantities to find the best flavor. However, they are strictly limited to the pantry they were given; they cannot invent a new ingredient or a new cooking technique that falls outside their pre-defined list. This limitation means that if the best solution requires a structure the system has never seen before, it will remain undiscovered.

A team of researchers at Leiden University has now introduced a different approach, one that treats the creation of these data pipelines not as a selection from a menu, but as the writing of a new recipe from scratch. They developed a system called LACE, which uses a large language model—an artificial intelligence trained on vast amounts of text and code—to act as an evolutionary engine. Instead of choosing from a fixed list of tools, LACE asks the AI to write complete, executable computer programs that define how to process data and make predictions. The system then tests these programs, learns from their mistakes, and asks the AI to write improved versions, repeating this cycle until it finds a highly effective solution. The result is a system that does not just tweak existing tools but can compose entirely new structures, producing code that human experts can read, understand, and modify directly.

The researchers tested this method on 68 different data classification tasks, ranging from small datasets with a few hundred entries to massive collections with nearly five million rows. They compared LACE against several established automated systems and a set of fixed, pre-trained models. The results showed that LACE, when powered by a specific language model, performed as well as the strongest existing automated systems and significantly better than older methods. Crucially, while some of the newer pre-trained models were slightly more accurate on the specific tasks they could handle, they failed to work on many of the larger or more complex datasets in the test set. LACE, by contrast, successfully generated a working solution for every single task it was given.

What makes this achievement particularly notable is the nature of the output. Traditional automated systems often return a "black box"—a complex, fitted object that works well but is difficult for a human to inspect or change without deep technical knowledge of the software framework. LACE returns ordinary Python code. This means a data scientist can look at the final program, see exactly how the data was cleaned, which models were chosen, and how they were combined, and then edit the code to improve it further or adapt it for a new purpose. The researchers found that the AI did not just copy existing patterns; it evolved diverse solutions, often combining different types of models in novel ways and writing custom logic to blend their predictions. In many cases, the AI wrote its own code to mix the outputs of multiple models, a level of flexibility that fixed systems cannot easily replicate.

The study also addressed a common concern in artificial intelligence research: that the AI might simply be memorizing the answers to the test questions because it has seen them before during its training. To prevent this, the researchers hid the names and specific details of the datasets from the AI, giving it only a general description of the data's size and type. Despite this lack of specific knowledge, the system still found high-performing solutions, suggesting it was genuinely learning how to construct effective pipelines rather than recalling answers. The researchers further verified that the AI followed strict rules, such as not using hidden internal searches to tune its own settings, ensuring that the improvements came from the evolutionary search process itself.

One of the most striking findings was how quickly the system improved. The first program generated by the AI was often flawed or incomplete, but as the system iterated, it learned to fix these errors and refine the logic. By the time the process finished, the final programs were not only valid but significantly more accurate than the initial attempts. The researchers observed that the system tended to favor certain types of models, such as tree-based ensembles, but it did not get stuck in a single pattern; it explored a wide variety of structural combinations. This diversity suggests that the system is capable of discovering solutions that a human might not think to try, or that a rigid system would never consider.

The work demonstrates that treating automated machine learning as a process of writing and evolving code opens up a new frontier. It moves beyond the constraint of a fixed library of tools, allowing the system to adapt its structure to the specific problem at hand. While the computational cost of running these experiments was high, requiring significant time to generate and test thousands of candidate programs, the results suggest that the ability to produce transparent, editable, and highly effective code is a valuable trade-off. The researchers conclude that this approach offers a promising path forward, where the search for the best data pipeline is defined by the creativity of the code itself, rather than by the limitations of a pre-selected set of components.

Technical Summary: Evolving Executable Pipeline Programs for AutoML with Language Models

Problem Statement

Traditional Automated Machine Learning (AutoML) systems operate within a fixed, predefined space of operators, learners, and hyperparameters. While systems like auto-sklearn, AutoGluon, and H2O AutoML can effectively select and tune components from curated libraries, they cannot generate structural innovations outside their specified grammars. Conversely, tabular foundation models (e.g., TabPFN) bypass per-task search by applying pretrained predictors, but they return fixed models rather than editable, task-specific programs. Furthermore, existing Large Language Model (LLM) agents for data science often target open-ended experimentation or feature engineering, lacking a rigorous framework for evolving constrained pipeline programs where the search space is defined by code generation rather than a static operator inventory.

A critical challenge in applying LLMs to AutoML is the risk of benchmark contamination, where models may recognize public datasets from their pretraining data, conflating pipeline design with dataset memorization. Additionally, outputs from classical AutoML are often opaque framework objects, while foundation models are black-box predictors, limiting practitioner inspectability and reusability.

Methodology: LACE

The authors introduce LACE (LLM-Automated Code Evolution), an AutoML framework that formulates the search for tabular pipelines as the evolutionary optimization of complete, executable Python programs.

Core Architecture

LACE employs a $(\mu + \lambda)$ evolutionary loop where the LLM acts as the variation operator:

Representation: Candidates are Python classes implementing a scikit-learn-compatible interface: __init__(X, y) fits the model once, and __call__(X) returns predictions. The search space is implicitly defined by the programs the LLM can generate under specific constraints, rather than a fixed grammar.
Constraints: The prompt restricts candidates to standard libraries (scikit-learn, numpy, scipy, pandas), limits ensembles to three learners, enforces a single shared preprocessing stage, and explicitly forbids internal cross-validation, nested AutoML, or internal hyperparameter search loops. Tunable hyperparameters must be exposed as constructor arguments.
Evolutionary Process:
- Initialization: An initial population of $\mu=4$ pipelines is generated from a coarse task summary.
- Variation: For each generation, a parent is selected, and the LLM generates an offspring by modifying the parent's code based on evaluation feedback and a randomly selected mutation instruction (e.g., change model family, refine preprocessing, adjust ensemble).
- Selection: The best $\mu$ individuals from the combined parent and offspring pool are retained (elitist truncation).
Evaluation Protocol:
- Leakage Control: To prevent dataset recognition, the generator receives only a coarse summary (task type, approximate sample count, feature counts) and no dataset names, identifiers, or raw values.
- Fitness: Candidates are evaluated on inner-validation splits derived from the official training folds. The official test folds are never accessed during the search.
- Final Assessment: After the candidate budget ( $B=100$ ) is exhausted, the best pipeline is refit on the full training data and evaluated once on the official test folds.

Experimental Setup

The framework was evaluated on 68 classification tasks from the OpenML amlb-classification-all benchmark suite. The study compared LACE (using three LLM backbones: gpt-5.4-mini, deepseek-v4-flash, and qwen3-coder-next) against:

Classical AutoML: AutoGluon-Tabular, auto-sklearn, H2O AutoML.
Tabular Foundation Models: TabPFN v2.5, TabPFN v3, TabICL.
Baseline: A fixed XGBoost pipeline.

All methods were subject to identical time budgets per task repetition to ensure a fair comparison of search efficiency.

Key Results

Performance and Coverage

Competitive Accuracy: On the 45 tasks where all methods produced valid results, LACE with gpt-5.4-mini achieved a mean rank of 4.71, significantly outperforming auto-sklearn, H2O, and the fixed XGBoost baseline. It showed no statistically significant difference from AutoGluon (the strongest search-based baseline) and TabPFN v2.5.
Superior Coverage: Unlike foundation models (TabPFN v2.5, v3, and TabICL), which failed on tasks exceeding their input size or feature limits, LACE covered 100% of the 68 tasks. When accounting for coverage (penalizing missing results), LACE (gpt-5.4-mini) and AutoGluon remained tied for the top position, while TabPFN v2.5 dropped significantly in rank due to its inability to handle 23 tasks.
Backbone Sensitivity: The choice of LLM backbone impacted performance. gpt-5.4-mini was the strongest configuration, outperforming deepseek-v4-flash and qwen3-coder-next with consistent, albeit small, median gains.

Search Dynamics and Program Quality

Zero-Shot Recovery: The initial zero-shot candidates had validity rates ranging from 37.6% to 79.7% depending on the backbone. However, the evolutionary process successfully recovered valid pipelines in 100% of runs, demonstrating the framework's robustness to initial generation failures.
Generalization: The search improved generalization, not just the inner-validation objective. When valid initial programs were refit and tested on held-out data, the final evolved pipelines improved accuracy by a median of 2.99 percentage points.
Structural Diversity: The generated pipelines did not collapse into a single template. Among 1,020 final pipelines, 99.9% included custom preprocessing, and 73.3% combined multiple learners. The search space exhibited high structural diversity, with 65,298 unique structural signatures across valid candidates.
Inspectability: A unique contribution of LACE is the output format. Unlike framework-specific objects or fixed pretrained weights, LACE returns ordinary Python code. The authors audited 89 final pipelines (8.7%) that implemented prediction logic (e.g., weighted ensembles) directly in NumPy or Python, allowing practitioners to inspect, modify, and reuse specific task decisions directly.

Significance and Claims

The paper claims that LACE establishes executable programs as a viable and promising search representation for AutoML. Its primary contributions are:

Code-Defined Search Space: It replaces fixed operator grammars with a search space defined by the LLM's ability to generate constrained code, allowing for the discovery of novel pipeline structures and preprocessing combinations that fixed grammars might miss.
Full Coverage with Editability: It achieves complete coverage of a diverse benchmark while returning task-specific, editable programs. This contrasts with foundation models that offer high accuracy on supported tasks but lack flexibility and with classical AutoML that returns opaque framework objects.
Rigorous Evaluation: The study demonstrates that LLM-driven evolution can compete with state-of-the-art search-based AutoML systems when evaluated under strict leakage-controlled protocols that withhold dataset identity.

The authors modestly note that the effective search space is shaped by the components announced in the prompt (as shown in an ablation study where unannounced TabPFN components were never used) and that the computational cost is high, though search dynamics suggest potential for adaptive budgeting. The work does not claim raw accuracy superiority over all foundation models on all tasks but emphasizes the trade-off between the flexibility of editable code and the performance of specialized pretrained predictors.