🤖 machine learning

LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers

This paper introduces LLM-FE, a novel framework that combines evolutionary search with Large Language Models to iteratively discover effective feature transformation programs for tabular data, thereby outperforming existing automated feature engineering methods by better leveraging domain knowledge and data-driven feedback.

Original authors: Nikhil Abhyankar, Parshin Shojaee, Chandan K. Reddy

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikhil Abhyankar, Parshin Shojaee, Chandan K. Reddy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a pile of clues (data). Usually, you have to manually sift through these clues, combine them in clever ways, and create new "super-clues" to help you solve the case faster. This process is called Feature Engineering. In the world of machine learning, doing this manually is slow, boring, and requires a human expert to know exactly which combinations of clues might work.

The paper introduces a new tool called LLM-FE that automates this detective work using a "super-intelligent" AI (a Large Language Model) acting as a creative partner.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Guessing Game"

Traditional automated tools try to solve this by blindly mixing and matching numbers (like adding Column A to Column B) based on a fixed list of rules. They are like a robot that only knows how to turn left or right, never realizing that sometimes you need to jump over a fence. They often miss the "aha!" moments that come from understanding the story behind the data (like knowing that "insulin" and "glucose" are related to diabetes).

2. The Solution: The "Evolutionary Chef"

The authors propose LLM-FE, which treats feature engineering like a cooking competition where the AI is the chef. Instead of just following a recipe, the AI is given:

The Ingredients: The raw data (e.g., patient age, blood sugar levels).
The Menu: The specific problem to solve (e.g., "predict if a patient has diabetes").
The Cookbook: The AI's own internal knowledge of the world (e.g., "I know that high sugar and low insulin are bad for health").

3. How It Works: The Four-Step Loop

The paper describes a cycle that repeats over and over, similar to how nature evolves species to survive better:

Step A: The Creative Spark (Generation)
The AI looks at the data and the problem. It says, "Hmm, if I divide the insulin level by the glucose level, that might tell us something about how the body processes sugar." It writes a small computer program (a recipe) to create this new "super-clue."
- Analogy: The chef invents a new sauce based on the ingredients and the dish they are trying to make.
Step B: The Taste Test (Evaluation)
The new recipe is applied to the data. Then, a prediction model (a simple test) tries to solve the problem using this new data.
- Analogy: The chef serves the new sauce to a judge. If the judge likes it (the score goes up), the recipe is good. If not, it's discarded.
Step C: The Memory Bank (Experience Management)
This is the secret sauce of LLM-FE. The system doesn't just forget the failed recipes. It keeps a "Hall of Fame" of the best recipes it has ever made.
- Analogy: Imagine a chef who keeps a notebook of their 10 best dishes. When they need to invent a new dish, they don't start from scratch; they look at their best past dishes and say, "What if I took the best part of Dish #3 and mixed it with the best part of Dish #7?"
Step D: The Evolution (Refinement)
The AI uses the "Hall of Fame" to create even better versions of the recipes. It tries to "mutate" (change slightly) the best recipes to see if they can get even better.
- Analogy: This is like natural selection. The strongest, tastiest recipes survive and evolve, while the weak ones die out.

4. Why It's Better Than the Rest

The paper compares LLM-FE to other methods and finds it wins for three main reasons:

It Knows the Context: Unlike other robots that just do math, this AI understands what the data means. It knows that "Age" and "BMI" are related to health, so it creates features that make sense to humans, not just numbers.
It Doesn't Get Stuck: Other methods often get stuck trying to improve one single idea. LLM-FE runs multiple "islands" of ideas at the same time. If one island gets stuck in a dead end, another island might find a shortcut.
It Learns from Mistakes: By keeping a memory of what worked before, it gets smarter with every attempt, rather than just guessing randomly.

5. The Results

The researchers tested this on many different datasets (like predicting heart disease, housing prices, and car sales). They found that LLM-FE consistently made the prediction models more accurate than the old methods. It didn't just work with one type of AI; it helped different types of models get better at their jobs.

In a nutshell: LLM-FE is a system that uses a smart AI to invent new, meaningful ways to look at data. It acts like a creative chef who learns from its best past dishes, uses its knowledge of the world, and constantly experiments until it finds the perfect recipe to solve a problem.

Technical Summary: LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers

Problem Statement

Feature engineering is a critical, yet labor-intensive, component of the tabular machine learning workflow. While well-designed features can significantly enhance the performance of tree-based models (often outperforming deep learning approaches on structured data), the process of exploring the vast combinatorial space of feature transformations is time-consuming and resource-intensive. Traditional automated feature engineering methods rely on pre-defined, manually designed search spaces and fixed transformations, often failing to incorporate valuable domain knowledge. Conversely, recent Large Language Model (LLM) based approaches attempt to integrate domain knowledge but typically rely on direct prompting or simple validation scores. These existing LLM methods often fail to leverage insights from prior discovery experiments, lack meaningful reasoning between feature generation and data-driven performance, and tend to generate simple, repetitive features due to inherent biases in LLM pretraining.

Methodology: The LLM-FE Framework

The authors propose LLM-FE, a novel framework that formulates feature engineering as a program search problem, combining the domain knowledge and reasoning capabilities of LLMs with evolutionary search and data-driven feedback. The framework operates as an iterative loop consisting of four primary stages:

New Feature Generation (Prompting):
The LLM acts as a knowledge-guided evolutionary optimizer. It receives a structured prompt containing:
- Task Description & Metadata: Specific details about the downstream task and feature descriptions.
- Data Samples: Serialized examples of the dataset to provide context.
- Evaluation Function: A code snippet defining how feature quality is measured (training a model and scoring on a validation set).
- In-Context Demonstrations: A selection of high-performing feature transformation programs from previous iterations.
  The LLM is explicitly instructed to generate complex, novel feature transformation programs (Python code) rather than simple arithmetic, leveraging its embedded domain knowledge to reason about feature relevance.
Feature Engineering (Execution):
The generated Python programs are executed on the underlying dataset. This step applies the proposed transformations to create an augmented dataset. To ensure robustness, the system enforces execution time and memory limits, discarding error-prone or inefficient programs.
Feature Evaluation (Data-Driven Feedback):
The augmented dataset is used to train a downstream prediction model (e.g., XGBoost, MLP, TabPFN). The model's performance on a held-out validation set serves as the fitness score for the feature transformation program. This score provides the data-driven feedback necessary to guide the search.
Experience Management (Evolutionary Search):
To avoid local optima and maintain diversity, LLM-FE employs a multi-population "island" model.
- Multi-Island Memory: The search space is divided into $m$ independent islands. Each island maintains a buffer of high-scoring programs.
- Clustering & Sampling: Programs within an island are clustered based on their validation scores. The system uses Boltzmann sampling to select clusters for in-context examples, balancing exploration (lower-scoring clusters) and exploitation (higher-scoring clusters).
- Iterative Refinement: In each iteration, the LLM samples from the memory buffer to generate new candidates, effectively performing implicit mutation and crossover through prompt-conditioned generation.

Key Contributions

The paper outlines three primary contributions:

Novel Framework: The introduction of LLM-FE, which casts feature engineering as an LLM-guided evolutionary optimization problem. It uniquely integrates domain knowledge, data-driven evaluation, and long-term memory for iterative refinement.
Empirical Effectiveness: Experimental results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines (both classical methods like OpenFE/AutoFeat and LLM-based methods like CAAFE/FeatLLM) across diverse tabular datasets, tasks, and prediction models.
Component Analysis: Through comprehensive ablation studies, the authors highlight the critical roles of domain knowledge, evolutionary search, data-driven feedback, and data samples in guiding the LLM to efficiently explore the feature space and discover impactful features.

Experimental Results

The authors evaluated LLM-FE using GPT-3.5-Turbo and Llama-3.1-8B-Instruct as backbones across 19 classification and 10 regression datasets. The framework was tested against baselines using XGBoost, MLP, and TabPFN as downstream predictors.

Performance: LLM-FE achieved the lowest mean rank (indicating the best overall performance) across both classification and regression tasks. It consistently improved predictive performance over raw data baselines and other feature engineering methods.
Generalizability: The framework demonstrated robustness across different LLM backbones and prediction model architectures. Features generated by LLM-FE improved performance for tree-based models, neural networks, and transformer-based tabular models.
Complexity & Bias Mitigation: Unlike naive LLM prompting which tends to favor simple operators (addition, subtraction), the evolutionary refinement in LLM-FE successfully discovered complex transformations (e.g., group-by aggregations, residuals, sigmoid functions). Approximately 45% of discovered features were classified as complex, compared to negligible amounts in baseline LLM methods.
Efficiency: A Pareto analysis of performance versus runtime showed that LLM-FE lies on the Pareto frontier, achieving higher accuracy than competitors like CAAFE and OCTree with comparable or lower computational costs.
Ablation Studies: Removing domain knowledge caused a significant performance drop (from ~0.687 to ~0.626 accuracy), underscoring its importance. Similarly, removing evolutionary refinement led to the greatest decline (to ~0.587), highlighting the necessity of iterative data-driven feedback.
Memorization: Experiments on datasets released after the LLM training cutoff dates showed that LLM-FE provided consistent, modest gains, suggesting that improvements stem from genuine reasoning and evolutionary selection rather than data memorization.

Significance and Claims

The paper positions LLM-FE as a significant step forward in automating the data-centric pipeline. Its significance lies in bridging the gap between domain-informed reasoning and optimization. By treating the LLM not just as a generator but as an evolutionary optimizer, the framework overcomes the limitations of static search spaces and the bias toward simple features inherent in direct prompting.

The authors claim that LLM-FE offers a robust, generalizable solution for tabular learning that reduces manual effort while improving predictive performance. They emphasize that the framework is designed to augment, not replace, domain experts, providing interpretable, hypothesis-driven features that can be reviewed and validated. The work suggests that combining LLMs with evolutionary search mechanisms is a viable path toward discovering high-quality, interpretable features that are difficult to engineer manually.