🤖 machine learning

TopoFE: topology-aware LLM-guided Automated Feature Engineering

TOPOFE is a topology-aware multi-island evolutionary framework that enhances LLM-guided automated feature engineering by integrating family-specialized exploration, adaptive prompt memory, and topology-guided knowledge transfer to overcome the limitations of stateless generation and homogeneous search, thereby discovering diverse and high-performing feature programs across 29 tabular datasets.

Original authors: Sha Li, Naren Ramakrishnan

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Sha Li, Naren Ramakrishnan

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to make smart decisions, like predicting if a loan applicant will pay back their money or if a patient has a specific illness. The computer doesn't start with the answers; it needs to look at a spreadsheet of raw data—numbers, categories, and dates—and figure out what patterns matter. This process is called machine learning. But here's the tricky part: the raw data is often messy and hard to understand. Just like a chef can't make a great soup with only raw, unchopped vegetables, a computer often can't find the best patterns unless the data is transformed first. This transformation step is called feature engineering. It's the art of taking raw numbers and mixing them together to create new, smarter clues that help the computer see the truth. For a long time, humans had to do this by hand, guessing which combinations of numbers might be useful. Recently, we've started using super-smart computer programs called Large Language Models (LLMs) to help. Think of these LLMs as brilliant, well-read assistants who know a lot about math and science and can suggest new ways to mix the data. However, even these brilliant assistants have a problem: they sometimes get stuck in a rut, suggesting the same types of mixes over and over, or they forget what they tried before, wasting time on dead ends.

This is where a new framework called TOPOFE comes in. The researchers, Sha Li and Naren Ramakrishnan from Virginia Tech, realized that the space of possible data mixes is so huge and varied that a single, uniform search strategy isn't enough. They proposed a system that treats the search for the best data clues like a team of specialized explorers working in different territories, rather than one giant crowd.

Instead of having one big group of computer programs all trying to solve the problem at once, TOPOFE splits the work into five distinct "islands." Each island is a specialist team focused on a specific type of math trick. One island only looks at simple arithmetic like adding or multiplying numbers. Another focuses on statistics like averages and counts. A third looks at time-based patterns, like how sales change over the last week. Another handles relationships between different groups, and the last one deals with curved, non-linear shapes. By keeping these specialists separate, the system ensures that no single type of math trick takes over and blocks the others from being discovered.

But the real magic happens when these islands talk to each other. In older systems, if a team got stuck, they might just randomly swap ideas with another team, which often didn't help. TOPOFE is smarter. It uses a "topology graph," which is like a dynamic map that learns which teams are best at helping each other. If the "Time" island gets stuck, the system checks its map and sees that the "Arithmetic" island has the best ideas for helping them. It then triggers a special event where the two teams combine their best ideas to create something entirely new—a "hybrid" feature that neither team could have invented alone. For example, it might combine a time-based average with a division operation to create a brand-new, highly predictive clue.

The system also has a "memory" that learns from its own mistakes. If a certain type of math trick keeps failing on a specific dataset, the system remembers this and stops suggesting it, while reinforcing the tricks that are working. This happens without needing to retrain the main computer brain, making the search incredibly efficient.

The researchers tested this idea on 29 different real-world datasets, ranging from predicting heart disease to forecasting bike rentals. They found that TOPOFE consistently outperformed the best existing methods. It didn't just find better answers; it found more diverse answers. While other methods often ended up with a pile of very similar clues (redundancy), TOPOFE's specialized islands and smart teamwork produced a set of clues that covered more ground and worked well with different types of prediction models. The study suggests that by organizing the search into specialized, communicating groups and letting them learn when to swap ideas, we can unlock much more powerful ways to teach computers to understand our data. The results show that this approach is robust, working well even when the underlying computer "brain" changes, proving that the secret sauce is in the structure of the search itself, not just the intelligence of the model.

Technical Summary: TOPOFE

Problem Statement

Automatic Feature Engineering (AutoFE) for tabular data is fundamentally a program synthesis problem aimed at discovering predictive transformations from an exponentially large search space. While recent Large Language Model (LLM) approaches have expanded the expressiveness of AutoFE beyond predefined operator libraries, they suffer from three critical structural limitations:

Homogeneous Search Dynamics: Existing LLM-evolutionary methods operate over a single, undifferentiated population. This causes the search to collapse onto a narrow subset of transformation types (e.g., arithmetic interactions) once a few high-performing motifs saturate the experience buffer, leaving orthogonal families (e.g., temporal dynamics, relational encodings) under-explored.
Stateless LLM Querying: Feature generation lacks persistent memory of prior evaluations. Proposals are generated from static prompts without cumulative adaptation to dataset-specific patterns, leading to repeated, unproductive suggestions.
Rigid Transfer Structure: When multiple populations are used, migration is typically periodic or random. There is no principled mechanism to detect when local search has stagnated or to identify which source family is most complementary to a stagnating target, preventing the discovery of complex cross-family compositions.

Methodology: TOPOFE

TOPOFE (Topology-Aware LLM-Guided Automated Feature Engineering) reframes AutoFE as a graph-structured multi-island evolutionary search. It decomposes the feature program space into semantically coherent "transformation families" and coordinates their exploration through three core components:

1. Multi-Island Decomposition (Inductive Bias Decomposition)

The program space $\mathcal{P}$ is partitioned into $M$ dedicated islands, each assigned a specific transformation family (e.g., arithmetic interactions, statistical aggregates, temporal features, relational encodings, nonlinear univariate maps).

Island State: Each island $i$ maintains a population $\Pi_i$ , an elite archive $A_i$ , an accept/reject history $H_i$ , and a prompt-memory vector $\rho_i$ .
Specialized Exploration: Within each island, the LLM performs mutation and crossover operations conditioned on the specific family's inductive bias and the island's accumulated history, preventing premature convergence to a single transformation type.

2. Prompt Adaptation Memory (PAM)

To address stateless generation, TOPOFE implements a lightweight, per-island memory mechanism that adapts LLM proposals without parameter updates.

Mechanism: After every generation, the system summarizes the accepted and rejected program patterns from a sliding window into a natural-language "prefer/avoid" signal ( $\rho_i$ ).
Conditioning: This signal is injected into every LLM prompt alongside the task metadata and elite demonstrations, enabling cumulative, dataset-specific adaptation throughout the search.

3. Topology-Aware Cross-Island Transfer

To discover features requiring operators from multiple families (e.g., a temporal feature combined with an arithmetic ratio), TOPOFE employs a learned, directed topology graph $G(t)$ .

Saturation Detection: Transfer is not periodic but triggered by a saturation criterion. An island is deemed saturated when its marginal improvement over a sliding window falls below a threshold $\epsilon$ , indicating local search exhaustion.
Adaptive Routing: Upon saturation, the system queries the topology graph to select a precursor island $j^*$ that maximizes the empirically observed transfer utility $w_{j \to i}$ .
Hybrid Synthesis: Instead of copying programs, the LLM performs hybrid synthesis. It receives programs from both the saturated target island and the selected precursor island as context, generating a novel program that composes structural elements from both families.
Graph Learning: Edge weights in the topology graph are updated online via an exponential moving average based on the observed fitness gain of transfer events, learning a data-driven model of inter-family complementarity.

Key Contributions

Topology-Aware Formulation: The paper proposes formulating AutoFE as a graph-structured multi-island search where the migration topology is not fixed but learned online via a bandit-style rule based on empirical transfer utility.
Saturation-Triggered Adaptive Transfer: A utility-based criterion triggers cross-island transfer only when local search is exhausted, coupled with a Prompt Adaptation Memory that enables cumulative adaptation without parameter updates.
Principled Evaluation Framework: The authors introduce three purpose-built metrics:
- Mean Pairwise Output Correlation (MPOC): Measures feature redundancy.
- Effective Rank (EffRank): Measures subspace coverage.
- Topology Graph Specialisation Score (TGSS): Quantifies the degree to which the topology graph acquires task-specific transfer knowledge.
Empirical Validation: Extensive experiments on 29 public tabular datasets (19 classification, 10 regression) demonstrate consistent improvements over state-of-the-art classical and LLM-based AutoFE methods.

Results

Predictive Performance: TOPOFE achieved the best performance on 15 of 19 classification datasets and 9 of 10 regression datasets. The performance gains were most pronounced on datasets requiring complex cross-family interactions (e.g., balance-scale, heart, bike), where single-population methods failed to discover necessary compositional features.
Robustness: The framework remained stable across different LLM backbones (Qwen2.5-Coder, Qwen3-8B, GPT-4o-mini) and different downstream predictors (XGBoost, CatBoost, MLP, TabPFN), confirming that the gains are driven by the architectural mechanisms rather than specific model capabilities.
Feature Diversity: TOPOFE produced feature sets with significantly lower redundancy (lower MPOC) and higher subspace coverage (higher EffRank) compared to baselines. This was attributed to the explicit redundancy filter and the ability to import structurally orthogonal templates via hybrid synthesis.
Topology Specialization: The TGSS metric confirmed that the topology graph successfully learned task-specific transfer pathways. Datasets with high TGSS values (indicating strong learned preferences for specific cross-family transfers) correlated with larger performance gains over single-family baselines.

Significance and Claims

The paper claims that inter-family complementarity is a learnable and exploitable quantity that existing AutoFE methods leave untapped. By moving beyond homogeneous search and static prompting, TOPOFE demonstrates that:

Structured decomposition of the program space preserves diversity and prevents premature convergence.
Adaptive, saturation-triggered transfer allows the system to discover complex, compositional features that are unreachable by single-family grammars.
The learned topology graph acts as a principled mechanism to route knowledge flow precisely when and where it is most valuable.

The authors conclude that their approach opens promising directions for automated family schema induction and task-type-specific transfer scheduling, establishing a new paradigm for LLM-guided feature engineering that balances specialized exploitation with structured exploration.