SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE

In the world of data science, computers are often given vast spreadsheets filled with numbers to solve problems, from predicting loan defaults to diagnosing diseases. However, these raw numbers are rarely enough on their own. To get the best results, experts must perform a process called automated feature engineering, which involves creating new, smarter columns of data by combining or transforming the existing ones. Imagine a chef who doesn't just use the ingredients in the pantry but invents new flavor combinations to make a dish taste better. For decades, computers have tried to do this by blindly mixing and matching numbers, but this approach is slow and often produces confusing results. Recently, scientists began using large language models—powerful artificial intelligence systems that can reason and learn from context—to act as these creative chefs. These AI systems were thought to be the key to unlocking better predictions, but they faced a major hurdle: they usually needed detailed descriptions of what every number meant to work properly. In the real world, such descriptions are often missing due to privacy rules or simply because the data comes from sensors that don't speak human language. Furthermore, when these AI systems tried to learn from their past attempts to improve the data, the list of their history grew so long that it overwhelmed the computer's memory, causing the system to get stuck in a loop of repeating the same mistakes.

A team of researchers at Yokohama National University has developed a new method called SIGMA to solve these problems, allowing artificial intelligence to engineer better data without needing any descriptions of what the numbers represent. Instead of relying on human language to understand the data, SIGMA uses a mathematical tool called SHAP to measure how much each piece of information actually matters for the final prediction. Think of this tool as a spotlight that highlights the most important numbers and dims the less useful ones, giving the AI a clear sense of direction without needing to know what the numbers are called. The researchers then grouped these numbers into three categories: the most important, the moderately useful, and the weak. They asked the AI to create new data columns by mixing numbers within these groups or bridging the strong ones with the weak ones, effectively teaching the system to focus its creativity where it counts most.

The most significant innovation in this work is a technique the researchers call an "exposed-feature implicit trajectory." In previous attempts, AI systems would keep a written list of every feature they had ever created to avoid making duplicates, but this list quickly became too long to fit in the computer's memory. SIGMA takes a different approach. Instead of writing down a history, it simply hides the features that have already been used from the AI's view for a short time. If the AI tries to create a feature using a number that is currently hidden, it is forced to look elsewhere. This simple act of hiding and revealing acts as a silent guide, steering the AI away from repeating itself without ever needing to store a long history log. This allows the system to run for much longer periods, refining its work over many steps without getting bogged down by memory limits or stuck in a cycle of generating the same useless features over and over.

The results of this new approach are striking. When tested on sixteen different real-world datasets, SIGMA performed just as well as the best existing AI methods that rely on detailed descriptions, proving that the system does not need human labels to be effective. More importantly, it solved the problem of repetition. In earlier systems, nearly thirty-seven percent of the features generated were duplicates, wasting valuable computing power. With SIGMA's method of hiding used features, that duplication rate dropped dramatically to less than seven percent. The system also proved to be incredibly efficient. While traditional methods often generate hundreds of new features to find a few good ones, SIGMA achieved top-tier performance using an average of only five new features per dataset. This means the system finds the most valuable improvements quickly, leaving the rest of the data clean and manageable.

The researchers also discovered that this method allows the AI to build complex, multi-layered structures of data that were previously out of reach. In some tests, the system successfully combined features in a chain, using the output of one step as the input for the next, creating a deep network of relationships that improved predictions significantly. This capability allowed SIGMA to outperform older, non-AI methods that rely on rigid, pre-defined rules. The study confirms that by replacing the need for human language with mathematical importance signals and using a clever system of hiding and revealing data, artificial intelligence can become a more powerful and practical tool for improving data analysis, even in situations where no human descriptions are available.