🤖 machine learning

Adaptive Protection for Evolutionary Feature Construction in Symbolic Regression with Application to Credit Classification

This paper proposes an adaptive protection mechanism that utilizes feature importance metrics to selectively preserve valuable constructed features during evolutionary symbolic regression, thereby preventing the loss of critical genetic material and significantly improving solution quality across both regression and classification tasks.

Original authors: Hengzhe Zhang, Qi Chen, Bing Xue, Lean Yu, Wolfgang Banzhaf, Mengjie Zhang

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hengzhe Zhang, Qi Chen, Bing Xue, Lean Yu, Wolfgang Banzhaf, Mengjie Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent the perfect recipe for a new dish. You have a pantry full of basic ingredients (the raw data), and your goal is to mix them together to create something delicious (a mathematical model that predicts the future). In the world of computer science, this is called Symbolic Regression. Instead of just following a fixed recipe, the computer uses a method called Evolutionary Feature Construction. Think of this like a chaotic cooking show where the computer tries thousands of random ingredient combinations, tastes them, and keeps the ones that taste good. Over time, it "evolves" better and better recipes by mixing and mutating the successful ones.

However, there's a catch. In this chaotic kitchen, the computer's tools (called genetic operators) sometimes accidentally chop up a perfect, delicious sauce you just made, just to try a new combination. It's like a sous-chef smashing a perfect cake to see if a new flavor works, only to ruin the cake and not find anything better. This paper tackles that problem. It introduces a smart "safety net" that protects the best ingredients and sauces you've already discovered, ensuring they don't get accidentally destroyed while the computer keeps experimenting with the rest. The researchers tested this on a massive library of 98 different data puzzles and even on real-world credit card applications to see if their safety net helps computers find better answers faster.

The Problem: The "Smashing Chef"

In the world of Symbolic Regression, computers try to find the mathematical formula that best explains a set of data. To do this, they often use Evolutionary Feature Construction. Imagine the computer is building a tower out of blocks. It starts with simple blocks (the original data) and tries to glue them together in new ways to build complex structures (new features).

The computer uses a process similar to biological evolution: it creates a population of these block towers, checks which ones are the strongest (best at predicting data), and then mixes them up. It takes a piece of one tower and swaps it with a piece of another (crossover) or randomly changes a block (mutation). The goal is to find the ultimate tower.

But here is the trouble: sometimes, the computer finds a really strong, stable block structure—a "building block" that makes the tower very strong. When the computer tries to mix and match, it might accidentally break this perfect block, replacing it with a weaker one. It's like if a master builder spent hours perfecting a single brick, only for a clumsy assistant to knock it off the table and replace it with a crumbly stone. The paper argues that existing methods often lack a way to protect these valuable, hard-earned discoveries from being accidentally smashed during the mixing process.

The Solution: The "Adaptive Shield"

The authors of this paper propose a clever solution called an Adaptive Protection Mechanism. Think of this as a magical shield that wraps around the computer's building blocks.

Here is how it works in plain language:

The Taste Test: Before the computer starts mixing and smashing blocks, it tastes every single block structure to see how important it is. It uses a "score" to decide how valuable each block is. Some blocks are super important (like the secret spice in a soup), while others are less critical.
The Shield: The computer then applies a shield to these blocks. But it's not a one-size-fits-all shield. The more important a block is, the stronger the shield becomes.
- If a block is super important, the shield is very thick. If the computer tries to smash it during a mix, the shield holds, and the block is restored to its perfect state.
- If a block is less important, the shield is thin or non-existent. The computer is free to smash, change, or replace it to see if it can find something even better.
The Result: This way, the computer keeps its best discoveries safe while still having the freedom to experiment with the rest. It's like a chef who protects the secret sauce but is free to try new vegetables in the salad.

What They Found

The researchers tested this idea on 98 different benchmark datasets (think of these as 98 different puzzles the computer had to solve). They also tried it on two real-world credit classification datasets (determining if a person is likely to pay back a loan).

Does it work? Yes. The paper shows that using this adaptive shield consistently improved the quality of the solutions compared to not using it. The computers found better formulas and made more accurate predictions.
Is it better than just slowing down? The researchers asked, "Is this just because we are doing fewer changes?" They found that simply reducing the number of changes (mutations) didn't work as well. The "smart shield" was better because it knew what to protect, not just how much to protect.
Does the "taste test" matter? They tried different ways to calculate the importance score (like checking the recipe's ingredients or how much the flavor changed). They found that while different methods worked, the approach was robust—it worked well no matter which specific "taste test" they used.

Why This Matters

This paper suggests that by giving computers a way to "remember" and protect their best ideas during the chaotic process of evolution, we can build better models for everything from predicting weather patterns to assessing credit risk. The method doesn't require the computer to stop experimenting; it just makes the experimentation smarter. The authors found that this approach works across different types of computer learners (like decision trees and linear models), suggesting it's a versatile tool for anyone trying to teach computers to find patterns in messy data.

In short, the paper proves that if you want a computer to evolve a perfect solution, you shouldn't just let it smash everything randomly. You need to give it a way to hold onto the good stuff while it keeps looking for the great stuff.

Technical Summary: Adaptive Protection for Evolutionary Feature Construction in Symbolic Regression

Problem Statement

Evolutionary feature construction (EFC) within symbolic regression aims to automatically discover informative transformations of input features to enhance predictive models. While Genetic Programming (GP) effectively evolves these transformations, a fundamental challenge persists: balancing the preservation of useful "building blocks" (high-performing feature transformations) against the generation of new combinations. Standard genetic operators, such as mutation and crossover, often inadvertently disrupt valuable existing features. Modifying or removing an important feature can degrade solution quality by replacing a highly effective building block with a less effective alternative, even if the offspring introduces other useful features. Existing approaches often lack explicit, universal mechanisms to preserve these critical components without requiring the redesign of genetic operators.

Methodology

The paper proposes an Adaptive Protection Mechanism (APM) embedded within a universal protection framework. This mechanism is designed to be compatible with arbitrary genetic operators (e.g., standard subtree crossover and mutation) and various base learners.

Core Workflow

The evolutionary process follows a generational framework with the following key stages:

Initialization & Evaluation: A population of individuals, where each individual consists of a set of feature transformation trees, is initialized. Each individual is evaluated by training a base learner (e.g., Ridge Regression, Random Decision Trees, or Logistic Regression) on the constructed features. Fitness is determined by $R^2$ (regression) or AUC (classification).
Feature Importance Calculation: After evaluation, the importance ( $I_j$ $I_{j}$ ) of each constructed feature (tree) within an individual is calculated. The paper evaluates three distinct methods to demonstrate generality:
- Built-in Coefficients: Using absolute coefficients for linear models or impurity decrease for decision trees.
- SHAP Values: Using mean absolute Shapley values to measure marginal contribution across feature subsets.
- Marginal Contribution: Measuring the drop in $R^2$ when a specific feature is removed and the model is refitted.
Adaptive Protection (The Novelty):
- Before genetic operations (crossover/mutation), the original trees of an individual are saved.
- After operations are applied, a protection score ( $s_j$ ) is computed for each tree $j$ : $s_j = m_{protect} \cdot \tilde{I}_j$ , where $\tilde{I}_j$ is the normalized importance and $m_{protect}$ is a tunable protection coefficient.
- A uniform random variable $u \sim U(0,1)$ is sampled. If $u < s_j$ , the modified tree is restored to its original saved state.
- Logic: Highly important features have a higher probability of being restored (protected), while less important features remain free to be modified or replaced, allowing the search to explore new combinations.

Experimental Setup

Datasets: 98 regression benchmark datasets from the Penn Machine Learning Benchmarks (PMLB) and two credit classification datasets (Australian and German Credit).
Base Learners: Ridge Regression, Random Decision Trees, and Logistic Regression.
Metrics: $R^2$ for regression and AUC for classification. Statistical significance was tested using the Wilcoxon signed-rank test.

Key Contributions

Universal Adaptive Protection Mechanism: The paper introduces a mechanism that selectively preserves constructed features based on importance scores. Unlike previous methods that often require redesigning genetic operators, this approach wraps around existing operators, providing stronger protection for important features while maintaining diversity in less important ones.
Generality Across Metrics and Learners: The study systematically compares multiple feature importance calculation methods (coefficients, SHAP, marginal contribution) and demonstrates the mechanism's robustness across different base learners (linear models, decision trees, and classification models).
Empirical Validation: Extensive experiments on 98 regression datasets show consistent improvements in solution quality over baseline approaches without protection. The method is also validated on credit classification tasks, proving its applicability beyond pure symbolic regression.

Results

Performance Improvement: The proposed mechanism consistently improves solution quality (higher $R^2$ and AUC) compared to baseline approaches across the 98 regression datasets.
Robustness: The approach demonstrates effectiveness regardless of the specific feature importance metric used or the base learner employed.
Credit Classification: Experiments on Australian and German Credit datasets confirm that the method extends effectively to classification problems, improving search effectiveness in real-world financial contexts.

Significance and Claims

The authors claim that the Adaptive Protection Mechanism addresses a critical gap in evolutionary feature construction: the loss of valuable genetic material due to disruptive genetic operators. By leveraging feature importance metrics to guide restoration, the method ensures that high-quality building blocks are preserved more reliably than in standard evolutionary processes.

The paper positions this work as a "universal" solution that does not bind the user to specific genetic operators or model types. It claims to provide a more adaptive form of elitism that operates at the feature (building block) level rather than the individual level, thereby preserving the most valuable components of the population while still allowing the evolutionary search to explore new feature combinations. The study concludes that this mechanism offers a robust, operator-agnostic way to enhance the performance of evolutionary symbolic regression and feature construction.