📊 statistics

Comparing Model-agnostic Feature Selection Methods through Relative Efficiency

This paper introduces a general framework based on relative efficiency to compare model-agnostic feature selection methods, demonstrating through theoretical analysis, simulations, and real-world data that Generalized Covariance Measure (GCM) approaches generally outperform Leave-One-Covariate-Out (LOCO) methods under specific regularity conditions across linear, non-linear additive, and single-index models.

Original authors: Chenghui Zheng, Garvesh Raskutti

Published 2026-07-23

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenghui Zheng, Garvesh Raskutti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you have a massive pile of clues—hundreds of variables like temperature, humidity, shoe size, and the number of times someone blinked. Your goal is to figure out which of these clues actually matter for solving the case (predicting the outcome) and which ones are just noise. In the world of data science and machine learning, this is called feature selection. It's crucial because if you try to solve a puzzle using every single piece of cardboard in the box, you'll get confused, make mistakes, and waste time. You need to find the specific pieces that actually form the picture.

For a long time, detectives had to guess which clues mattered based on simple rules. But now, we have super-smart, "black box" computers (like neural networks) that can find complex patterns humans can't see. The problem is, these black boxes don't tell us why they made a decision. So, statisticians invented "wrapper" methods—tools that wrap around these black boxes to test each clue one by one. They do this by asking: "If I remove this clue, does the computer get worse at solving the mystery?" If the answer is yes, the clue is important. The big question researchers have been asking is: Which wrapper method is the best detective? Is the one that takes a quick peek better, or the one that does a slow, thorough investigation?

This paper sets out to answer that question by comparing two top-tier detective methods: LOCO (Leave-One-Covariate-Out) and GCM (Generalized Covariance Measure). Think of LOCO as the detective who takes a suspect out of the lineup, re-runs the entire investigation from scratch with the remaining suspects, and sees if the case falls apart. It's thorough but incredibly slow and exhausting. GCM, on the other hand, is like a detective who looks at the "leftover" clues after accounting for everything else, checking if the suspect still has a hidden connection to the crime without needing to restart the whole case.

The authors of this paper built a mathematical "scorecard" to measure how efficiently these two detectives work. They didn't just guess; they ran simulations with thousands of fake datasets and tested them on real-world problems, like predicting Airbnb prices and social media addiction. Their main finding is that GCM is generally the more efficient detective. In many scenarios, especially when the clues are related in complex, non-linear ways, GCM finds the important variables more accurately and with less "noise" (statistical variability) than LOCO.

However, the paper also points out a specific weakness in GCM: if the relationship between a clue and the outcome is perfectly symmetrical (like a mirror image) and the data is balanced, GCM might miss it entirely, thinking the clue is useless when it's actually vital. LOCO doesn't have this blind spot. Despite this, the simulations showed that GCM usually wins, identifying the right features more often and leading to better predictions, even though it requires a bit more computing power to run. The researchers also compared these methods to newer, faster shortcuts (like "Dropout" and "Lazy-VI"), finding that while the shortcuts are speedy, they sometimes miss the mark compared to the thoroughness of GCM. Ultimately, the paper suggests that if you want the most reliable results and can afford the extra computing time, GCM is currently the superior tool for uncovering the truth in complex data.

Technical Summary: Comparing Model-agnostic Feature Selection Methods through Relative Efficiency

Problem Statement

Feature selection and variable importance estimation in model-agnostic settings remain significant challenges in statistics and machine learning. While "black-box" prediction methods (e.g., neural networks, random forests) are increasingly popular, traditional embedded methods (like Lasso) rely on parametric assumptions. Consequently, wrapper methods that are model-agnostic have become the standard for feature selection. However, there is a lack of quantitative theoretical comparison between state-of-the-art wrapper methods, specifically regarding their statistical efficiency. Existing literature establishes asymptotic normality for individual methods but does not provide a unified framework to compare their performance in distinguishing informative features from noise.

Methodology

The authors develop a general comparison framework based on Asymptotic Relative Efficiency (ARE). To address the issue that different feature importance estimators may operate on different scales (having different means), the paper utilizes Relative Variability ( $R(T)$ ), defined as the ratio of the standard deviation to the expected importance under the alternative hypothesis ( $H_1$ ), relative to the null hypothesis ( $H_0$ ).

The core metric is the Asymptotic Relative Efficiency (ARE), defined as:
$e_{T_1, T_2} := \lim_{n \to \infty} \frac{\text{Var}(T_2)/E^2[T_2]}{\text{Var}(T_1)/E^2[T_1]}$
where $T_1$ and $T_2$ are two estimators. A value greater than 1 indicates that $T_1$ is more efficient than $T_2$ .

The paper focuses on comparing two primary model-agnostic methods:

Generalized Covariance Measure (GCM): A regression-based conditional independence test that measures the normalized covariance between residuals of $Y$ and $X_j$ given $X_{-j}$ .
Leave-One-Covariate-Out (LOCO): A method that measures the change in predictive accuracy (specifically, the reduction in negative mean-squared error) when a feature $X_j$ is removed and the model is retrained.

Theoretical comparisons are derived under three specific model settings:

Linear Models: $Y = X^T\beta + \epsilon$ .
Non-linear Additive Models: $Y = \sum f_j(X_j) + \epsilon$ .
Single Index Models: $Y = \eta(X^T\beta) + \epsilon$ , which mimic single-layer neural networks.

The analysis also extends to computationally efficient approximations: Dropout (replacing a feature with its marginal mean) and Lazy-VI (approximating reduced models via linearized ridge regression for neural networks).

Key Contributions

Theoretical Comparison Framework: The paper provides the first direct asymptotic relative efficiency comparison between GCM and LOCO. It proves that under suitable regularity conditions, GCM is asymptotically more efficient than LOCO for linear, additive, and single-index models.
Efficiency Conditions: The authors identify a specific "adjusted correlation" quantity that determines the efficiency gap. Specifically, for non-linear additive models, GCM outperforms LOCO if the correlation between the residualized feature $\tilde{X}_j$ and the residualized function $\tilde{f}_j(X_j)$ is sufficiently strong. The paper notes that GCM may fail when the functional relationship is an even function and the feature distribution is symmetric (e.g., $Y=X^2$ with $X \sim N(0,1)$ ), as the covariance becomes zero despite dependence.
Extension to Approximations: The study compares the asymptotic variance of LOCO against the computationally cheaper Dropout and Lazy-VI estimators, establishing variance relationships under regularity conditions.
Empirical Validation: The theoretical findings are validated through extensive simulations (linear, non-linear additive, interaction, and single-index models) and real-world data analysis (Airbnb pricing and social media addiction), utilizing diverse learners including Gradient Boosted Machines (GBM), Kernel SVMs, and Neural Networks.

Results

Theoretical Findings:

Linear Models: GCM is strictly more efficient than LOCO. The efficiency ratio depends on the signal-to-noise ratio ( $\beta_j^2/\sigma^2$ ) and the variance of the residualized feature.
Non-linear Additive Models: GCM is asymptotically more efficient than LOCO provided an adjusted correlation condition (Assumption A) holds. If the correlation is zero (e.g., even functions with symmetric inputs), GCM loses efficiency.
Single Index Models: Under assumptions regarding the smoothness of the link function $\eta$ and the sub-Gaussian nature of predictors, GCM demonstrates superior or comparable efficiency to LOCO.
Approximations: The variance of LOCO and Lazy-VI are shown to be asymptotically equivalent and generally lower than that of the Dropout method.

Empirical Findings:

Accuracy and Stability: In simulations, GCM consistently achieves higher accuracy, F1 scores, and recall with lower False Discovery Rates (FDR) compared to LOCO, Dropout, and Permutation tests, particularly in non-linear settings where the theoretical conditions are met.
Robustness: Under model misspecification (e.g., heavy-tailed covariates or unmodeled interactions), GCM remains more stable than LOCO, which is more sensitive to correlation structures and tail behavior.
Real Data: In analyses of Airbnb pricing and social media addiction data, GCM-selected models consistently yielded lower Mean Squared Error (MSE) across Linear, GBM, and Neural Network learners compared to LOCO-selected models. GCM also demonstrated higher selection stability across cross-validation folds.
Computational Cost: The paper acknowledges that GCM is computationally intensive, requiring the training of $2p$ models (for $p$ features), whereas LOCO requires $p+1$ , and Dropout/Lazy-VI require significantly fewer. Despite this cost, GCM's statistical gains are evident in the results.

Significance and Claims

The paper claims that GCM-related methods generally outperform LOCO in terms of statistical efficiency for feature selection under suitable regularity conditions defined by a specific correlation quantity. This superiority is demonstrated both theoretically through asymptotic relative efficiency and empirically through simulations and real-world applications.

The authors emphasize that while GCM is computationally more demanding, its ability to provide more accurate and stable variable importance scores makes it a superior choice for model-agnostic feature selection in linear, additive, and single-index settings. The work fills a gap in the literature by providing a quantitative comparison between these two prominent wrapper methods, moving beyond individual asymptotic normality results to a direct efficiency comparison. The paper concludes that future work could focus on developing novel GCM-based test statistics that improve computational efficiency or balance Type II error rates.