📊 statistics

Aggregate Models, Not Explanations: Improving Feature Importance Estimation

This paper demonstrates through theoretical analysis and empirical validation that aggregating feature importance estimates across individual models within an ensemble yields more accurate results than explaining the ensemble as a whole, particularly for expressive models where reducing excess risk is critical for reliable scientific discovery.

Original authors: Joseph Paillard, Angel Reyero Lobo, Denis A. Engemann, Bertrand Thirion

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joseph Paillard, Angel Reyero Lobo, Denis A. Engemann, Bertrand Thirion

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which ingredients in a giant, complex soup are responsible for its specific flavor. You have a team of expert chefs (machine learning models) who can predict the taste perfectly. However, because the kitchen is chaotic (data sampling) and the chefs have different moods or starting points (algorithmic randomness), each chef gives you a slightly different list of "important ingredients." Sometimes Chef A says "salt" is key, while Chef B insists it's "pepper," even though they both agree the soup tastes great.

This paper, titled "Aggregate Models, Not Explanations," tackles the problem of how to get the most reliable list of important ingredients when your chefs are inconsistent.

Here is the breakdown of their findings using simple analogies:

The Problem: The "Rashomon" Effect

In the movie Rashomon, different witnesses tell conflicting stories about the same event. In machine learning, this is called the Rashomon effect. You can have many different models that are all equally good at predicting the outcome (the soup tastes great), but they all blame different ingredients for the flavor.

When scientists try to use these models to discover why something happens (like finding a cure for a disease), this inconsistency is dangerous. If you invest money to test "pepper" as a cure based on one chef's list, but another chef said "salt," you might waste resources.

The Two Strategies: Two Ways to Ask the Chefs

The paper compares two ways to handle this chaos:

The "Sub-Models" Approach (Asking the Chefs Individually):
You ask Chef A, "What is the most important ingredient?" Then you ask Chef B, "What is the most important ingredient?" Finally, you take all their answers and average them out.
- The Flaw: If every chef is slightly biased (e.g., they all slightly overestimate the importance of salt because of how they were trained), averaging their answers won't fix that bias. You just get a "consensus" of a wrong answer.
The "Model-Level Ensemble" Approach (Creating a Super-Chef):
Instead of asking individual chefs, you have all the chefs work together while cooking. They combine their predictions in real-time to create one "Super-Chef" (an ensemble model) that is more accurate than any single chef. Then, you ask this Super-Chef: "What is the most important ingredient?"
- The Advantage: Because the Super-Chef is more accurate at predicting the taste, the explanation it gives is also more accurate.

The Big Discovery: "Fix the Prediction, Fix the Explanation"

The authors ran a theoretical analysis and found a crucial rule: For many complex models, the biggest source of error in figuring out "what matters" comes from the model's own prediction errors.

Think of it like this: If a chef is bad at cooking (high error), their explanation of why the food tastes good will also be bad.

The Sub-Models approach tries to smooth out the noise (variance) by averaging, but it doesn't fix the fundamental cooking errors (bias).
The Model-Level Ensemble approach actually improves the cooking (reduces the prediction error). Because the Super-Chef is a better cook, their explanation of the ingredients is naturally more trustworthy.

The paper shows that for popular methods like LOCO (removing one ingredient to see what happens) and SAGE (a complex game-theory method), building a Super-Chef first is almost always better than averaging the opinions of individual chefs.

When Does This Matter?

The paper tested this on:

Synthetic Data: Mathematical puzzles where the "true" ingredients were known. The Super-Chef approach consistently found the right ingredients more often and with less confusion.
Real-World Data (UK Biobank): They looked at a massive dataset of 46,000 people and 2,922 proteins to predict Body Mass Index (BMI).
- They found that the Super-Chef approach identified known biological markers (like Leptin and FABP4) much more clearly.
- The individual-chef approach (Sub-Models) produced results that were so shaky (high error bars) that they often couldn't distinguish real signals from noise.

The Exception: The "Linear" Methods

The paper notes that this rule isn't universal. For some specific methods like CFI (Conditional Feature Importance) and PFI (Permutation Feature Importance), the math is different. These methods are like linear equations where the "cooking error" cancels itself out. For these specific tools, averaging the individual chefs' answers works just as well as making a Super-Chef. However, for the more complex, non-linear methods used in modern AI, building the Super-Chef is the winner.

Summary

If you want to understand why a complex AI model makes a decision, don't just average the opinions of many different models. Instead, combine the models first to make a single, more accurate predictor, and then ask that single predictor for its explanation. This reduces the "noise" and "bias" that usually make scientific discovery with AI so difficult.

Title: Aggregate Models, Not Explanations: Improving Feature Importance Estimation

Problem Statement
Feature importance methods are increasingly utilized to transform machine learning (ML) models from predictive engines into tools for scientific discovery, particularly in high-stakes biomedical applications. However, expressive models (e.g., deep neural networks, random forests) often exhibit instability due to data sampling variations and algorithmic stochasticity (e.g., random initialization). This phenomenon, known as the "Rashomon effect," results in models with similar predictive performance providing conflicting explanations of the same data.

A critical practical question arises when addressing this instability via ensembling: Should one aggregate the importance scores derived from individual sub-models (the "sub-models" strategy), or should one aggregate the predictions of individual models to form a single ensemble predictor and then derive importance from this aggregate (the "ensemble" strategy)? Due to the nonlinearity of common importance measures (e.g., risk differences), these two approaches are not mathematically equivalent. Prior literature has largely focused on aggregating individual explanations without a formal analysis of the resulting estimation error, leaving the optimal strategy unclear.

Methodology and Theoretical Framework
The authors develop a theoretical analysis to decompose the error in feature importance estimation, moving beyond previous assumptions that required extremely fast convergence rates for ML models (e.g., $o_P(n^{-1/4})$ ), which are often unmet by modern overparameterized models.

Error Decomposition: Building on the framework of Bottou & Bousquet (2007), the authors decompose the excess risk ( $E$ ) of a model into approximation error ( $E_{app}$ ), estimation error ( $E_{est}$ ), and optimization error ( $E_{opt}$ ). They argue that for modern expressive models, the excess risk is the dominant source of error in feature importance estimation.
Risk-Difference Measures: The analysis focuses on methods that estimate importance via risk differences, specifically:
- LOCO (Leave-One-Covariate-Out): Compares the risk of a full model against a model excluding a specific feature.
- SAGE (Shapley Additive Global importancE): Aggregates risk differences over all possible feature subsets.
- CFI (Conditional Feature Importance) & PFI (Permutation Feature Importance): Methods where the same model is evaluated on original and perturbed inputs.
Theoretical Comparison:
- For LOCO and SAGE, the estimation error depends quadratically on the model's excess risk. The authors prove that model-level ensembling reduces this excess risk (via Jensen's inequality for convex losses), thereby reducing the leading error term (bias). In contrast, aggregating individual importance scores (sub-models) only reduces the stochastic remainder (variance) but leaves the bias (excess risk) unchanged.
- For CFI and PFI, the same model is applied to both terms of the risk difference, causing the quadratic excess risk terms to cancel out. The remaining error depends linearly on the model estimation error. Consequently, the theoretical advantage of model-level ensembling is less pronounced for these methods compared to LOCO and SAGE.

Key Contributions

Theoretical Analysis: The paper provides a rigorous error decomposition for feature importance estimation under relaxed assumptions suitable for modern ML models. It formally identifies the model's excess risk as the primary driver of estimation error for risk-difference methods.
Strategy Comparison: The authors demonstrate that aggregating models at the prediction level (model-level ensembling) is theoretically superior to aggregating individual explanations (sub-models) for methods like LOCO and SAGE, particularly in bias-dominant regimes where convergence is slow.
Empirical Validation: The findings are validated across classical synthetic benchmarks (Friedman 1, G-function, Ishigami) and a large-scale real-world proteomic study from the UK Biobank.

Results

Synthetic Benchmarks: Experiments on Friedman 1, G-function, and Ishigami datasets using Multi-Layer Perceptrons (MLP) and Random Forests (RF) show that model-level ensembling consistently yields lower Mean Squared Error (MSE) in importance estimation and higher ROC AUC for feature selection compared to the sub-models strategy.
Bias-Variance Decomposition: Analysis reveals that while the sub-models strategy effectively reduces variance, it fails to mitigate the significant bias introduced by the excess risk of individual models. Model-level ensembling directly addresses this bias.
Method-Specific Gains: The performance gap between the two strategies is most significant for LOCO and SAGE. For CFI and PFI, the differences are negligible, aligning with the theoretical prediction that the quadratic risk terms cancel out in these methods.
UK Biobank Application: In a study predicting Body Mass Index (BMI) from 2,922 plasma proteins, model-level ensembling identified biologically relevant proteomic signatures (e.g., FABP4, LEP, ADM, IGFBP-1, IGFBP-2) with greater precision and lower variance than the sub-models approach. The sub-models approach showed importance estimates for key proteins that overlapped with zero, whereas the ensemble strategy provided clear, significant rankings.
Stability: The ensemble strategy produced more stable importance rankings across cross-validation folds, as measured by Spearman's rank correlation.

Significance and Claims
The paper claims that the choice of aggregation strategy is not merely a heuristic but is driven by the model's excess risk. By shifting the focus from aggregating explanations to aggregating models, researchers can significantly improve the reliability of feature importance estimates for expressive architectures. This is particularly crucial for scientific discovery in fields like biomedicine, where unstable explanations can undermine the identification of therapeutic targets or risk factors.

The authors emphasize that their findings are specific to methods estimating well-defined theoretical indices (like LOCO and SAGE) and do not necessarily apply to heuristic methods lacking population-level estimands. They conclude that for risk-difference measures, the "ensemble" strategy is the preferred approach to mitigate the Rashomon effect and improve the robustness of model-agnostic feature importance estimation.