📊 statistics

Correcting Variable Importance Scored by Random Forests

This paper proposes a method to correct Random Forest variable importance scores by grouping variables based on their conditional correlations to prevent correlated variables from being masked or undervalued.

Original authors: Guancheng Zhou, Haiping Xu, Jason Liu, Donghui Yan

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guancheng Zhou, Haiping Xu, Jason Liu, Donghui Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Shadow" Effect

Imagine you are a talent scout trying to find the best singer in a room full of people. You have a microphone (the Random Forest algorithm) that measures how much each person contributes to the group's overall sound.

Usually, the microphone works great. But imagine two singers, Alice and Bob. They are twins who sing the exact same song in perfect harmony. They are so similar that when you ask, "How much did Alice contribute?" the microphone gets confused. It thinks, "Well, Bob is right there singing the same thing, so I can't tell if the sound is coming from Alice or Bob."

Because of this confusion, the microphone gives Alice a very low score, even though she is a fantastic singer. It's as if Bob is casting a "shadow" over Alice, hiding her true talent. In the world of data, this happens when variables (like "Age" and "Years of Experience") are highly correlated. The standard method often underestimates the importance of one because the other is doing the same job.

The Solution: The "Silence" Test

The authors of this paper propose a new way to measure importance. Instead of just asking, "What happens if we mess up Alice's voice?" (which is what the old method does), they suggest a different approach: Remove the shadows.

They propose two main ways to do this:

Method 1: The "One-by-One" Detective

Imagine you want to know how important Alice is.

First, you identify everyone in the room who is singing the exact same song as Alice (her "conditionally correlated" friends).
You ask all those friends to leave the room and go silent.
Now, with only Alice left (and no one mimicking her), you ask her to sing.
You measure how much the group's sound improves just because Alice is there.

Because her "twin" Bob is gone, Alice's true value shines through. The paper calls this Method 1. It checks every single variable, finds its "twins," removes them, and sees how much the model (the prediction) drops without them.

Method 2: The "Group Hug" (Clustering)

This method is a bit more organized. Instead of looking at one person at a time, you look at the whole room and divide everyone into small, tight-knit groups based on who sounds like whom.

Group A: Alice, Bob, and Charlie (all sing the same song).
Group B: Dave and Eve (they sing a different song).
Group C: Frank (he sings alone).

To test Alice's importance, you don't just remove Bob; you remove the entire Group A. Then you see how much the sound drops. If the sound drops a lot, it means that whole group was crucial. Since no one outside Group A sounds like them, Alice's contribution is no longer hidden by her friends.

The paper calls this Method 2, and it uses a mathematical technique called "Spectral Clustering" to figure out who belongs in which group.

Why Not Just "Mess Up" the Voice?

You might ask, "Why not just scramble Alice's voice (permute it) instead of removing her?"

The authors explain that scrambling a voice works fine if you only do it to one person. But if you have to scramble three or four people at once (because they are all twins), the math gets messy. It's like trying to untangle a knot by pulling on multiple strings at once; the result is unpredictable.

Instead, the paper suggests simply removing the correlated variables entirely. It's cleaner, more stable, and gives a clearer picture of who actually matters.

What Did They Find?

The authors tested this idea on real-world data sets (like predicting heart disease, wine quality, and obesity levels).

The Result: In many cases, the standard method (Random Forest) was giving "zero" or very low scores to variables that doctors and experts knew were actually very important.
The Fix: By using their new methods, those "hidden" variables suddenly got high scores.
- Example: In a liver disease dataset, a specific enzyme marker was being ignored by the old method because it was correlated with another marker. The new method realized, "Hey, this enzyme is actually super important!" and gave it a high score.
- Example: In a heart disease dataset, "Age" and "Gender" were being underestimated. The new method corrected this, showing they are indeed critical factors.

The Bottom Line

The paper argues that when variables are "best friends" (highly correlated), the standard way of measuring importance often makes one of them look unimportant.

By removing the "best friends" before measuring, the authors show that we can see the true value of each variable. They offer two tools to do this:

Method 1: A flexible, detailed check for every single variable.
Method 2: A faster, grouped approach that clusters similar variables together.

Both methods help us stop the "shadows" from hiding the true stars of the data.

Technical Summary: Correcting Variable Importance Scored by Random Forests

Problem Statement

Random Forests (RF) are widely recognized as powerful tools for statistical data analysis and machine learning, offering robust performance in classification and regression tasks. A key feature of RF is its ability to output variable importance scores, which are essential for model interpretation, selection, and diagnosis. However, the standard calculation of variable importance in RF—specifically the permutation accuracy method—suffers from a critical limitation: it fails to account for correlations among variables.

When a variable $U$ is highly correlated with other variables ( $V_1, V_2, \dots$ ) conditional on the response variable $Y$ , the importance of $U$ is often underestimated or completely masked. In the standard permutation approach, shuffling the values of $U$ breaks its association with $Y$ . However, if $V_1, V_2, \dots$ remain in the model and are highly correlated with $U$ , they can effectively "fill in" the missing information, resulting in a negligible drop in predictive accuracy. Consequently, the calculated importance for $U$ remains artificially low, despite $U$ being a true predictor. This phenomenon leads to significant discrepancies between the calculated importance and the actual predictive utility of variables.

Methodology

To address the masking effect caused by correlated variables, the authors propose a framework that groups variables based on their conditional correlations (correlation given the response variable $Y$ ) before calculating importance. The core idea is to remove the influence of correlated variables during the importance assessment of a specific variable of interest.

The paper defines the conditional correlation coefficient $\rho(U, V | Y)$ as the ratio of the conditional covariance to the product of conditional standard deviations. Using this metric, the authors explore two distinct algorithmic approaches:

Method 1: Individual Variable Correction (corrVI-Individual)

This approach operates on a per-variable basis:

For a target variable $V_i$ , the algorithm identifies the set of variables $V_{cor}$ that are conditionally correlated with $V_i$ .
It constructs a reduced dataset containing all variables except those in $V_{cor}$ (denoted as $V_{nc}$ ).
The predictive accuracy of the Random Forest is calculated on the set $V_{nc}$ and then on the set $V_{nc} \cup \{V_i\}$ .
The importance of $V_i$ is defined as the increase in predictive accuracy when $V_i$ is added to the set of non-correlated variables.
By removing all correlated variables, the method ensures that the accuracy gain is attributed solely to $V_i$ , preventing masking.

Note: The authors acknowledge that this method removes "more" information than strictly necessary (as $V_{cor}$ may contain information beyond $V_i$ ), potentially leading to a slight overestimation of importance. However, they argue this overestimation is negligible if the remaining variables provide decent predictive separation.

Method 2: Spectral Clustering Correction (corrVI-Spectral)

This approach groups the entire set of variables into disjoint subgroups based on pairwise conditional correlations:

A pairwise conditional correlation matrix is computed for all variables.
This matrix is treated as a similarity matrix, and spectral clustering is applied to partition the variables into disjoint groups ( $G_1, \dots, G_K$ ) where variables within a group are highly correlated, and variables across groups are weakly correlated.
For a variable $V$ belonging to group $G_i$ , the algorithm removes the entire group $G_i$ from the dataset.
The importance of $V$ is calculated as the increase in predictive accuracy when $V$ is added back to the set of variables $V_{nc} = \text{All Variables} \setminus G_i$ .
Since no variable in $V_{nc}$ is conditionally correlated with $V$ , the importance score is not masked.

Feature Removal vs. Permutation:
The authors explicitly choose to remove correlated variables rather than permute them during the importance calculation. They demonstrate that while removing or permuting a single variable yields similar results due to RF's built-in variable selection, permuting multiple variables simultaneously can degrade predictive accuracy more than simple removal. This is because permuting multiple noise variables increases the probability that a noise variable is selected as a split point, artificially inflating the drop in accuracy. Therefore, removal is deemed the more stable approach for correcting multi-variable correlations.

Key Contributions

Identification of the Masking Problem: The paper formally highlights how conditional correlations in RF lead to the underestimation of variable importance, a problem often overlooked in standard implementations.
Conditional Correlation Framework: The authors introduce the use of conditional correlation (given $Y$ ) as the metric for grouping variables, rather than unconditional correlation.
Two Correction Algorithms: The proposal of two computationally efficient methods:
- Method 1: A flexible, individual-based approach that removes specific correlated variables for each target.
- Method 2: A clustering-based approach using spectral clustering to handle variable groups globally.
Empirical Validation: The methods are tested on eight real-world datasets from the UC Irvine Machine Learning Repository, demonstrating that the corrections align better with domain knowledge than standard RF importance scores.

Experimental Results

The authors evaluated both methods on datasets including Seeds, Indian Liver Patient, Heart Disease, Wine, Maternal Health, Obesity, and Bank Marketing.

Seeds Dataset: Original RF assigned near-zero importance to "Compactness" ( $V_3$ ) and low importance to "Area" ( $V_1$ ) and "Perimeter" ( $V_2$ ). Both correction methods significantly increased the importance of $V_1$ and $V_2$ . Method 1 correctly identified $V_3$ as having non-zero importance (as it contains unique information), whereas Method 2 (due to clustering limitations) slightly underestimated it.
Indian Liver Patient Dataset: Variables such as Alkaline Phosphatase ( $V_5$ ) and Total Protein ( $V_8$ ) were assigned near-zero importance by standard RF. Method 1 correctly adjusted these upwards, reflecting their medical significance. Method 2 failed to adjust $V_5$ because the clustering algorithm did not fully capture its correlation with other variables, masking its importance.
Heart Disease Dataset: Critical variables like Age ( $V_1$ ), Gender ( $V_2$ ), and Resting Blood Pressure ( $V_4$ ) were significantly under-ranked by standard RF. Both methods corrected these scores. However, Method 2 failed to adjust $V_4$ and $V_{11}$ (Slope of ST segment) as effectively as Method 1 due to clustering boundaries.
Obesity and Bank Marketing Datasets: Similar trends were observed where domain-critical variables (e.g., Family History of Overweight, Previous Contact History) were under-ranked by RF but corrected by the proposed methods. Method 1 generally proved more robust in cases where the correlation structure did not form perfect disjoint clusters.

In general, Method 1 demonstrated greater flexibility and accuracy in correcting importance scores, particularly when variables formed complex correlation structures that spectral clustering could not perfectly partition.

Significance and Claims

The paper claims that its approach provides "sensible corrections" to variable importance scores produced by Random Forests. The significance lies in:

Improved Interpretability: By mitigating the masking effect, the corrected scores better reflect the true predictive contribution of variables, aiding in model interpretation and feature selection.
Domain Alignment: The corrected importance scores align more closely with established domain knowledge (e.g., medical markers in liver and heart datasets) compared to standard RF scores.
General Applicability: While motivated by RF, the authors note that the approach of removing correlated variables to assess importance is applicable to general predictive methods.

The authors maintain a modest tone, acknowledging that Method 1 may slightly overestimate importance by removing more information than necessary, and that Method 2's performance depends on the quality of the spectral clustering. They do not claim to solve all issues of feature importance but offer a practical, computationally efficient mechanism to address the specific problem of correlation-induced masking.