🤖 machine learning

Causally-Guided Diffusion for Stable Feature Selection

This paper introduces Causally-Guided Diffusion for Stable Feature Selection (CGDFS), a framework that leverages diffusion models and guided annealed Langevin sampling to perform stability-aware posterior inference, thereby selecting robust feature subsets that maintain high predictive performance across distribution shifts by prioritizing causal invariance over spurious correlations.

Original authors: Arun Vignesh Malarkkan, Xinyuan Wang, Kunpeng Liu, Denghui Zhang, Yanjie Fu

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arun Vignesh Malarkkan, Xinyuan Wang, Kunpeng Liu, Denghui Zhang, Yanjie Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect recipe for a soup that will taste delicious no matter where you serve it—whether in a snowy mountain cabin, a humid beach town, or a bustling city.

The Problem: The "Spurious" Ingredient
Most traditional cooking methods (standard feature selection) work like this: They taste the soup in the mountain cabin and say, "Ah! The pine needles make it taste amazing!" So, they decide pine needles are essential.

But when you take that same recipe to the beach, the pine needles taste terrible because the local water and ingredients are different. The "pine needle" flavor was only working because of a lucky coincidence (a spurious correlation) in the mountains, not because it's actually a good ingredient. In data science, this is called distribution shift. The model learned a trick that only works in one specific environment and fails everywhere else.

The Solution: CGDFS (The "Causal Diffusion" Chef)
The authors of this paper propose a new method called CGDFS (Causally-Guided Diffusion for Stable Feature Selection). Instead of just looking for the ingredient that tastes best right now, they want to find the ingredients that will taste good everywhere.

Here is how CGDFS works, using three simple metaphors:

1. The "Crowdsourced Recipe Book" (The Diffusion Prior)

Imagine you don't just guess which ingredients to pick. Instead, you ask 1,000 different expert chefs to write down their favorite soup recipes. Some are simple, some are complex, some use fish, some use vegetables.

What CGDFS does: It gathers a huge "pool" of candidate feature sets (like these 1,000 recipes).
The Magic: It uses a Diffusion Model (a type of AI that learns patterns by slowly adding and removing noise) to study these recipes. It learns the structure of a good soup. It realizes, "Oh, chefs who use carrots usually also use onions, but rarely use both chocolate and salt."
The Benefit: Instead of randomly guessing ingredients, the AI now has a "gut feeling" (a prior) about which combinations of ingredients are likely to work well together. It understands the relationships between features.

2. The "Traveling Taste Test" (The Stability Likelihood)

Now, you have a list of potential recipes. How do you pick the best one?

Old Way: You taste the soup in the mountain cabin. If it's good, you pick it.
CGDFS Way: You take the soup to the mountain, the beach, the desert, and the city. You taste it in all these different environments.
The Goal: You aren't just looking for the soup that tastes best in one place. You are looking for the soup that tastes consistently good everywhere. If a recipe tastes amazing in the mountains but terrible in the desert, CGDFS rejects it. It only keeps recipes that are stable across all environments.

3. The "Guided Hike" (Annealed Langevin Sampling)

Finding the perfect recipe is like trying to find the highest peak in a massive, foggy mountain range with billions of possible paths.

The Problem: If you just walk up the steepest hill you see (greedy optimization), you might get stuck on a small hill that looks like the top, but isn't.
The CGDFS Solution: Imagine a hiker who has two guides:
1. Guide A (The Diffusion Prior): "Hey, don't go that way! The terrain there is usually bad for soup. Stick to the valleys where good recipes usually live."
2. Guide B (The Stability Test): "But wait, if you go that way, the soup will taste bad in the desert. Turn back!"
The Process: The hiker takes small, careful steps, listening to both guides. They don't just pick one path; they explore many different paths, gathering a collection of "good" recipes. This is called sampling.

The Final Result: The "Consensus Menu"

Instead of picking just one recipe and saying, "This is the only truth," CGDFS looks at all the good recipes it found during the hike.

It asks: "How often did carrots appear in the top recipes?" (High frequency = Very important).
It asks: "How often did pine needles appear?" (Low frequency = Probably just a fluke).

By aggregating these results, CGDFS gives you a list of ingredients that are robust. Even if the environment changes (the weather changes, the data shifts), your soup (your AI model) will still taste great because you selected ingredients based on their true, stable value, not just a lucky coincidence.

Why is this a big deal?

Old AI: "I learned that 'Home Ownership' predicts 'Credit Risk' because in this specific city, rich people own homes." (Fails when you move to a new city).
CGDFS AI: "I learned that 'Income Level' predicts 'Credit Risk' because that relationship holds true in the city, the suburbs, and the rural areas." (Works everywhere).

In short, CGDFS stops AI from memorizing tricks and starts it learning the real rules of the game, making it much more reliable when the world changes.

1. Problem Statement

The Challenge: Traditional feature selection methods (e.g., LASSO, Mutual Information, Random Forests) optimize for predictive performance under a single training data distribution. Consequently, they often select spurious features that correlate with the target only due to specific environmental conditions. When these models are deployed in new environments with distribution shifts, their performance degrades significantly because the selected features lack causal invariance.

The Limitation of Existing Solutions:

Correlation-based methods: Fail to distinguish between stable causal features and unstable spurious correlations.
Invariant Risk Minimization (IRM) / Invariant Causal Prediction (ICP): While principled, these methods face two major hurdles:
1. Non-convex Optimization: Stability-driven objectives create highly non-convex landscapes where gradient-based methods get stuck in poor local optima.
2. Combinatorial Complexity: Feature selection is inherently discrete ( $2^p$ possible subsets), making standard continuous optimization techniques difficult to apply directly without losing structural dependencies.

The Goal: To develop a feature selection framework that identifies subsets of features which are not only predictive but also stable across multiple environments, while effectively handling the combinatorial nature of the search space and quantifying uncertainty.

2. Methodology: CGDFS Framework

The authors propose Causally-Guided Diffusion for Stable Feature Selection (CGDFS), which reframes feature selection as approximate posterior inference over feature subsets. Instead of finding a single optimal subset, CGDFS samples from a distribution of plausible subsets.

The framework consists of three core components:

A. Feature Representation and Diffusion Prior

Continuous Relaxation: Instead of optimizing a discrete binary vector $z \in \{0,1\}^p$ , CGDFS uses a continuous selection mask $s \in [0,1]^p$ . This allows for gradient-based inference.
Mask Pool Construction: A diverse pool of candidate masks is generated using various heuristics (random selection, LASSO, Mutual Information, Random Forest, etc.) to capture different structural biases.
Diffusion Model as a Learned Prior: A score-based diffusion model is trained on this mask pool.
- Forward Process: Adds Gaussian noise to valid masks.
- Reverse Process: The model learns to predict the noise, effectively learning the score function $\nabla_s \log p(s)$ .
- Role: This learned prior captures complex structural dependencies (redundancy, complementarity) among features, guiding the search toward plausible regions of the combinatorial space without explicit enumeration.

B. Stability-Based Likelihood

The likelihood function evaluates feature subsets based on two criteria across $E$ distinct environments:

Predictive Performance: The mean loss $\bar{\ell}(s)$ across environments.
Stability (Robustness): The variance of the loss $\text{Var}_e(\ell_e(s))$ across environments.

The objective function $U(s)$ is defined as:
$U(s) = \bar{\ell}(s) + \lambda_{var} \cdot \text{Var}_e(\ell_e(s))$
The likelihood is implicitly defined as $p(D|s) \propto \exp(-U(s))$ . This ensures that subsets with high variance (unstable across environments) are penalized, even if their average accuracy is high.

C. Guided Posterior Sampling (Annealed Langevin Dynamics)

To approximate the posterior $p(s|D) \propto p(D|s)p(s)$ , CGDFS uses Annealed Langevin Dynamics:
$s^{(k+1)} = s^{(k)} + \eta_k \left( \nabla_s \log p(s^{(k)}) + \beta \nabla_s \log p(D|s^{(k)}) \right) + \sqrt{2\eta_k}\epsilon^{(k)}$

Gradient Decomposition: The update combines the prior score (structural plausibility from the diffusion model) and the likelihood gradient (stability and performance from the environment-specific predictors).
Causal Guidance (Soft Bias): The framework incorporates Invariant Causal Prediction (ICP) scores not as hard constraints, but as a soft bias during the mask pool construction. Features with higher invariance scores are more likely to appear in the training pool for the diffusion model, subtly guiding the prior toward causal structures without enforcing rigid rules.
Sampling: Multiple independent chains are run to capture posterior uncertainty.

D. Discretization and Aggregation

Posterior Inclusion Frequencies: Instead of picking one sample, the method aggregates results from multiple posterior samples. For each feature $j$ , it calculates the frequency $\pi_j$ with which it appears in the top- $k$ selected features across all samples.
Final Selection: The final feature subset consists of the $k$ features with the highest inclusion frequencies, providing a stable and interpretable selection that reflects uncertainty.

3. Key Contributions

Probabilistic Formulation: Reframed feature selection under distribution shift as approximate posterior inference, moving away from deterministic point-estimates to a distribution over subsets.
Diffusion-Based Prior: Introduced a data-driven diffusion model to learn a structured prior over feature masks, enabling scalable exploration of the combinatorial space and capturing feature dependencies (redundancy/complementarity).
Stability-Aware Likelihood: Defined a likelihood that jointly optimizes for accuracy and cross-environment variance, operationalizing causal invariance as a soft inductive bias.
Guided Sampling: Developed a guided annealed Langevin sampling procedure that combines the learned prior with the stability objective, avoiding discrete optimization pitfalls.
Soft Causal Integration: Demonstrated how to integrate causal signals (ICP) as soft guidance within a probabilistic framework, making the system robust to imperfect causal assumptions.

4. Experimental Results

The authors evaluated CGDFS on 12 real-world tabular datasets (UCI and OpenML) covering both classification and regression tasks, with artificially constructed distribution shifts.

Predictive Performance (RQ1): CGDFS consistently achieved the highest F1-macro (classification) and lowest MSE (regression) compared to baselines (LASSO, MI, Random Forest, Stability Selection, Greedy Forward, and ICP).
Robustness and Stability (RQ2): CGDFS demonstrated significantly lower variance in performance across environments compared to other methods. It successfully balanced high accuracy with stability, whereas methods like ICP often reduced variance at the cost of average accuracy.
Ablation Study (RQ3):
- Removing Diffusion Prior: Led to substantial performance degradation, proving the prior is critical for navigating the combinatorial space.
- Replacing Sampling with Optimization (Grad-Stab): Resulted in higher variance and sensitivity to initialization, highlighting the benefit of sampling multiple high-probability solutions.
- Removing Causal Guidance (ICP): Performance remained competitive, confirming that the stability likelihood is the primary driver, while ICP acts as a helpful accelerator.
Uncertainty and Interpretability (RQ4): The posterior inclusion frequencies provided a quantifiable measure of feature uncertainty. CGDFS showed higher overlap in selected features across different random seeds compared to baselines, indicating greater reproducibility.

5. Significance and Impact

Robust AI: CGDFS addresses a critical gap in deploying AI in dynamic, real-world environments where data distributions shift. It provides a principled way to select features that generalize beyond the training set.
Beyond Determinism: By treating feature selection as a probabilistic inference problem, the method acknowledges the inherent uncertainty in high-dimensional data, avoiding the brittleness of selecting a single "best" subset.
Scalability: The use of diffusion models and continuous relaxation allows the method to scale to high-dimensional datasets where traditional combinatorial search or strict causal discovery is computationally infeasible.
Practical Applicability: The framework does not require interventional data or explicit causal graphs, making it applicable to observational data in fields like healthcare, finance, and algorithmic fairness where causal structures are often unknown.

In summary, CGDFS represents a significant advancement in stable machine learning by unifying causal invariance principles, diffusion generative models, and Bayesian inference to solve the feature selection problem under distribution shift.