Quantifying uncertainty and stability among highly correlated predictors: a subspace perspective

1. Problem Statement

The paper addresses the challenge of linear feature selection in settings where predictors are highly correlated (multicollinear). Existing methods face two fundamental limitations in these scenarios:

1. Discrete Similarity Measures: Traditional metrics for comparing models (e.g., the Jaccard index or simple set overlap $|S_1 \cap S_2|$) are "discrete." They treat two models as completely different if they share no features, even if the features are highly correlated and the models have nearly identical predictive power. This fails to capture the concept of feature interchangeability.
2. Instability and "Vote Splitting": Standard stability selection (Meinshausen & Bühlmann, 2010) assesses stability by counting how often a specific feature is selected across subsamples. In highly correlated settings, "vote splitting" occurs: if features $X_1$ and $X_2$ are nearly identical, a subsample might select $X_1$ while another selects $X_2$. Consequently, neither receives a high stability score, leading to the erroneous conclusion that both are unstable, or resulting in the selection of very small, underpowered models to avoid instability.

The core issue is that existing notions of similarity and stability are binary (discrete), whereas highly correlated data requires continuous measures that account for the geometry of the feature space.

2. Methodology: The Subspace Perspective

The authors propose a framework based on feature subspaces—the subspaces spanned by the columns of the feature matrix $X$ corresponding to a selected set of features.

A. Continuous Similarity Metrics

Instead of counting shared features, similarity is defined by the alignment between subspaces $\text{col}(X_{S_1})$ and $\text{col}(X_{S_2})$.

• Similarity ($\tau$): Defined as the trace of the product of projection matrices:
  $$\tau(S_1, S_2) := \text{trace}(P_{\text{col}(X_{S_1})} P_{\text{col}(X_{S_2})})$$
  This equals the sum of squared cosines of the principal angles between the subspaces. It varies smoothly with correlation; if features are perfectly correlated, the similarity approaches the size of the smaller set, even if the feature indices are disjoint.
• Variants:
  • Normalized Similarity ($\bar{\tau}$): Scales $\tau$ by model size.
  • Conservative Similarity ($\tilde{\tau}$): Based on the cosine squared of the largest principal angle. This measures the worst-case predictive discrepancy between models.

B. Continuous Stability Metrics

Stability is redefined not as the frequency of selecting a specific feature, but as the alignment of a feature's subspace with the average subspace generated by subsampling.

• Stability of a Set ($\pi(S)$):
  $$\pi(S) := \sigma_{|S|} \left( P_{\text{col}(X_S)} P_{\text{avg}} P_{\text{col}(X_S)} \right)$$
  where $P_{\text{avg}} = \frac{1}{B} \sum P_{\text{col}(X_{\hat{S}_\ell})}$ is the average projection matrix from $B$ subsamples, and $\sigma_{|S|}$ is the $|S|$-th singular value.
• Interpretation: A set $S$ is stable if its subspace is well-aligned with the "average" subspace found across subsamples. This naturally handles vote splitting: if $X_1$ and $X_2$ are correlated, the subspace spanned by $X_1$ will align well with a subspace containing $X_2$, yielding a high stability score for the concept of the feature, even if the specific index varies.

C. Algorithm: Feature Subspace Stability Selection (FSSS)

The authors propose FSSS, an algorithm to enumerate maximal $\alpha$-stable sets (sets where $\pi(S) \geq \alpha$).

• Search Strategy: FSSS performs a sequential search (similar to forward stepwise selection) but uses the subspace stability metric.
• Optimization: It employs search-space reduction techniques:
  • Monotonicity: If a set is unstable, any superset is also unstable.
  • Pre-evaluation: Uses a computationally cheaper proxy (trace of projection onto a single new direction) to reject candidates early.
  • Path Pruning: Avoids re-evaluating paths that lead to already explored maximal sets.
• Output: Unlike standard stability selection which outputs a single model, FSSS outputs a collection of stable models. This captures the "Rashomon effect"—the existence of multiple distinct feature sets that are equally valid and predictive due to correlation.

3. Key Contributions

1. Subspace Framework: Introduced a continuous, geometry-based perspective for defining similarity, true positives, false positives, and stability in feature selection. These metrics reduce to standard discrete definitions when features are orthogonal but provide nuanced assessments when correlated.
2. FSSS Algorithm: Developed an efficient algorithm to find multiple stable models, overcoming the "vote splitting" problem and the tendency of existing methods to select overly sparse, underpowered models.
3. Theoretical Guarantees:
   • Provided False Positive Error (FPE) bounds for models selected by FSSS, dependent on the quality of the base selection procedure.
   • Proved consistency: Under specific clustering assumptions (features grouped into highly correlated clusters with at most one signal per cluster), FSSS consistently recovers "equally good models" (sets containing exactly one feature from each signal cluster) with high probability.
4. Implementation: Released the substab R package, providing tools for computing these metrics and running FSSS.

4. Results

The paper validates the method through synthetic experiments and a real-world application.

• Synthetic Data:

  • Performance: FSSS significantly outperforms standard Stability Selection (SS) and Cluster Stability Selection (CSS).
  • Metrics: FSSS achieves lower Test Mean Squared Error (MSE), higher True Positives (TP), and maintains low False Positives (FP).
  • Stability: FSSS produces models with higher "Output Stability" (similarity of selected sets across runs) compared to SS, which often selects empty or unstable sets due to vote splitting.
  • Example: In one experiment, SS selected an empty model (MSE 17.38), while FSSS selected a size-12 model with MSE 0.32.

• Real Data (Breast Cancer Gene Expression):

  • Applied to the GSE2990 dataset ($n=189$, $p=1111$).
  • FSSS identified multiple stable models with better predictive performance than SS or CSS.
  • Interpretability: The method successfully identified groups of substitutable genes (e.g., CEP55 and SETD8 as part of different stable models) that are biologically relevant to breast cancer prognosis, demonstrating how the subspace perspective reveals feature interchangeability that discrete methods miss.

5. Significance

• Paradigm Shift: Moves variable selection from a discrete "feature counting" paradigm to a continuous "subspace geometry" paradigm. This is crucial for modern high-dimensional data where multicollinearity is ubiquitous.
• Handling Uncertainty: By outputting a set of stable models rather than a single "best" model, the method provides a more honest representation of uncertainty in the presence of correlated predictors.
• Robustness: The approach is robust to the "Rashomon effect," acknowledging that multiple different feature combinations can yield similar predictive performance.
• Theoretical Rigor: The paper bridges the gap between discrete model selection and continuous subspace selection, offering the first theoretical guarantees for stability selection in highly correlated settings that account for feature interchangeability.

In summary, this work provides a mathematically rigorous and practically effective solution to the long-standing problem of feature selection in the presence of high multicollinearity, offering tools to find stable, predictive, and interpretable models where previous methods fail.