Detection of Multiple Influential Observations on Model Selection

1. Problem Statement

In high-dimensional statistical settings where the number of predictors ($p$) exceeds the sample size ($n$), standard model selection techniques (e.g., LASSO, SCAD, MCP) are susceptible to distortion by influential observations (outliers).

• The Challenge: Existing methods for detecting influential points often focus on deterministic submodels or single outliers. They struggle with stochastic model selection (where the selected submodel varies based on data) and multiple outliers simultaneously.
• Limitations of Prior Work: Previous methods like the High-Dimensional Influence Measure (HIM) or the Generalized Difference in Model Selection (GDF) metric lacked a rigorous theoretical foundation for their distributional properties. Specifically, the asymptotic distribution of the GDF metric was unknown, and there were no principled methods to derive accurate diagnostic thresholds for outlier identification in finite samples.
• Consequence: Without accurate detection, outliers can lead to the selection of incorrect submodels, compromising the generalizability and reproducibility of statistical inferences, particularly in complex fields like neuroimaging.

2. Methodology

The authors propose a comprehensive framework to detect multiple influential observations by revisiting the concept of exchangeability and extending the ClusMIP (Clustering-based Multiple Influential Points) algorithm.

A. Theoretical Foundation: Exchangeability and Asymptotic Distribution

• GDF Metric: The paper utilizes the Generalized Difference in Model Selection metric ($\tau_i$), which measures the change in the set of selected variables when the $i$-th observation is removed.
• Theorem 1 (Distributional Result): The authors prove that as $p \to \infty$, the GDF metric $\tau_i$ (and the specific LASSO version $\delta_i$) follows a finite mixture of binomial distributions. This is derived using de Finetti's Representation Theorem (DFRT), characterizing the sequence of selection indicators as an exchangeable sequence of Bernoulli trials conditioned on a latent variable.
• Theorem 2 (Parametric Consistency): They establish that Maximum Likelihood Estimation (MLE) remains consistent for approximating the distribution of $\tau_i$ even when observations are dependent (non-independent), provided the data is exchangeable. This justifies fitting parametric families to the diagnostic metrics.

B. Approximation Strategies

To derive diagnostic thresholds for outlier detection, the paper introduces three distinct approaches to approximate the distribution of $\tau_i$:

1. Parametric Approaches: Based on the mixture-of-binomials theory, the authors fit six parametric families to the data:
   • Conway-Maxwell-Binomial (CMB)
   • Conway-Maxwell-Poisson (CMP)
   • Beta-Binomial (BB)
   • Generalized Poisson (GP)
   • Mixtures of Binomial (MB) and Poisson (MP).
   • Note: These models account for potential over-dispersion caused by the dependence structure of outlier effects.
2. Nonparametric Approaches: Three bootstrap schemes are employed to estimate thresholds without assuming a specific distribution:
   • Method I: Standard bootstrap on $\tau[n]$ to infer conditional averages (based on DFRT extensions).
   • Method II & III: $m$-out-of-$n$ bootstrap schemes to consistently approximate sample and mid-quantiles.
3. Mid-Quantiles: To handle the discreteness of the count-based metric $\tau_i$, the authors utilize mid-quantiles (interpolating probability masses) to generate stable, asymptotically reliable thresholds.

C. The Revised ClusMIP Algorithm

The framework integrates these approximation methods into the ClusMIP procedure:

1. Clustering: Data is partitioned into potentially influential and clean subsets using a high-dimensional clustering scheme.
2. Assessment: Each candidate point is assessed using the merged sample (itself + clean data) to determine if it is the sole outlier.
3. Extension: The method is extended to Logistic Regression models, adapting the clustering and assessment steps for binary outcomes.
4. Implementation: A new R package, ClusMIP, implements these ten new approximation options (6 parametric + 3 nonparametric + CLT baseline).

3. Key Contributions

1. Theoretical Advancement: Proved that GDF/DF(LASSO) metrics asymptotically follow mixtures of binomial distributions, providing the first rigorous distributional justification for these high-dimensional diagnostics.
2. Methodological Expansion: Developed a flexible toolbox of 10 approximation methods (parametric and nonparametric) to derive diagnostic thresholds, overcoming the limitations of previous CLT-only approaches.
3. Model Generalization: Successfully extended the detection framework from linear regression to logistic regression and demonstrated its applicability to zero-inflated Poisson models.
4. Software Release: Provided an open-source R package (ClusMIP) implementing the revised algorithm, making these advanced diagnostics accessible to practitioners.

4. Results

Simulation Studies

• Detection Power: The revised ClusMIP significantly outperformed existing methods (HIM, MIP, DF(LASSO)).
  • ClusMIP(SLASSO) achieved near-perfect detection power (~1.00) across various perturbation schemes.
  • ClusMIP(LASSO) maintained power > 0.90.
  • DF(LASSO) consistently underperformed (power < 0.25).
  • MIP performed well for response contamination but failed when contamination was confined to predictors.
• Approximation Performance: Among the approximation methods, Boot-I (nonparametric bootstrap) consistently yielded the highest detection power and robustness.
• Model Performance: Removing detected outliers resulted in substantial reductions in Mean Squared Error (MSE) for linear models (up to 99.4% reduction) and improved classification accuracy for logistic models.
• Computational Cost: ClusMIP is computationally more intensive (due to leave-one-out refitting) than existing methods, but the trade-off is justified by superior detection accuracy.

Real Data Application: fMRI Pain Prediction

• Dataset: Task-based fMRI data from 33 participants predicting physical pain ratings from thermal stimuli.
• Findings:
  • The method identified two influential observations ({61, 169}) that were missed in prior studies. These outliers corresponded to low-temperature stimuli where pain responses were inconsistent.
  • Predictive Improvement: Removing these outliers improved prediction accuracy by at least 10% for both linear and logistic models.
  • Biological Plausibility: The brain regions selected after outlier removal were more scientifically plausible (e.g., retaining somatosensory and pain-processing areas while removing regions near the brain boundary driven by motion artifacts).

5. Significance

This work bridges a critical gap in high-dimensional statistics by providing a theoretically grounded, distribution-free, and flexible framework for detecting multiple influential observations.

• Reproducibility: By offering a robust mechanism to identify and remove data points that distort model selection, the method enhances the reproducibility of downstream analyses in fields like genomics and neuroimaging.
• Practical Utility: The introduction of the ClusMIP package allows researchers to move beyond "black box" outlier detection, offering a choice of approximations based on their specific needs for interpretability (parametric) or robustness (nonparametric).
• Scientific Insight: The application to fMRI data demonstrates that influential observations are not just statistical nuisances but can fundamentally alter the understanding of brain-behavior relationships, specifically in pain perception studies.

In summary, the paper transforms the detection of influential observations from an ad-hoc process into a rigorous, theoretically supported statistical procedure, significantly improving model selection reliability in high-dimensional settings.