Robust and Sparse Generalized Linear Models for High-Dimensional Data via Maximum Mean Discrepancy

Technical Summary: Robust and Sparse Generalized Linear Models for High-Dimensional Data via Maximum Mean Discrepancy

Problem Statement
High-dimensional datasets in fields such as genomics and finance are frequently contaminated by outliers and heavy-tailed noise. Standard regularized estimators, such as the Lasso (ℓ₁-penalized Maximum Likelihood Estimation), are highly sensitive to such contamination, leading to biased parameter estimates and poor variable selection. While robust methods like M-estimators (e.g., Huber loss) and Density Power Divergence (DPD) exist, they often face optimization challenges in non-convex landscapes or are tailored to specific distribution families. Furthermore, existing Maximum Mean Discrepancy (MMD) frameworks, which offer "universal" robustness by treating estimation as a minimum distance problem in a Reproducing Kernel Hilbert Space (RKHS), have primarily been designed for low-to-moderate dimensional settings. They lack the shrinkage mechanisms necessary to handle sparsity constraints in high-dimensional regimes where the number of predictors $p$ exceeds the sample size $n$ ($p \gg n$), often resulting in overfitting or identifiability failure.

Methodology
The authors propose a penalized MMD framework for Generalized Linear Models (GLMs) that integrates ℓ₁-penalties for variable selection with the robustness of MMD.

1. Objective Function: The core innovation is the formulation of a penalized MMD objective. The authors define two estimators:

   • Full Estimator ($\hat{\theta}_n$): An $O(n^2)$ version that minimizes the squared MMD distance between the empirical joint distribution of data and the model's joint distribution, plus an ℓ₁ penalty.
   • Approximate Estimator ($\tilde{\theta}_n$): A computationally efficient $O(n)$ approximation. This version assumes the input kernel $K_x$ decays rapidly, allowing the cross-terms ($i \neq j$) in the MMD expansion to be neglected. This reduces the complexity from quadratic to linear in $n$.
   • Both estimators minimize: $\sum \text{MMD terms} + \lambda \|\theta\|_1$.

2. Specific Formulations:

   • Gaussian Linear Regression: The authors derive simplified loss functions using Gaussian kernels for the response variable, leveraging the similarity between the kernel and the normal density.
   • Binary Logistic Regression: A geometric kernel is employed for the discrete response variable to construct the MMD distance.

3. Optimization Algorithm:

   • Since the resulting optimization problems are non-convex, the authors employ an algorithm combining the Alternating Direction Method of Multipliers (ADMM) with AdaGrad.
   • ADMM is used to decouple the non-smooth ℓ₁ penalty from the MMD loss function via an auxiliary variable and Lagrange multipliers.
   • AdaGrad is utilized to update the model parameters ($\theta$), adaptively scaling learning rates to handle the non-convexity of the MMD loss, particularly in high-dimensional settings.
   • The algorithm includes updates for the variance parameter ($\sigma^2$) in the linear regression case to maintain kernel scale consistency.

Key Contributions

• High-Dimensional Robust GLMs: The paper bridges the gap between MMD-based robust inference and high-dimensional statistics by introducing sparsity-inducing penalties to the MMD framework.
• Handling Leverage Points: Unlike traditional robust M-estimators that rely on coordinate-wise bounded influence functions (and often fail with high-leverage points), the MMD approach utilizes a joint-distribution perspective. This allows the estimator to simultaneously handle outliers in both the response ($Y$) and covariate ($X$) spaces.
• Computational Efficiency: The development of the $O(n)$ approximate estimator makes the method scalable for large datasets, a significant improvement over the $O(n^2)$ complexity of standard MMD approaches.
• Algorithmic Integration: The successful combination of ADMM and AdaGrad to solve non-convex, kernel-based regularized problems in high dimensions.

Results
Extensive simulation studies and real-data applications were conducted to evaluate the proposed methods against benchmarks like Lasso, Huber-Lasso, Loh's robust M-estimator, and enetLTS.

• Simulation (Gaussian Linear Regression):

  • Robustness: The proposed MMD estimators ($\hat{\theta}_n$ and $\tilde{\theta}_n$) demonstrated superior stability under heavy-tailed error distributions (Laplace, $t_5$) and high-leverage point contamination ($X_1, X_2$) compared to Lasso and Huber methods.
  • Variable Selection: The MMD methods consistently achieved lower False Selection Loss (FSL), indicating better ability to distinguish true signals from noise even when predictors were contaminated.
  • Prediction: The $O(n)$ approximate estimator performed nearly identically to the full $O(n^2)$ version, validating its use for computational efficiency without significant accuracy loss.

• Simulation (Binary Logistic Regression):

  • In high-dimensional settings ($p=100, 200$), standard Lasso and existing robust methods (enetLTS) showed significant degradation in Mean Squared Error (MSE) and misclassification rates under contamination.
  • The proposed MMD estimators maintained low MSE and misclassification error, particularly in scenarios involving flipped labels and corrupted active predictors.
  • The approximate estimator ($\tilde{\theta}_n$) often yielded the best predictive accuracy in classification tasks.

• Real Data Applications:

  • NCI-60 Cancer Data: The full MMD estimator ($\hat{\theta}_n$) achieved lower median Mean Square Prediction Error (MSPE) and higher stability than the sparseLTS benchmark.
  • Credit Card Default Data: The approximate estimator ($\tilde{\theta}_n$) provided a more robust classification boundary than standard logistic Lasso in the presence of noise and class imbalance, demonstrating efficiency in large-scale settings ($n=30,000$).

Significance and Claims
The paper claims that the proposed penalized MMD framework offers a "universal" robustness profile that is particularly effective in high-dimensional scenarios where traditional methods struggle. The authors emphasize that the joint-distribution perspective of MMD is crucial for handling leverage points, a common failure mode for coordinate-wise robust estimators.

The work establishes that:

1. MMD can be successfully extended to high-dimensional GLMs with variable selection.
2. The $O(n)$ approximation is a viable and efficient alternative to the full $O(n^2)$ computation.
3. The method maintains a balance between estimation accuracy and variable selection across diverse contamination scenarios, including heavy-tailed errors and high-leverage points.

The authors modestly note that future work should investigate the theoretical oracle properties of these estimators and explore automated bandwidth selection methods to further enhance applicability. They also suggest extending the framework to group-sparse penalties for multi-modal data integration.