Post-reduction inference for confidence sets of models

1. Problem Statement

The paper addresses the challenge of statistical inference in high-dimensional regression settings (e.g., genomics) where the number of candidate variables ($p$) far exceeds the sample size ($n$).

• The Goal: Instead of selecting a single "best" model (which is often misleading for scientific understanding due to model uncertainty), the authors advocate for reporting a confidence set of models ($\mathcal{M}$). This set contains all sparse models compatible with the data at a chosen significance level.
• The Challenge (Post-Reduction Inference): In high dimensions, constructing $\mathcal{M}$ requires a preliminary reduction phase to select a manageable encompassing model $\hat{E}$ (where $E^* \subseteq \hat{E} \subset [p]$).
• The Core Difficulty: Standard likelihood-ratio tests (LRT) or F-tests applied to submodels within $\hat{E}$ suffer from miscalibration. Because $\hat{E}$ is selected based on the same data used for testing, it fits the data better than an arbitrary model of the same size. Consequently, standard tests reject the null hypothesis too frequently, leading to confidence sets with coverage probabilities far below the nominal level (e.g., much lower than 95%).
• Limitations of Existing Solutions:
  • Sample Splitting: Using one part of the data for reduction and another for testing restores validity but discards information, reducing power, especially in small samples.
  • Post-Selection Inference: Existing literature (e.g., Lee et al., 2016) focuses on inference for a data-dependent parameter. Here, the object of inference (the true model $E^*$) is not data-dependent; the reduction phase merely focuses the analysis.
  • Monte Carlo Methods: Exact conditional inference via MCMC is computationally prohibitive for the combinatorial problem of testing all low-dimensional subsets.

2. Methodology

The authors propose a framework based on Fisherian conditional inference, utilizing two fundamental separations: Sufficiency/Co-sufficiency and Ancillarity.

A. Theoretical Framework

1. Co-sufficient Manifold ($Q(s_0)$):

   • For a model with sufficient statistic $S$, the data $Y$ can be decomposed into $S$ (used for estimation) and a co-sufficient component $Q$ (used for model assessment).
   • Conditioned on $S=s_0$, the data lies on a manifold $Q(s_0)$. Under the correct model, the distribution of $Y$ on this manifold is uniform (or follows a known distribution).
   • Gaussian Case: For linear regression, $Q(s_0)$ corresponds to a unit hypersphere in the orthogonal complement of the design matrix. Under the null, the projection of $Y$ onto this sphere is uniformly distributed.

2. Ancillary Separation:

   • The residual sum of squares ($R^2$) acts as an ancillary statistic for the regression coefficients $\theta$ (its distribution depends on $\sigma^2$ but not $\theta$).
   • This allows for model checking by comparing the observed $R^2$ against its theoretical distribution (e.g., $\chi^2$), provided $\sigma^2$ is estimated consistently.

B. Operational Implementation (Gaussian Linear Model)

To make these theoretical separations actionable without sample splitting, the authors introduce a randomization-based co-sufficient information partitioning:

1. Pseudo-Replicates: Instead of discarding data, they generate $k$ pseudo-replicates of the response vector $Y$ ($\tilde{Y}^{(1)}, \dots, \tilde{Y}^{(k)}$) using a linear transformation involving the original data and independent Gaussian noise.

   • These replicates share the same mean structure but have inflated variance ($k\sigma^2$).
   • Crucially, they allow for the construction of $k$ independent realizations of the co-sufficient statistic $Q$ from a single dataset.

2. Variance Estimation:

   • To handle the unknown $\sigma^2$, the authors adapt the Refitted Cross-Validation (RCV) estimator (Fan et al., 2012).
   • They propose a Modified RCV (mRCV) estimator computed on a subset (e.g., 60%) of the data to minimize dependence between the variance estimate and the test statistics.

3. Testing Procedures:

   • Co-sufficient Test (Rayleigh Test): Tests the uniformity of the $k$ pseudo-replicate projections on the hypersphere. If the model is misspecified, the projections cluster (non-uniformity). The test statistic $R_n$ follows an asymptotic standard normal distribution.
   • Ancillary Test: Uses the statistic $\hat{V} = R^2 / \hat{\sigma}^2_{mrcv}$ and compares it to a $\chi^2$ distribution.

3. Key Contributions

1. Formalization of Post-Reduction Inference: The paper distinguishes post-reduction inference (where the target model is fixed) from post-selection inference (where the target is data-dependent), justifying the use of conditional inference methods that bypass the need for data splitting.
2. Randomization Strategy: Introduces a novel method to generate independent co-sufficient replicates from a single dataset, avoiding the information loss inherent in sample splitting.
3. Theoretical Guarantees: Proves that the proposed test statistics (specifically the Rayleigh test on pseudo-replicates) maintain correct asymptotic distributions ($N(0,1)$) even when $\sigma^2$ is estimated, provided the sample size is moderate and the number of replicates $k$ is chosen appropriately ($k \lesssim n^{1/2}$).
4. Extension to General Models: Discusses how these principles extend to canonical exponential families (e.g., logistic regression) and accelerated life models (e.g., log-normal), noting that while exact analytic forms may differ, the geometric intuition of the co-sufficient manifold remains valid.

4. Results (Numerical Simulations)

The authors conducted extensive simulations ($n=100, 120$; $p=400$) comparing their methods against:

• Naive F-tests (using same data for reduction and testing).
• Sample-splitting F-tests.
• Co-sufficient and Ancillary tests.

Key Findings:

• Naive F-test: Catastrophic failure. Coverage probabilities dropped to near 0% (e.g., 0.02) because the encompassing model $\hat{E}$ was overfitted.
• Sample Splitting: Restored coverage (e.g., ~0.94) but resulted in very large confidence sets (low power to reject false models) due to reduced sample size in the testing phase.
• Co-sufficient & Ancillary Tests:
  • Achieved coverage probabilities close to the nominal 0.95 level (e.g., 0.90–0.96).
  • Superiority in Small Samples: In low-information settings (small $n$, weak signal), the co-sufficient and ancillary methods produced significantly smaller confidence sets (higher precision) than sample splitting while maintaining valid coverage.
  • Robustness: The ancillary test was slightly more sensitive to variance estimation errors in very small samples, but the co-sufficient test remained robust.
  • Effect of $k$: Increasing the number of pseudo-replicates ($k$) generally improved power (smaller sets) but required careful calibration to avoid over-coverage in very small samples.

5. Significance and Implications

• Scientific Integrity: The paper provides a rigorous solution to the "Rashomon effect" in regression, where many models fit the data equally well. By reporting a confidence set rather than a single model, it honestly reflects model uncertainty.
• Efficiency: The proposed methods demonstrate that sample splitting is not always necessary to correct for data reuse. By leveraging the geometry of the co-sufficient manifold, one can utilize the full dataset for both reduction and assessment without sacrificing validity.
• Computational Feasibility: Unlike MCMC-based conditional inference, the proposed randomization and analytic approaches are computationally efficient, making them viable for high-dimensional problems involving thousands of variables.
• Practical Guidance: The authors show that in high-dimensional settings where reduction is difficult (e.g., high correlation between signal and noise), the co-sufficient approach offers a distinct advantage over traditional splitting methods, providing more precise inference with the same data.

In conclusion, the paper establishes a new paradigm for high-dimensional regression inference, moving away from single-model selection and naive splitting toward a geometric, conditional inference framework that maximizes information usage while rigorously controlling error rates.