Technical Summary: Towards Efficient Inference under Nonmonotone Missingness with General Imputation

1. Problem Statement

The paper addresses the challenge of parameter estimation and inference in the presence of nonmonotone missingness (also known as blockwise missingness). In this setting, different units observe arbitrary, non-nested subsets of variables (modalities), a scenario common in modern multi-omics studies, electronic health records, and large surveys.

Standard approaches face two primary limitations:

1. Complete-case analysis discards units with incomplete data, potentially wasting substantial information and reducing efficiency.
2. Direct imputation (substituting missing values) often fails to preserve valid uncertainty quantification or efficiency, particularly when imputation models are misspecified or biased.

The goal is to combine information across all observed patterns to achieve semiparametric efficiency while maintaining validity under general imputation functions (which may be misspecified) and under the Missing Completely at Random (MCAR) mechanism. The paper also explores extensions to Missing at Random (MAR).

2. Methodology

2.1 Theoretical Framework

The authors frame the problem within semiparametric theory. Let $Z$ be the full data vector and $\theta^*$ be the target parameter defined by a full-data estimating equation $E_P[\psi_F(Z, \theta^*)] = 0$. Under MCAR, the semiparametric efficient estimating function is given by $\psi_{opt} = L\{M^{-1}[\psi_F]\}$, where $M$ and $L$ are linear operators constructed from pattern-specific conditional expectations. However, the inverse operator $M^{-1}$ involves complex compositions of conditional expectations across patterns, rendering the efficient estimator generally intractable.

2.2 The RAY Decomposition

To overcome the intractability of $M^{-1}$, the authors introduce the Restricted ANOVA hierarchY (RAY).

• Functional Decomposition: RAY decomposes any function $f$ into a hierarchy of components $P_s(f)$ using an inclusion-exclusion principle restricted to the observed patterns $\mathcal{Q}$.
• Almost-Eigen Structure: Under MCAR, the operator $M$ acts on these components $P_s(f)$ as an "almost-eigen-operator": $M[P_s(f)] = \lambda_s P_s(f) + \text{Rem}_s(f)$, where $\lambda_s = P(R \supseteq s)$ is the probability that a unit observes at least the modalities in set $s$. The remainder term $\text{Rem}_s$ involves compositions of conditional expectations across non-nested patterns.
• Closed-Form Approximation: By ignoring the remainder terms (which vanish under specific structural conditions) and the cross-pattern compositions, the authors derive a tractable, closed-form approximation to $M^{-1}$. This yields the RAY estimating function:
  $$\psi(R, Z_R, \theta^*) = \frac{I(R=[p])}{\pi_{[p]}} \psi_F(Z, \theta^*) + \sum_{r \in \mathcal{Q}, r \neq [p]} \omega_r A_r[\psi_F(Z, \theta^*)]$$
  where $\omega_r$ are closed-form weights derived from the pattern probabilities.

2.3 Estimator Construction and Tuning

• General Imputation: The term $A_r[\psi_F]$ represents the conditional expectation of the estimating function given the observed pattern $r$. In practice, this is replaced by an imputation function $F_r(Z_r; \theta)$, which can be a predictive model (e.g., neural networks) or a Monte Carlo approximation. Crucially, under MCAR, the RAY weights are mean-zero and independent of the imputation functions, ensuring the estimator remains unbiased even if the imputation models are misspecified.
• Safeguard Tuning: To improve efficiency, the authors introduce tuning parameters $\alpha_r$ for each imputation function. The estimator solves a sample-calibrated equation where the weights are adjusted to minimize an estimated variance criterion.
• Adaptive RAY (aRAY): Recognizing that neither RAY nor existing Imputation-Powered Inference (IPI) methods uniformly dominate the other, the authors propose a broader class of estimators that includes both as special cases. The adaptive RAY (aRAY) estimator optimizes the tuning parameters within this class to achieve minimal asymptotic variance.

2.4 Extension to MAR

The paper investigates extending RAY to the MAR setting.

• Direct Extension: A direct analogue of the MCAR weights exists but generally depends on unobserved modalities, making it infeasible as an observed-data estimator.
• Projected Extension: By projecting the tail propensities onto observed modalities, a feasible estimator is constructed. However, this introduces a "projected tail propensity" nuisance. The authors show that estimating this nuisance contributes to the target equation at the first order. Regular inference is possible if the nuisance admits root-$N$ consistent parametric estimators, utilizing stacked estimation.

3. Key Contributions

1. Closed-Form Efficient Approximation: The paper provides a closed-form, computable approximation to the semiparametric efficient estimator for arbitrary nonmonotone missing patterns under MCAR, avoiding the need for iterative or nested regression required by the exact inverse operator.
2. Imputation-Robust Validity: The proposed RAY estimator is valid for arbitrary independent imputation functions, including misspecified or biased models. Validity is guaranteed by the mean-zero property of the RAY weights under MCAR.
3. Adaptive Efficiency: The introduction of the adaptive RAY (aRAY) estimator, which embeds RAY and IPI into a larger class, allows for the attainment of minimal asymptotic variance within that class.
4. Exactness Conditions and Efficiency Bounds:
   • The authors establish a necessary and sufficient condition for RAY to attain the efficiency lower bound: the per-pattern residuals must vanish.
   • They identify verifiable sufficient conditions for exactness, including: (i) intersection-closed observed patterns with independent modalities, (ii) monotone patterns under arbitrary dependence, and (iii) anchored patterns with conditionally independent non-core modalities.
   • When these conditions fail, the efficiency gap is bounded by two interpretable factors: a correlation factor (driven by dependence between private modalities of overlapping patterns) and a structural factor (driven by the absence of required pattern intersections).
5. MAR Extension Analysis: The paper delineates the theoretical obstacles to extending RAY to general MAR, identifying the specific role of projected tail propensities and the requirements for root-$N$ inference.

4. Results

4.1 Simulation Studies

The authors conducted four simulation experiments:

• Varying Sample Size: RAY and aRAY consistently reduced Root Mean Square Error (RMSE) and confidence interval width compared to Complete-Case (CC) and Prediction-Powered Inference (PPI++) estimators while maintaining nominal 95% coverage.
• Varying Imputation Quality: The estimators demonstrated robustness; as imputation quality degraded to pure noise, the tuned estimators automatically reverted to the performance of the CC estimator, illustrating the effectiveness of the safeguard tuning.
• Linear Regression: In low-dimensional regression settings, RAY, IPI, and aRAY achieved 6–10% RMSE reduction and 12–18% variance reduction relative to CC.
• Sensitivity Analysis: Under controlled MAR and MNAR departures, the MCAR-based estimators exhibited increased bias and undercoverage, with the degree of deterioration scaling with the strength of the missingness mechanism violation.

4.2 Application to Single-Cell Multi-Omics

The method was applied to estimate mean surface protein abundance (ADT) in single-cell data where different assays (DOGMA-seq, CITE-seq, ASAP-seq) provide different combinations of RNA, ATAC, and ADT measurements.

• Setup: Neural networks were pretrained to predict ADT from RNA and/or ATAC. These were used as imputation functions in the RAY framework on a benchmark dataset with randomized missingness patterns.
• Performance: The augmented estimators (RAY, IPI, aRAY) achieved lower absolute error than the complete-case estimator. The aRAY estimator was uniformly the most efficient, reducing confidence interval lengths by up to 35% for highly abundant proteins compared to the complete-case approach.

5. Significance and Claims

The paper claims to provide a practical and theoretically grounded solution for efficient inference in the increasingly common setting of blockwise missing data. Its significance lies in:

• Bridging Theory and Practice: It translates the abstract semiparametric efficiency bound into a closed-form, implementable estimator that does not require complex iterative algorithms.
• Robustness: It decouples the validity of the inference from the accuracy of the imputation model, a critical feature when using flexible machine learning models that may be misspecified.
• Efficiency Gains: It demonstrates that combining information across arbitrary missing patterns can yield substantial efficiency gains over complete-case analysis, even with imperfect imputation.
• Clarifying Limitations: The paper modestly acknowledges that the closed-form advantage is fundamentally an MCAR result. Under general MAR, the method requires specific parametric assumptions or orthogonalization techniques to achieve regular inference, a direction left for future work.

The authors emphasize that their structural results (exactness conditions) and efficiency bounds provide a clear understanding of when and why the method works, offering a diagnostic framework for practitioners dealing with complex missing data structures.