Deep learning with missing data

Technical Summary: Pattern Embedded Neural Networks for Nonparametric Regression with Missing Covariates

Problem Setting
The paper addresses the challenge of supervised deep learning in multivariate nonparametric regression when covariates contain missing values. Unlike prior work that often assumes Missing Completely At Random (MCAR) or relies on specific parametric missingness models, this study considers a general setting where the missingness mechanism is arbitrary and potentially dependent on the covariates (Missing Not At Random, MNAR). The goal is prediction: estimating the Bayes regression function $f^\star(z, \omega) = \mathbb{E}(Y | Z=z, \Omega=\omega)$, where $Z$ represents imputed covariates and $\Omega$ represents the revelation vector (observation indicators) denoting which entries were observed.

A central difficulty identified is that while revelation vectors contain critical predictive information regarding the structure of the missingness, naively concatenating them with imputed covariates as input to a neural network often leads to poor performance. This is due to the exponential growth of possible observation patterns ($2^d$) with dimension $d$, causing overfitting, and the misalignment between the Euclidean geometry of the hypercube $\{0,1\}^d$ and the actual predictive structure of the problem.

Methodology: Pattern Embedded Neural Networks (PENNs)
The authors propose Pattern Embedded Neural Networks (PENNs), a novel architecture designed to integrate observation patterns into deep learning frameworks without relying on specific imputation techniques. The PENN estimator takes the form:
$$\hat{f}(z, \omega) = f_3(f_1(z), f_2(\omega))$$
where:

1. $f_1$: A neural network acting as a feature map for the imputed covariates $z$.
2. $f_2$: A neural network acting as an embedding function for the revelation vectors $\omega$. This maps the high-dimensional, discrete observation patterns into a lower-dimensional continuous space where geometric proximity reflects similarity in predictive behavior.
3. $f_3$: A final neural network that combines the learned covariate features and the pattern embeddings to produce the prediction.

This approach allows the model to "borrow strength" across similar observation patterns, effectively compressing the information in the revelation vectors. The method is compatible with any existing imputation technique (e.g., mean imputation, MissForest, MICE) applied prior to training.

Theoretical Contributions
The paper provides a rigorous theoretical analysis of the PENN estimator, establishing finite-sample excess risk bounds without assuming independence between data generation and missingness mechanisms.

• Oracle Inequality: A general oracle inequality is derived, decomposing the excess risk into optimization, approximation, and estimation error terms. This holds for arbitrary neural network estimators and sub-Gaussian responses.
• Minimax Optimality: Under a "piecewise smoothness" assumption (Assumption 1), the Bayes regression function is partitioned into cells where the function behaves similarly. Within each cell, the function belongs to a compositional Hölder class.
  • Theorem 2 establishes an upper bound on the excess risk. The bound is a weighted average of minimax rates for each cell of the partition, scaled by the effective sample size of that cell. Crucially, the rate is achieved as if the partition were known in advance, up to a poly-logarithmic factor in the sample size.
  • Theorem 3 provides a matching minimax lower bound, confirming that the PENN estimator attains the optimal convergence rate for this problem setting.
• Novelty: The theory introduces the concept of F-separability for partitions of observation patterns, allowing the neural network to learn the grouping of patterns that share similar regression functions. This addresses the "curse of dimensionality" associated with the $2^d$ possible missingness patterns.

Empirical Results
Extensive numerical experiments were conducted on simulated, semi-synthetic, and real-world datasets:

• Simulated Data: Across four models covering MCAR and MNAR scenarios with varying correlations, PENNs consistently outperformed standard neural networks (trained on imputed data without pattern embedding), XGBoost, and Random Forests. The improvement was particularly dramatic in MNAR settings where standard methods struggled.
• Semi-Synthetic Data: Experiments on the CT relative location dataset (regression) and MNIST (classification) demonstrated that PENNs significantly reduced prediction error (measured by Proportion of Unexplained Variance and Misclassification Error) compared to baselines, especially under MNAR mechanisms.
• Real Data: On a credit score prediction dataset with inherent missingness, PENNs again showed consistent and notable improvements over standard neural networks and tree-based methods.
• Visualization: Learned embeddings of observation patterns revealed that the network successfully clustered patterns with similar predictive behaviors, validating the theoretical premise that the embedding captures relevant structural information.

Significance and Claims
The authors claim that this work is the first to provide minimax optimality guarantees for multivariate nonparametric regression with missing data under general missingness mechanisms. The significance of the work lies in:

1. Methodological Innovation: Introducing pattern embedding as a mechanism to handle missingness in deep learning, bridging the gap between imputation-based approaches and direct modeling of missingness indicators.
2. Theoretical Rigor: Proving that PENNs can achieve the optimal rate of convergence even when the missingness mechanism is complex and dependent, provided the regression function exhibits piecewise compositional smoothness.
3. Practical Utility: Demonstrating that the method is robust, works with standard imputation tools, and yields substantial empirical gains over existing state-of-the-art methods in both synthetic and real-world scenarios.

The paper concludes that PENNs offer a principled and effective solution for deep learning with missing covariates, overcoming the limitations of simply treating missingness indicators as additional raw features.