Neural Parameter Estimation with Incomplete Data

1. Problem Statement

The paper addresses a critical bottleneck in Neural Bayes Estimation (NBE): the inability of standard neural network architectures to handle incomplete data (missing values) in a statistically principled and efficient manner.

• Context: NBEs use neural networks to map data directly to point estimates of parameters (approximating the posterior mean or mode). They offer "amortized inference," meaning once trained, they provide instant estimates for new data, bypassing expensive iterative optimization.
• The Challenge: In many real-world applications (e.g., remote sensing with cloud cover, medical records with missing entries), data is incomplete. Standard architectures like Convolutional Neural Networks (CNNs) require fixed-dimensional, complete inputs.
• Existing Solution & Limitations: A recent approach, "Masking," handles missing data by replacing missing values with a constant (e.g., zero) and feeding a binary "mask" vector indicating missingness into the network.
  • Drawback: The Masking approach requires the neural network to learn a mapping from a high-dimensional space ($2n$ dimensions: data + mask) to parameters. Crucially, it requires specifying a stochastic model for the missingness mechanism. If this model is misspecified (e.g., assuming data is Missing Completely At Random when it is actually Missing in a Contiguous Block), the resulting estimators become biased and statistically inefficient.

2. Methodology

The authors propose an alternative framework based on the Monte Carlo Expectation-Maximization (MCEM) algorithm, integrating it with Neural Bayes Estimators.

A. The Proposed EM Approach (Algorithm 2)

Instead of feeding missing data patterns directly to the network, the authors embed the NBE within an iterative data-augmentation loop:

1. Imputation (E-step approximation): Given the current parameter estimate $\hat{\theta}^{(l-1)}$ and observed data $Z_1$, the missing data $Z_2$ are conditionally simulated (imputed) $m$ times to create $m$ complete data vectors $\{Z^{(j)}\}_{j=1}^m$.
2. Estimation (M-step approximation): Instead of performing numerical optimization to maximize the likelihood of these complete datasets, the authors use a pre-trained Neural Bayes Estimator to approximate the Maximum A Posteriori (MAP) estimate.
   • The NBE is trained to map a set of $m$ conditionally independent complete data vectors directly to the parameter estimate: $\hat{\theta}^{(l)} = h(\{Z^{(j)}\}_{j=1}^m; \gamma^*)$.
   • This step is instantaneous (fraction of a second) because it avoids numerical optimization.
3. Iteration: The new estimate $\hat{\theta}^{(l)}$ is used for the next round of conditional simulation. The process repeats until convergence, often using an averaging scheme (burn-in) to stabilize the sequence.

B. Key Technical Innovations

• Likelihood-Free: The method does not require evaluating the likelihood function (which is often intractable for spatial models like Potts models). It only requires the ability to simulate complete data and conditionally impute missing data.
• Neural MAP Approximation: To train the NBE for the M-step, the authors approximate the 0-1 loss function (which yields the MAP estimator) using a smooth surrogate, specifically $L(\theta, \hat{\theta}) = \tanh(\|\hat{\theta} - \theta\|/\kappa)$.
• Architecture for Replicates: To handle the $m$ imputed datasets, they utilize a DeepSets architecture, which ensures the estimator is permutation-invariant (order of imputations doesn't matter) and scalable.
• No Missingness Model Required: Unlike the Masking approach, the EM approach does not require a model for how data went missing. It only requires a model for the data itself to perform conditional simulation.

3. Key Contributions

1. Theoretical Insight: The authors demonstrate that while the Masking approach is theoretically sufficient (Theorem 1), its practical performance relies heavily on the correct specification of the missingness distribution. Misspecification leads to significant bias in high-dimensional settings.
2. Algorithmic Framework: They introduce a novel EM-based Neural Bayes Estimator that combines the speed of amortized inference with the statistical robustness of classical data augmentation.
3. Efficiency Trade-off Analysis: They quantify the trade-off between the Masking approach (fastest inference, sensitive to missingness model) and the EM approach (slower inference due to simulation, but robust and statistically efficient).
4. Application to Intractable Models: They successfully apply the method to the Hidden Potts Model, a spatial model with an intractable likelihood (due to a normalizing constant), where traditional likelihood-based inference is computationally prohibitive.

4. Results

The authors conducted extensive simulation studies and a real-world application:

Simulation Studies

• Models Tested: Spatial Gaussian Process (GP) and Hidden Potts Model.
• Missingness Scenarios: Missing Completely At Random (MCAR) and Missing In a Contiguous Block (MICB).
• Findings:
  • Statistical Efficiency: The EM NBE achieved Root Mean Squared Errors (RMSE) comparable to the gold-standard MAP estimator under both MCAR and MICB.
  • Robustness: The Masking NBE performed well under MCAR (where it was trained) but suffered from massive bias and inflated RMSE under MICB, confirming that misspecifying the missingness mechanism degrades performance.
  • Comparison with ABC: In a Generalized Hyperbolic distribution study, the EM NBE outperformed Approximate Bayesian Computation (ABC), which relies on user-defined summary statistics and showed lower statistical efficiency.

Real-World Application: Arctic Sea Ice

• Data: 45 years of sea-ice concentration data (1979–2023) with complex missingness patterns (cloud cover, "Arctic Pole Hole").
• Model: A Hidden Potts model with 4 states (no ice, low, medium, full ice) using Beta and point-mass distributions.
• Performance:
  • The EM NBE estimated parameters for all 45 years in 44 seconds total.
  • This is orders of magnitude faster than fully Bayesian MCMC approaches (which took >1 day for a smaller dataset in previous literature).
  • Results confirmed a dramatic decrease in sea-ice area over time and strong spatial dependence (parameters exceeded the critical value for phase transition).

5. Significance and Conclusion

• Bridging AI and Statistics: The paper serves as a prototype for improving AI by integrating rigorous statistical inference tools (EM algorithm, data augmentation) into deep learning workflows.
• Handling Intractability: It provides a solution for parameter estimation in complex spatial models where likelihoods are intractable and data is incomplete, a common scenario in geosciences and epidemiology.
• Robustness: It establishes that for incomplete data, statistical efficiency (robustness to missingness mechanisms) can be prioritized over raw inference speed without sacrificing the "amortized" benefit of neural networks.
• Software: The methodology is implemented in the open-source NeuralEstimators package (R/Julia), making these advanced techniques accessible to practitioners.

In summary, the paper argues that while "masking" is a quick heuristic, it is statistically fragile. The proposed EM-based Neural Bayes Estimator offers a robust, likelihood-free, and computationally feasible alternative for high-dimensional, incomplete data problems, particularly in spatial statistics.