Boosting prediction with data missing not at random

1. Problem Statement

Context: Boosting is a powerful machine learning technique that iteratively combines weak learners to form a strong predictor. However, standard boosting algorithms (e.g., AdaBoost, Gradient Boosting) assume that the dataset is complete.
The Challenge: In many real-world applications, response variables ($Y$) are subject to missingness.

• Missing Completely at Random (MCAR) / Missing at Random (MAR): Standard methods can often be applied to complete cases or use simple imputation.
• Missing Not at Random (MNAR): The probability of missingness depends on the unobserved value of $Y$ itself. Directly applying standard boosting to MNAR data leads to biased estimators and invalid predictions.
• Identifiability Issue: MNAR data poses a fundamental statistical challenge: the joint distribution of the response and the missingness mechanism is non-identifiable without strong assumptions. Traditional approaches often rely on instrumental variables (shadow variables), which are difficult to find and verify in practice.

Objective: The authors aim to develop a boosting prediction framework specifically for MNAR data that:

1. Corrects for the bias introduced by non-ignorable missingness.
2. Avoids reliance on hard-to-find instrumental variables.
3. Provides rigorous theoretical guarantees (convergence and consistency).

2. Methodology

The proposed approach integrates semiparametric estimation with a functional gradient descent boosting algorithm. The core innovation lies in adjusting the loss function to account for missingness.

A. Adjusted Loss Functions

The authors propose two strategies to construct an adjusted loss function $L^*$ such that its expectation equals the true risk $E[L(Y, f(X))]$, even when $Y$ is missing.

1. Inverse Propensity Weighting (IPW):
   $$L_{IPW}(y, f(x), r) = \frac{r \cdot L(y, f(x))}{\pi(y, x)}$$
   where $\pi(y, x) = P(R=1|Y=y, X=x)$ is the propensity score. This re-weights observed data to represent the full population.

2. Buckley-James (BJ) Type Adjustment:
   $$L_{BJ}(y, f(x), r) = r \cdot L(y, f(x)) + (1-r) \cdot \Psi_0(x)$$
   where $\Psi_0(x) = E[L(Y, f(X)) | X=x, R=0]$. This imputes the loss for missing cases based on the conditional distribution of $Y$ given $X$ and $R=0$.

   • To calculate $\Psi_0(x)$, the authors use Bayes' rule to express the conditional density $f(y|x, R=0)$ in terms of the observed density $f(y|x, R=1)$ and the propensity $\pi(y, x)$.

B. Semiparametric Estimation of Unknown Components

Since $\pi(y, x)$ and $f(y|x, R=1)$ are unknown, the authors employ a semiparametric approach to estimate them consistently:

• Propensity Score $\pi(y, x)$: Modeled parametrically (e.g., logistic regression) with parameter $\gamma$.
• Conditional Density $f(y|x, R=1)$: Can be estimated either parametrically or non-parametrically (using kernel density estimation).
• Estimation Algorithm: The authors adapt the method of Morikawa and Kim (2021). They utilize a set of identification conditions (A1–A4) that are testable with observed data (avoiding the need for external instrumental variables). They solve a modified score equation to obtain consistent estimators for $\gamma$, even if the model for $f(y|x, R=1)$ is misspecified (provided $\pi$ is correctly modeled).

C. The Boosting Algorithm (Algorithm 1)

The prediction procedure follows a functional gradient descent:

1. Initialization: Start with an initial function $f^{(0)}$.
2. Iteration:
   • Compute the gradient of the adjusted loss function $\hat{L}^*$ with respect to the current prediction.
   • Fit a weak learner $h^{(m+1)}$ (using B-splines) to the negative gradient.
   • Determine the step size $\alpha^{(m+1)}$ via line search.
   • Update: $f^{(m+1)} = f^{(m)} + \alpha^{(m+1)}h^{(m+1)}$.
3. Termination: Stop when the change in empirical risk falls below a threshold $\eta$.

3. Key Contributions

1. Novel Framework for MNAR: This is one of the first works to extend boosting specifically to the MNAR setting without requiring external instrumental variables.
2. Semiparametric Robustness: By combining parametric modeling of the propensity score with flexible (parametric or non-parametric) modeling of the outcome density, the method achieves consistency even if the outcome model is misspecified (double robustness in spirit, though formally relying on the identification conditions).
3. Theoretical Guarantees:
   • Convergence: Proved that the iterative algorithm converges to the optimal risk $R(f^*)$ as the number of iterations increases (Proposition 2, Theorem 1).
   • Consistency: Established that the final estimator $\hat{f}_{AL}$ converges in probability to the true function $f^*$ in the $L_\infty$ norm as the sample size $n \to \infty$ (Theorem 2).
4. Practical Implementation: Provided a concrete algorithm using B-splines for the weak learners and specific strategies for estimating the unknown nuisance parameters.

4. Results

Simulation Studies

The authors conducted extensive simulations with sample sizes $n=1000$ under two settings (low and high-dimensional covariates) and two missingness mechanisms (MAR and MNAR).

• Performance Metrics: Sample-based Maximum Absolute Error (S-MAE) and Root Mean Squared Error (S-RMSE).
• Comparisons: Compared against:
  • R: Standard boosting on full data (Oracle).
  • N: Standard boosting on observed data only (Naive).
  • IPW/IPWN: Proposed methods with parametric/non-parametric density estimation.
  • BJ: Proposed Buckley-James method.
• Findings:
  • Under MAR, all methods performed similarly.
  • Under MNAR, the Naive (N) method exhibited significant bias and high error.
  • The proposed methods (IPW, IPWN, BJ) performed comparably to the Oracle (R) method, successfully correcting the bias.
  • Model Misspecification: The IPW method was robust to misspecification of the outcome density $f(y|x, R=1)$ but sensitive to misspecification of the propensity model $\pi(y, x)$.
  • High Dimensions: Non-parametric estimation (IPWN) struggled in high-dimensional settings due to the curse of dimensionality, whereas parametric approaches remained stable.

Real Data Application (KLIPS)

The methods were applied to the Korean Labor and Income Panel Survey (KLIPS) to predict worker income.

• Data: 2501 workers, ~30% missing income data.
• Result: The proposed methods (IPW, IPWN, BJ) produced prediction curves that were consistent with each other but significantly different from the Naive method, particularly at the extremes of the covariate range. This suggests the Naive method was biased due to the MNAR nature of the missing income data.

5. Significance and Implications

• Theoretical Advancement: The paper bridges a critical gap between machine learning (boosting) and missing data theory (MNAR), providing the first rigorous theoretical justification for boosting under non-ignorable missingness.
• Practical Utility: It offers a viable solution for industries where missing data is often non-random (e.g., healthcare, economics, survey research) and where standard imputation or complete-case analysis yields misleading results.
• Flexibility: The framework is generalizable. While demonstrated with $L_1$, $L_2$, and Huber losses, the authors note it can be extended to other machine learning algorithms (SVMs, Neural Networks) by modifying their respective loss functions.
• Robustness: The reliance on testable identification conditions rather than untestable instrumental variables makes the method more applicable to real-world scenarios where such variables are unavailable.

In conclusion, this paper provides a robust, theoretically grounded, and practically implementable solution for improving prediction accuracy in the presence of Missing Not at Random data, significantly outperforming standard "complete-case" boosting approaches.