📊 statistics

Adaptive and Efficient Learning with Blockwise Missing and Semi-Supervised Data

This paper proposes DEFUSE, a novel data-adaptive estimation framework that effectively handles the combined challenges of blockwise missing covariates and semi-supervised learning under distributional shifts by adaptively integrating multiple data sources, employing a screening method to ensure alignment, and leveraging unlabeled data to reduce estimation variance without introducing bias.

Original authors: Yiming Li, Ruoyu Wang, Ying Wei, Molei Liu

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiming Li, Ruoyu Wang, Ying Wei, Molei Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect cake (predicting a health outcome), but you are a chef who has to gather ingredients from three different, very different kitchens.

The Problem: A Messy Kitchen

The "Gold Standard" Kitchen (Labeled Data): You have a small, high-quality kitchen where you have the recipe and the finished cake. You know exactly what ingredients made the cake taste good. But there are only a few cakes here.
The "Missing Ingredient" Kitchens (Blockwise Missing Data): You have several other kitchens. They have lots of ingredients, but they are messy. Kitchen A has flour and sugar but no eggs. Kitchen B has eggs and milk but no flour. You can't just mix them all together because you don't have the full picture for any single cake.
The "Massive Warehouse" (Unlabeled Data): You also have a giant warehouse with millions of raw ingredients (covariates), but no one has baked a cake with them yet. You don't know if they taste good or bad.
The "Different Standards" Issue: The problem is that these kitchens measure things differently. Kitchen A weighs flour in cups; Kitchen B weighs it in grams. Kitchen A uses organic eggs; Kitchen B uses factory eggs. If you just mix them, your cake will be ruined because the "flavor profiles" (distributions) don't match.

The Solution: DEFUSE
The authors of this paper created a new method called DEFUSE (Data-adaptive Estimation for data FUsion in the SEmi-supervised setting). Think of DEFUSE as a super-smart "Recipe Translator" and "Quality Control Inspector" that helps you bake the best cake possible using all these messy resources.

Here is how it works, step-by-step:

1. The "Anchor" (Start with what you know)

First, DEFUSE looks at the small, high-quality kitchen (the "Labeled Complete" data). It makes a rough guess at the recipe based only on that. But because the other kitchens measure things differently, this guess might be slightly off.

2. The "Translator" (Fixing the Differences)

Instead of just throwing the messy data into the mix, DEFUSE uses a clever trick called Control Variates.

Imagine you have a translator who knows how Kitchen A's "cup of flour" relates to Kitchen B's "gram of flour."
DEFUSE uses the massive warehouse of raw ingredients (the unlabeled data) to learn these translation rules. It figures out how to adjust the measurements so that the "flour" from Kitchen A is comparable to the "flour" from Kitchen B.
It then uses these adjustments to refine the initial recipe, making it much more accurate without needing to bake millions of new cakes.

3. The "Lie Detector" (Screening for Bad Data)

This is a crucial part. Sometimes, a kitchen is so different that no amount of translation will work. Maybe Kitchen C uses a completely different type of wheat that doesn't belong in your cake at all.

DEFUSE has a "Lie Detector" test. Before it mixes a new kitchen's data into the recipe, it checks: "Is this kitchen's data actually compatible with our goal?"
If the data is too "misaligned" (too different), DEFUSE politely says, "No thanks," and excludes that kitchen. This prevents the final cake from tasting weird or being ruined by bad ingredients.

4. The "Smart Adjuster" (Learning on the Fly)

DEFUSE is "adaptive." It doesn't just use a fixed rule. It learns which adjustments work best.

If the data is simple, it uses simple math.
If the data is complex (like high-dimensional medical records with thousands of variables), it uses powerful AI tools (like Kernel Ridge Regression) to find the best way to blend the data.
It constantly checks its own work to ensure it isn't introducing any new errors (bias) while trying to reduce the uncertainty (variance).

Why is this a big deal?

Efficiency: It gets you a "perfect cake" (highly accurate prediction) using far fewer expensive "baked cakes" (labeled data) than before. It makes the most of the millions of raw ingredients in the warehouse.
Robustness: It doesn't break if the kitchens are messy or if the ingredients are measured differently. It handles the "blockwise missing" problem (where some ingredients are missing in some kitchens) naturally.
Safety: By screening out the incompatible kitchens, it ensures the final result isn't biased by bad data.

In Summary:
DEFUSE is a smart system that takes a small amount of perfect data, a lot of messy data with missing pieces, and a massive amount of unlabeled data. It translates the messy data into a common language, filters out the incompatible sources, and combines everything to give you the most accurate prediction possible, even when the data sources are very different from one another.

The authors proved this works mathematically and tested it on real-world medical data (Alzheimer's and Heart Disease), showing it creates much better predictions than previous methods.

Technical Summary: Adaptive and Efficient Learning with Blockwise Missing and Semi-Supervised Data

1. Problem Statement

The paper addresses the challenges of data fusion in the presence of three simultaneous complications:

Blockwise Missingness (BM): Covariates are missing in blocks across different data sources because variables are collected or defined differently. This results in non-monotone missing patterns where specific subsets of covariates are observed in some sources but missing in others.
Semi-Supervised (SS) Setting: The outcome variable $Y$ is expensive to obtain (e.g., requiring manual clinical review), leaving the majority of samples in the target population unlabeled, while a large pool of unlabeled data with fully observed covariates is available.
Distributional Shifts: Data sources often exhibit heterogeneity due to varying measurement standards or recruitment criteria. Standard assumptions like Missing Completely at Random (MCAR) are often unrealistic; instead, missingness may depend on observed variables, and distributions between sources may differ.

The goal is to robustly and efficiently estimate parameters of a Generalized Linear Model (GLM) for a target population (Unlabeled Complete, UC) by fusing:

LC (Labeled Complete): Small sample with fully observed covariates and outcomes.
LM (Labeled Missing): Multiple sources with outcomes but blockwise missing covariates.
UC (Unlabeled Complete): Large sample with fully observed covariates but no outcomes.

A critical challenge is source misalignment: if the conditional distribution of outcomes given covariates differs between a labeled source and the target population (violating transportability), naive fusion introduces bias.

2. Methodology: DEFUSE

The authors propose DEFUSE (Data-adaptive Estimation approach for data FUsion in the SEmi-supervised setting), which operates under a Conditional Alignment under Missingness (CAM) assumption.

2.1 Core Assumption (CAM)

The method assumes that conditional on a prespecified subset of variables $Z_r$ (which may include outcomes and observed covariates), the data distributions of the labeled missing sources ( $LM_r$ ) and the target population ($UC$) are aligned:
$P_{LM_r}(Y, X | Z_r) = P_{UC}(Y, X | Z_r)$
This relaxes the strict MCAR assumption, allowing missingness to depend on observed data, provided the conditional alignment holds.

2.2 Two-Stage Estimation Procedure

Stage 1: Preliminary Estimation (DML Operator)
A preliminary estimator $\hat{\gamma}$ is derived using only the LC and UC samples. To handle distributional shifts between LC and UC without relying on unstable density ratio estimation, the authors employ a Double/Debiased Machine Learning (DML) operator:
$M_{LC}(D) = h_{LC}(X)(D - E_{LC}[D|X]) + E_{UC}(E_{LC}[D|X])$
where $h_{LC}(X)$ is the density ratio. This operator corrects for bias while maintaining $n^{1/2}$ -consistency even if nuisance models (density ratios and conditional means) converge at slower rates ( $o_P(n^{-1/4})$ ).

Stage 2: Data-Adaptive Control Variate Approach
To incorporate the $LM_r$ samples and reduce variance, DEFUSE uses a control variate strategy. The estimator is augmented as:
$\hat{\beta}(\phi) = c^\top \hat{\gamma} + \sum_{r=1}^R \left( \hat{E}_{LM_r}[\hat{M}_{LM_r}(\phi_r)] - \hat{E}_{LC}[\hat{M}_r(\phi_r)] \right)$

Operators: $\hat{M}_r$ and $\hat{M}_{LM_r}$ are constructed using DML principles to adjust for distributional shifts between sources.
Control Functions ( $\phi_r$ ): These are data-adaptive functions of the observed variables in $LM_r$ . The method seeks to minimize the empirical asymptotic variance $Q(\phi)$ with respect to $\phi$ .
Refinement Strategy: Instead of directly optimizing complex nonparametric functions, DEFUSE refines an "anchor" predictor (e.g., an AI model predicting the score function) using a reweighting function $w_r(W_r; \delta_r)$ . This leads to a finite-dimensional quadratic programming problem, making the method computationally tractable. The authors propose three examples for $w_r$ : constant coefficients, linear functions, and Kernel Ridge Regression (KRR).

2.3 Screening for Misaligned Sources

Recognizing that some sources may violate the CAM assumption, the authors develop a screening procedure to exclude them before fusion.

Metric: They propose a Mean Squared Discrepancy (MSD) measure, $\bar{\Delta}_r^2$ , which quantifies the deviation of a source from the target alignment.
Debiasing: To avoid bias from estimating conditional means, they use a debiased estimator based on a maximization over a function space $G$ .
Selection: Sources with $\hat{\Delta}_r$ exceeding a threshold are excluded. Theoretically, this MSD measure is shown to be sharper and more sensitive to misalignment than standard mean-difference sensitivity analyses.

3. Key Contributions

Novel Framework: DEFUSE is the first method to systematically handle the triad of blockwise missingness, semi-supervised learning, and distributional shifts under a conditional alignment assumption.
Data-Adaptive Efficiency: The method introduces a new data-adaptive control variate approach. It achieves intrinsic efficiency (optimal within a specified class of control functions) and can attain the semiparametric efficiency bound if the control function class is sufficiently rich (e.g., via KRR).
Robustness to Misalignment: The proposed screening procedure effectively identifies and excludes misaligned sources, preventing bias propagation, which is a limitation in many existing fusion methods that assume perfect transportability.
Computational Feasibility: By reducing the optimization of control functions to quadratic programming, the method accommodates high-dimensional auxiliary features and black-box machine learning models while remaining computationally efficient.
Theoretical Guarantees: The paper establishes asymptotic normality, consistency, and semiparametric efficiency bounds for the proposed estimator under regularity conditions.

4. Results

Simulation Studies

Efficiency: DEFUSE consistently outperforms existing methods (including Multiple Imputation, GMM-based fusion, and sample reweighting) in terms of Relative Efficiency (RE). In various settings (linear/nonlinear, binary/continuous outcomes), DEFUSE with KRR reweighting (Example 3) achieved up to 25-80% higher efficiency than benchmarks.
Robustness: Under distributional shifts and nonlinearity, DEFUSE maintained low bias-to-standard-error ratios, whereas competitors (like SC-PW and MBI) exhibited increasing bias.
Screening Performance: In settings with misaligned sources, the MSD-based screening method achieved near-perfect F1 scores and sensitivity in identifying misaligned sources, significantly outperforming mean-difference based strategies.

Real-World Applications

Alzheimer's Risk (NACC Data): Applied to build a prognostic model using data from multiple centers. The screening method excluded 3 out of 8 centers due to misalignment. DEFUSE with KRR reweighting showed 2.76x relative efficiency for the alcohol covariate compared to the preliminary estimator.
Heart Disease Risk (MIMIC-III Data): Applied to model heart disease risk with blockwise missing biomarkers. DEFUSE demonstrated substantial efficiency gains (up to 3.65x for age and 5.72x for diabetes in multiple BM scenarios) compared to preliminary estimators and existing fusion methods.

5. Significance and Claims

The paper claims that DEFUSE offers a practical and powerful solution for modern data fusion problems where data structures are complex and heterogeneous.

Theoretical Optimality: It bridges the gap between theoretical semiparametric efficiency and practical implementation, providing a mechanism to approach efficiency bounds even when classical closed-form estimators are unavailable.
Validity Protection: By explicitly addressing source misalignment through a sensitive screening procedure, the method protects against the bias that plagues fusion of non-aligned datasets.
Flexibility: The framework is flexible enough to incorporate high-dimensional auxiliary features (e.g., from EMRs) and powerful machine learning models without sacrificing computational tractability.

The authors note that while the method currently requires a labeled complete (LC) sample, future work could generalize the approach to settings where only unlabeled and partially labeled data are available, provided the union of observed covariates covers the target predictors.