Model Assisted Data Integration: An unbiased sampling strategy to use nonprobability data

1. Problem Statement

Official statistical agencies face increasing pressure to utilize Non-Probability Data (NPD) (e.g., administrative registers, web panels, transaction data) to reduce costs and respondent burden. However, NPD suffers from selection bias (lack of known inclusion probabilities) and often incomplete coverage of the target population.

Existing methods for integrating NPD with probability samples (e.g., imputation, propensity weighting, double robust estimators) often rely on strong, unverifiable assumptions such as Missing At Random (MAR). Furthermore, traditional model-assisted estimators like the Generalized Regression (GREG) estimator struggle when:

• Sample sizes are small relative to the number of auxiliary variables (leading to singular matrices).
• Complex machine learning models are used (risk of overfitting and biased variance estimation).
• The NPD has severe selection bias.

The authors aim to develop a sampling strategy and estimator that is design-unbiased, provides a design-unbiased variance estimator, and allows for the use of arbitrary machine learning models without relying on MAR assumptions.

2. Methodology: The MADI Strategy

The paper proposes the Model Assisted Data Integration (MADI) strategy, which extends the Data Integration (DI) estimator proposed by Kim and Tam (2021).

Core Concept

The population $U$ is partitioned into two disjoint sets:

• Set $A$ (NPD): A subset of the population where the study variable $y$ is known for all units (e.g., a complete register or large non-probability dataset), but selection probabilities are unknown.
• Set $B$ (Complement): The subset of the population not covered by $A$ ($B = U \setminus A$).

Sampling Design

Instead of sampling from the entire population $U$, the strategy draws a probability sample ($s_B$) exclusively from set $B$. This is crucial because the values of $y$ are already known for all units in $A$, so sampling from $A$ is unnecessary for estimation purposes.

The Estimator

The MADI estimator ($\hat{t}_{y, MADI}$) combines three components:

1. Total of Set $A$: The sum of known $y$ values in the NPD ($t_A = \sum_{i \in A} y_i$).
2. Model Prediction for Set $B$: A machine learning model $\mu(\mathbf{x}, A)$ is trained on the large NPD (Set $A$) using auxiliary variables $\mathbf{x}$ to predict $y$ for all units in $B$.
3. Correction Term: A difference estimator based on the probability sample $s_B$ to correct for the error between the model predictions and the true values in $B$.

The formula is defined as:
$$\hat{t}_{y, MADI} = \underbrace{\sum_{i \in A} y_i}_{\text{Known Total}} + \underbrace{\sum_{i \in B} \mu(\mathbf{x}_i, A)}_{\text{Model Prediction}} + \underbrace{\sum_{i \in s_B} \frac{y_i - \mu(\mathbf{x}_i, A)}{\pi_{iB}}}_{\text{Design-Based Correction}}$$

Where:

• $\pi_{iB}$ is the inclusion probability of unit $i$ in the sample drawn from $B$.
• $e_i = y_i - \mu(\mathbf{x}_i, A)$ is the residual.

Key Theoretical Properties

• Design-Unbiasedness: Because the model $\mu$ is trained on $A$ (which is fixed relative to the random sampling of $s_B$), the estimator remains design-unbiased regardless of the model's fit quality. The bias of the model is corrected by the third term.
• Unbiased Variance Estimation: The variance depends only on the residuals ($e_i$) in the probability sample $s_B$. An unbiased variance estimator is derived using standard design-based formulas (Horvitz-Thompson style) applied to the residuals.
• Model Flexibility: The strategy allows for arbitrary machine learning models (e.g., Random Forests, Neural Networks). Since the model is trained on the large NPD ($A$) rather than the small probability sample, the risk of overfitting to the small sample is mitigated, and complex models can be utilized without the "small $n$, large $p$" issues common in GREG.

3. Key Contributions

1. Novel Sampling Strategy: Unlike previous DI estimators that sample from the whole population, MADI explicitly samples only from the uncovered portion ($B$), optimizing resource allocation.
2. Robustness to Selection Bias: The method does not require the NPD to be MAR. It handles severe selection bias (Missing Not At Random) by relying on the design-based correction term.
3. Integration of Machine Learning: It provides a rigorous framework for using complex, non-parametric ML models in official statistics while maintaining design-based inference properties (unbiasedness and valid variance estimation).
4. Variance Reduction: By leveraging the large NPD for modeling, the method significantly reduces the variance of the estimator compared to traditional survey estimators, even with very small probability sample sizes.

4. Results (Empirical Simulations)

The authors conducted simulations using data from Statistics Sweden (Income and Taxation register) with $N \approx 9,838$.

Simulation 1: Performance Comparison

• Setup: Compared MADI (with OLS and Random Forest) against traditional HT, GREG, and "Naïve" RF estimators under various sample sizes ($n=25$ to $800$) and selection bias scenarios.
• Findings:
  • MADI-RF consistently achieved the lowest Root Mean Squared Error (RMSE) across all sample sizes.
  • GREG failed to produce estimates for small sample sizes ($n < 400$) due to singular matrices caused by the high number of auxiliary variables.
  • Naïve RF (training on the small sample) produced biased estimates.
  • MADI maintained near-zero bias and 95% coverage probability even with small samples ($n=25$), whereas traditional estimators showed high variance or bias.

Simulation 2: Sample Size Efficiency

• Setup: Determined the sample size required to achieve a Coefficient of Variation (CV) of 1% under different selection bias mechanisms (including extreme MNAR).
• Findings:
  • To achieve CV = 1%, the traditional HT estimator required ~3,044 units.
  • The GREG estimator required ~269 units (theoretically), but practically struggled with stability.
  • MADI-RF required between 42 and 157 units (depending on the bias scenario) to achieve the same precision.
  • Even in scenarios with extreme selection bias (where the NPD covers only the highest or lowest income earners), MADI-RF required significantly fewer units than HT and was often more efficient than GREG.

5. Significance and Implications

• Cost and Burden Reduction: The MADI strategy demonstrates that statistical agencies can drastically reduce the size of their probability samples (by a factor of 10x to 20x in the simulations) while maintaining or improving precision. This directly lowers survey costs and respondent burden.
• Production Feasibility: The method is designed for the "intense production cycle" of statistical agencies. It uses familiar design-based theory (unbiasedness, variance estimation) but incorporates modern ML tools, making it safer and easier to implement than black-box ML approaches.
• Handling "Big Data" Limitations: It solves the issue where NPD is large but biased. By using the NPD for modeling (training) and the probability sample for calibration (correction), it leverages the volume of big data without inheriting its bias.
• Future Outlook: The paper suggests this approach is particularly suitable for business statistics and continuous data collection, allowing agencies to transition from pure survey-based estimation to hybrid data integration strategies with rigorous theoretical backing.

In conclusion, the MADI strategy offers a theoretically sound, empirically validated, and practically efficient solution for integrating non-probability data into official statistics, overcoming the limitations of both traditional survey methods and standard machine learning applications.