Diagnostics-guided variance-inflated Fay-Herriot estimation from non-probability samples

Technical Summary: Diagnostics-guided variance-inflated Fay–Herriot estimation from non-probability samples

Problem Statement
The paper addresses the challenge of small area estimation (SAE) using non-probability data sources, such as administrative records or platform data. While inverse probability weighting (IPW) and calibration are standard techniques to reduce selection bias in these settings, the reliability of the resulting domain estimators varies significantly across different small areas. A critical limitation of the standard Fay–Herriot (FH) model in this context is its reliance on supplied area-level variance estimates ($\hat{V}_d$) as the sole measure of uncertainty. The authors argue that $\hat{V}_d$ often fails to capture risks associated with weak domain coverage, unstable weights, or poor auxiliary balance. Consequently, standard FH smoothing may inappropriately trust unreliable inputs, leading to misleading estimates.

Methodology
The proposed approach introduces a diagnostics-guided variance-inflated FH estimator for finite-population domain totals. The method proceeds in three main stages:

1. Calibrated IPW Estimation:
   Domain totals are first estimated using calibrated IPW. A logistic participation model is fitted to estimate inclusion probabilities ($\hat{\pi}_i$), generating raw IPW weights. These weights are then calibrated within each domain to known population totals (e.g., domain size or auxiliary totals) using a quadratic distance minimization approach. The variance of these calibrated estimators is estimated via a generalized multiplier bootstrap that refits the participation model and recalibrates weights in each replicate.

2. Domain Diagnostics and Reliability Scoring:
   To assess the quality of the IPW input, the method computes three domain-level diagnostics:

   • Coverage Rate ($c_d$): The proportion of the domain population represented in the sample.
   • Weight Variability ($CV_{w,d}$): The coefficient of variation of the calibrated weights, indicating the stability of the estimator (high variability suggests reliance on few units).
   • Auxiliary Balance ($B_d$): The Euclidean distance between the weighted sample means and the known population means of standardized auxiliary variables.

   These diagnostics are normalized and combined into a single reliability score ($Q_d$) for each eligible domain. Domains with low scores are flagged as having weaker information.

3. Mixture Variance-Inflated FH Model:
   The core innovation is a mixture model applied to the FH observation equation. A latent indicator $\xi_d \in \{0, 1\}$ determines whether a domain follows an "ordinary" component or a "variance-inflated" component.

   • The probability of the ordinary component ($\pi_d$) is set equal to the normalized reliability score $Q_d$.
   • The observation equation is defined as:
     $$\hat{Y}^{IPW}_d | \theta_d, \xi_d \sim N\{\theta_d, \hat{V}_d + (1-\xi_d)\tau\}$$
     where $\tau \geq 0$ is an inflation parameter.
   • Domains with low reliability scores (low $\pi_d$) are assigned to the inflated component, effectively increasing their observation variance. This causes the FH shrinkage to pull these estimates more strongly toward the regression mean ($\theta_d$), reducing the influence of potentially biased or unstable inputs.

   The model parameters ($\beta, A, \tau$) and posterior probabilities are estimated via an iterative algorithm.

Key Contributions
The paper makes three specific contributions:

1. Compact Diagnostics Construction: It proposes a unified reliability score based on coverage, weight stability, and auxiliary balance to quantify the quality of non-probability IPW inputs.
2. Mixture Variance-Inflation: It incorporates this reliability score into a mixture FH model, allowing the data-driven inflation of observation variance for unreliable domains rather than treating all inputs as equally reliable.
3. Truth-Known Validation: It validates the method using a pseudo-real population of Lithuanian business enterprises where the true domain totals are known, allowing for a direct assessment of estimation error.

Results
The validation was conducted on a population of 2,839 Lithuanian enterprises across 23 economic domains, using monthly turnover as the study variable. The proposed estimator was compared against:

1. Calibrated IPW estimators.
2. Standard FH estimators (without variance inflation).

The results demonstrated a substantial reduction in estimation error for the proposed method:

• Aggregate Relative Error: Reduced from 14.56% (Calibrated IPW) and 12.83% (Standard FH) to 2.67% (Proposed VI-FH) under domain-size calibration.
• Domain RMSE: Reduced from 7.87 million (IPW) and 7.59 million (Standard FH) to 1.31 million (Proposed).
• MAPE: Reduced from 61.22% (IPW) and 57.68% (Standard FH) to 13.39% (Proposed).

Furthermore, the study confirmed that the diagnostic scores were empirically correlated with the actual absolute relative errors of the IPW estimators, validating their utility as pre-smoothing reliability indicators. The authors also noted that adding richer auxiliary calibration information (beyond domain size) did not automatically improve performance, suggesting that calibration sets should be treated as statistical specifications rather than monotone improvements.

Significance and Claims
The paper claims that its method provides a transparent mechanism to propagate diagnostic evidence regarding weak non-probability support into area-level FH analysis. It explicitly states that the method does not identify arbitrary unmeasured selection bias without external benchmarks. Instead, its significance lies in preventing the area-level model from treating all non-probability IPW inputs as equally reliable when their variance estimates alone are insufficient to describe the uncertainty. The authors present this as targeted evidence of the variance-inflation mechanism's effectiveness, rather than an exhaustive assessment of all non-probability selection mechanisms.