Technical Summary: Debiased Inference for AI-Generated Data without Gold-Standard Labels

1. Problem Statement

Scholars increasingly utilize Artificial Intelligence (AI), particularly Large Language Models (LLMs), to measure variables for downstream statistical analyses (e.g., sentiment, tone, political violence). A critical issue arises when these AI-generated measurements are treated as error-free in downstream inference. Recent theoretical and empirical work demonstrates that ignoring measurement errors—even when AI accuracy is high (e.g., >90%)—leads to substantial bias and invalid confidence intervals. This occurs because measurement errors are often non-random, nonclassical, and correlated with variables relevant to the downstream analysis.

Existing solutions, such as Design-Based Supervised Learning (DSL) and Prediction-Powered Inference (PPI), address this by combining error-prone AI measurements with a small set of "gold-standard" labels (human-verified truth). However, obtaining gold-standard labels is often costly or infeasible in many application domains. This paper addresses the challenge of conducting valid downstream inference using AI-generated data when no gold-standard labels are available.

2. Methodology: Debiased Inference with Multiple Imperfect Measurements (DMM)

The authors propose a framework called Debiased Inference with Multiple Imperfect Measurements (DMM). This method enables valid inference by combining three or more imperfect measurements (proxies) without requiring gold-standard validation data.

2.1 Core Assumptions and Identification

The method relies on a relaxed conditional independence assumption. Unlike classical latent variable models (e.g., Dawid-Skene) that assume proxies are independent solely conditional on the latent true label, DMM assumes proxies are independent conditional on:

1. The latent true label ($X^*$).
2. Observed covariates ($W$) and auxiliary annotation-level information ($D$), such as text embeddings, task difficulty, or prompt variations.

Formally, for $J \geq 3$ proxies $X^{(1)}, \dots, X^{(J)}$:
$$X^{(1)} \perp \dots \perp X^{(J)} \mid X^*, \tilde{D}$$
where $\tilde{D} = (D, W, Y)$. This relaxation allows for shared sources of error (e.g., a specific text being difficult for all annotators) as long as those sources are captured by the conditioning variables.

Under this assumption, the paper utilizes results from CP (Candecomp/Parafac) decomposition (Kruskal, 1977; Allman, Matias and Rhodes, 2009) to nonparametrically identify the conditional classification rates (true positive and false positive rates) for each proxy.

2.2 Estimation Strategy

The estimation proceeds in two stages using cross-fitting to ensure Neyman orthogonality:

1. First Stage (Nuisance Estimation): The conditional classification rates $\eta^*_{j,a}(d) = \Pr(X^{(j)}=1 \mid X^*=a, \tilde{D}=d)$ are estimated using an Expectation-Maximization (EM) algorithm or sieve methods on disjoint folds of the data.
2. Second Stage (Bridge Construction): A robust bridge function $H_R$ is constructed to unbiasedly estimate the latent true label $X^*$.
   • Single-proxy bridges are defined as $M_j = \frac{X^{(j)} - \eta^*_{j,0}}{\eta^*_{j,1} - \eta^*_{j,0}}$.
   • A symmetric, robust bridge is formed using pair and triple products of these single proxies:
     $$H_R = 3H_2 - 2H_3$$
     where $H_2$ and $H_3$ are averages of pairwise and triple-wise products of $M_j$.
   • This specific weighting ($3$ and $-2$) ensures that the bridge function is unbiased ($E[H_R \mid X^*, \tilde{D}] = X^*$) and achieves Neyman orthogonality, meaning the estimator is robust to first-order errors in the nuisance function estimation.

2.3 Downstream Inference

The estimated bridge $H_R$ replaces the unobserved true label in the downstream moment function $\psi_F$. The DMM estimator $\hat{\beta}_{DMM}$ solves:
$$\frac{1}{n} \sum_{i=1}^n \left[ (1 - \hat{H}_{R,i})\psi_F(Y_i, 0, W_i; \beta) + \hat{H}_{R,i}\psi_F(Y_i, 1, W_i; \beta) \right] = 0$$
The authors prove that under mild regularity conditions, this estimator is consistent and asymptotically normal, allowing for valid confidence intervals even if the nuisance functions converge at slow nonparametric rates ($n^{-1/4}$).

2.4 Extensions

The framework is generalized to:

• Latent Dependent Variables: Where the outcome is measured with error.
• Multicategory Labels: Extending the identification and bridge construction to $K > 2$ categories, requiring sufficient rank conditions on the measurement matrices.

3. Key Contributions

1. Identification without Gold-Standard Labels: The paper develops a method to conduct valid downstream inference by exploiting multiple imperfect measurements, removing the requirement for costly gold-standard validation data.
2. General Applicability: The theoretical results extend beyond simple causal effects to a wide class of downstream analyses (linear/logistic regression, MLE) and handle cases where the error-prone variable is either a dependent or independent variable.
3. Robustness and Efficiency: By constructing a Neyman-orthogonal bridge function, the method allows for slow convergence of nuisance parameters. The authors also demonstrate that adding more informative proxies improves efficiency, though low-quality proxies can degrade performance.
4. Practical Diagnostics: The paper provides strategies to make the conditional independence assumption more plausible (e.g., using diverse LLM families, varying prompts, and conditioning on text embeddings) and offers diagnostic tools (overidentification tests, subset stability checks) to assess the assumption's validity.

4. Results

4.1 Simulation Study

Using a synthetic data-generating process calibrated to political advertising data (Fowler et al., 2021):

• Bias and Coverage: Standard estimators (majority voting) exhibited substantial bias (e.g., 0.186) and severe undercoverage (1.8%) even with high F1-score proxies. In contrast, DMM achieved near-zero bias (0.002) and nominal coverage (94.8%).
• Efficiency: As the number of proxies increased, DMM's Root Mean Squared Error (RMSE) decreased, approaching the infeasible oracle benchmark (using true labels). Adding more proxies improved precision, but the gains diminished with weaker labels.

4.2 Empirical Validation

Using real-world data from Pan and Chen (2018) regarding accusations of wrongdoing against Chinese officials:

• Comparison: The study compared DMM (no gold labels), DSL (500 gold labels), and standard regressions against an oracle benchmark.
• Performance: Standard estimates were highly biased and failed to cover the benchmark. Both DSL and DMM produced estimates close to the benchmark with valid coverage (DSL: 98.1%, DMM: 86.2%).
• Significance: DMM successfully recovered the oracle estimate without using any gold-standard labels, demonstrating its viability in real-world settings where the conditional independence assumption is not guaranteed but approximated.

5. Significance and Claims

The paper positions DMM as a complementary strategy to existing validation-based methods (like DSL and PPI).

• When to use DMM: It is particularly appropriate when gold-standard labels are unavailable or prohibitively expensive, and researchers can construct multiple imperfect measurements that satisfy the conditional independence assumption (e.g., by using diverse LLMs and rich conditioning variables).
• Limitations: The authors acknowledge that DMM relies on the structural assumption of conditional independence. While they make this assumption more plausible by conditioning on rich covariates, it is not guaranteed by high predictive accuracy.
• Hybrid Approach: The paper suggests that when a small amount of gold-standard data is available, it can be combined with DMM (via a DMM-assisted DSL estimator) to further improve precision and validate the conditional independence assumption.

In summary, the paper provides a rigorous statistical framework to mitigate the bias introduced by AI-generated measurements, enabling researchers to perform valid inference in the absence of ground truth by leveraging the structure of multiple imperfect proxies.