Partial Identification under Missing Data Using Weak Shadow Variables from Pretrained Models

Technical Summary: Using LLMs for Missing-not-at-Random Outcomes: A Weak Shadow Variable Approach

1. Problem Statement

The paper addresses the challenge of estimating population quantities, specifically the mean outcome $\theta = E[Y]$, when data is Missing Not at Random (MNAR). In many digital and social science contexts (e.g., user feedback, customer service reviews), the probability of an outcome being observed depends on the unobserved value itself. For instance, users with extreme opinions are more likely to leave feedback than those with moderate experiences.

Under MNAR, standard estimators based solely on observed data are biased, and the population mean is not point-identified without strong, often unverifiable, parametric assumptions (e.g., Heckman selection models) or restrictive auxiliary variable conditions (e.g., classical shadow variables requiring completeness). The authors propose a partial identification framework to characterize the sharp bounds of the estimand under minimal assumptions, while leveraging modern machine learning predictions to tighten these bounds.

2. Methodology

2.1. Partial Identification via Linear Programming (LP)
The authors first establish a baseline framework for identifying the sharp bounds of the mean outcome without auxiliary information.

• Decomposition: The population mean is decomposed into observed joint probabilities $\alpha(y) = P(Y=y, R=1)$ and unobserved response probabilities $\pi(y) = P(R=1|Y=y)$.
• Feasible Set: Since $\pi(y)$ is unknown, the authors define a feasible set of probability distributions consistent with the observed data.
• LP Formulation: The problem of finding the sharp lower and upper bounds ($\theta_{min}, \theta_{max}$) is reformulated as a pair of linear programs. The objective function represents the mean, and the constraints encode the probabilistic structure of the observed data. Without auxiliary variables, these bounds are derived analytically and scale linearly with the missingness probability.

2.2. Weak Shadow Variables
To tighten the identification bounds, the authors introduce Weak Shadow Variables ($F$), such as predictions from Large Language Models (LLMs).

• Definition: A variable $F$ is a weak shadow variable if it satisfies an exclusion restriction: $F \perp\!\!\!\perp R \mid Y, X$. That is, conditional on the true outcome $Y$ and covariates $X$, the shadow variable provides no additional information about the missingness mechanism.
• Relaxation of Completeness: Unlike classical shadow variable methods, this approach does not require a completeness condition (which demands that $F$ fully identifies the distribution of $Y$). This allows the use of imperfect predictors (like LLMs) that may only weakly correlate with the outcome.
• Refined LP: The inclusion of $F$ introduces additional linear constraints to the LP formulation. The sharp identification region is the intersection of the feasible sets defined by these constraints. If the shadow variable is sufficiently informative (satisfying a rank condition), the bounds collapse to a single point (point identification); otherwise, they remain a narrowed interval.

2.3. Finite-Sample Estimation and Inference
Direct plug-in estimators for the LP bounds are problematic in finite samples due to potential infeasibility (sample constraints may be inconsistent) and non-smoothness (small changes in estimates can switch the active constraints).

• Localized Penalized Estimator: The authors propose a localized penalized linear program. This method relaxes the equality constraints by adding an $L_1$ penalty term ($K\|Aw - \beta\|_1$) and restricts the decision variables to a bounded box $[0, K]$. This ensures feasibility and Lipschitz continuity of the value function.
• Data-Driven Tuning: A data-driven algorithm selects the localization radius $K$ by minimizing a "certification gap," ensuring the estimator is consistent with the population target while avoiding unnecessary variance.
• Subsampling Inference: Because the LP value function is non-smooth (due to potential multiple optimal solutions), standard bootstrap methods fail. The authors propose a subsample bootstrap (Politis and Romano, 1994) to construct valid confidence intervals for the identification region endpoints. This method accounts for the "optimizer-switching" behavior inherent in the LP.

2.4. Sensitivity Analysis
The framework includes a sensitivity analysis (Assumption 2) to quantify robustness against violations of the exclusion restriction. By introducing a sensitivity parameter $\rho$, the authors allow the conditional independence assumption to hold approximately, expanding the identification region monotonically as $\rho$ increases.

3. Key Contributions

1. Novel LP Framework: The paper introduces a linear programming framework for partial identification under MNAR that unifies settings with and without auxiliary predictions. It provides closed-form solutions for the baseline case and analytical results for how auxiliary signals tighten bounds.
2. Weak Shadow Variables: The authors generalize the shadow variable approach by relaxing the completeness condition. This enables the use of "imperfect" auxiliary signals (e.g., LLM predictions) that satisfy the exclusion restriction but may not yield point identification on their own.
3. Robust Estimation and Inference: The paper develops a feasible finite-sample estimator using localized penalization and establishes a subsampling-based inference procedure. This approach guarantees $\sqrt{n}$ convergence rates and valid confidence regions even when the underlying LP has multiple solutions or when standard bootstrap methods fail.

4. Experimental Results

The authors evaluate the method through simulations and semi-synthetic experiments using real customer-service dialogue data (USS dataset).

• Simulations:

  • The localized penalized estimator achieves the parametric $\sqrt{n}$ convergence rate.
  • The data-driven selector for the localization radius $K$ consistently recovers the optimal population radius.
  • Subsampling inference correctly approximates the non-Gaussian limit distribution of the estimator, whereas standard bootstrap methods exhibit significant bias and incorrect coverage (e.g., 82% vs. 90% target).

• Semi-Synthetic Experiments (LLM-based Shadow Variables):

  • Prompt Design: The study tests various LLM prompt strategies (Digital Twin, Generic User, Service Auditor) to generate shadow variables. Results indicate that depersonalized, task-focused prompts (e.g., "Service Auditor" evaluating "Goal Completion") offer the best trade-off between relevance (correlation with $Y$) and independence (satisfying the exclusion restriction).
  • Performance: Incorporating LLM-based weak shadow variables reduced the width of identification intervals by 83% compared to the baseline LP without shadow variables.
  • Comparison: The proposed method (using the midpoint of the shadow-variable interval) achieved a mean absolute error of 0.06, significantly outperforming classical MNAR baselines like Complete Case Analysis (CCA), Naive Imputation, Heckman Selection Models, and Pattern-Mixture models. In terms of set estimation (Hausdorff distance to truth), the shadow-variable method reduced the distance from 1.07 (aggregated LP) to 0.18.

5. Significance and Claims

The paper claims that its framework offers a robust and tractable alternative to traditional MNAR methods.

• Robustness: By adopting a partial identification perspective, the method avoids the risks of model misspecification inherent in parametric selection models.
• Practicality: It leverages the growing availability of LLMs as auxiliary signals without requiring them to be perfect predictors. The "weak" nature of the shadow variable assumption makes it more plausible in practice than the strict completeness conditions of classical methods.
• Validity: The proposed inference procedure provides valid coverage guarantees for the identification region, addressing the statistical challenges posed by the non-smoothness of LP-based estimators.

The authors conclude that even simple binary or low-dimensional LLM predictions can significantly tighten identification bounds, making this approach highly valuable for platform evaluation and social science research where feedback is systematically missing.