📊 statistics

Efficient Transported Distributional and Quantile Treatment Effects with Surrogate-Assisted Missing Primary Outcomes

This paper proposes a novel, efficient framework for estimating transported distributional and quantile treatment effects in settings where primary outcomes are missing in a target population, leveraging post-treatment surrogates from a source study to improve estimation precision without relying on strict surrogacy assumptions.

Original authors: Pengyun Wang

Published 2026-05-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengyun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Long-Run" Problem

Imagine you are a policymaker trying to decide if a new job-training program works. You have two groups of people:

The Source Group (The Pilot): You have detailed data on everyone in this group. You know who got the training, their background, and their short-term earnings (a "surrogate"). However, you only know their long-term earnings (the "primary outcome") for a small, lucky subset of them. For the rest, the long-term data is missing.
The Target Group (The Real World): You have a massive list of people you want to help. You know their backgrounds (covariates), but you have zero data on their training, their short-term earnings, or their long-term earnings.

The Goal: You want to predict how the training would change the entire distribution of long-term earnings for the Target Group. Not just the average, but the whole picture: Will it help the bottom 10%? The middle? The top?

The Challenge: Missing Pieces and Different Crowds

This is hard for two reasons:

Missing Data: In the Source Group, we are missing the "long-term" score for most people.
Different Crowds: The Source Group and Target Group are different. The Source Group might be younger or more motivated. You can't just copy-paste the results; you have to "transport" them.

Usually, statisticians try to use the "short-term" earnings (the surrogate) as a perfect stand-in for the long-term earnings. This paper says: No. The short-term earnings aren't a perfect replacement. Instead, think of the short-term earnings as a clue or a hint that helps us guess the missing long-term scores more accurately.

The Solution: The "Three-Layer Detective"

The authors built a new statistical method that acts like a three-layer detective to solve this puzzle. They call it a "Transported One-Step Estimator."

Here is how the three layers work, using a House Renovation analogy:

Imagine you want to know the final value of a house (the long-term outcome) after a renovation (the treatment).

Layer 1: The Neighborhood Survey (Target Covariates). You have a map of the Target neighborhood. You know the average house size and age there. You need to adjust your prediction to fit this specific neighborhood, not the one where the renovation happened.
Layer 2: The Blueprint (Source Surrogates). In the Source neighborhood, you have blueprints (short-term data) for every house. Even if you don't know the final sale price for every house, you know the blueprint. This paper uses the blueprint to guess what the missing sale prices might be. It's not a perfect guess, but it's better than nothing.
Layer 3: The Appraisal (Validated Outcomes). For a few houses in the Source neighborhood, you do know the final sale price. You use these known prices to check and correct your guesses based on the blueprints.

The magic of this paper is that it combines all three layers into one single, efficient calculation. It doesn't just average them; it mathematically subtracts the "guessing errors" from the blueprint layer and the "sampling errors" from the appraisal layer to get a super-accurate result.

Key Concepts Explained Simply

1. The "Surrogate" is a Helper, Not a Replacement
Think of the short-term earnings (surrogate) like a weather forecast.

If you want to know if it will rain next month (long-term outcome), a forecast for today (short-term) isn't a guarantee.
But if you have a forecast, you can make a better guess than if you had no forecast at all.
This paper proves that using the forecast (surrogate) makes your guess about the rain (long-term outcome) much sharper, even if the forecast isn't perfect.

2. The "Efficient Influence Function" (The Perfect Recipe)
In statistics, there is a theoretical limit to how accurate any method can be. The authors found the "perfect recipe" (called the canonical gradient) that hits this limit.

Their recipe has three distinct ingredients (the three layers mentioned above).
Because they found the exact recipe, they can prove that their method is the most efficient possible way to solve this problem. You can't do better than this without getting more data.

3. Quantile Treatment Effects (The Whole Story)
Most studies only look at the average effect (e.g., "The program raises earnings by $500 on average").

This paper looks at the whole distribution (e.g., "The program helps the poorest people a lot, but does nothing for the rich").
They calculate the "Quantile Treatment Effect," which tells you how the program shifts the bottom 10%, the middle 50%, and the top 10% separately.
The Surprise: The "hint" (surrogate) helps you guess the bottom 10% much better than the top 10%, or vice versa, depending on the data. The paper gives a formula to calculate exactly how much extra accuracy you get for each specific part of the distribution.

How They Proved It Works

The authors didn't just guess; they did two things:

Mathematical Proof: They used advanced calculus to prove that their method is unbiased (it doesn't lie) and efficient (it's the fastest way to get the answer). They showed that even if their "guessing" tools (the nuisance functions) aren't perfect, the final answer is still correct as long as the errors cancel out.
Simulations: They created fake data on a computer that mimicked the real-world problem. They tested their method against older methods.
- Result: Their method was significantly more accurate (lower error) than the old methods.
- Result: It worked even when the "Target" crowd was very different from the "Source" crowd.

The Bottom Line

This paper provides a new, mathematically perfect tool for policymakers and researchers. It allows them to take a small study with some missing long-term data and a large pool of people with no outcome data, and accurately predict how an intervention will affect the entire range of outcomes (from the worst to the best) for the new population.

It treats short-term data not as a magic replacement, but as a powerful clue that, when combined with the right math, reveals the full story of long-term success.

Technical Summary: Efficient Transported Distributional and Quantile Treatment Effects with Surrogate-Assisted Missing Primary Outcomes

Problem Setting
The paper addresses the estimation of target-population distributional and quantile treatment effects (QTEs) in a setting where long-run primary outcomes are expensive, delayed, or missing, while short-run surrogate outcomes are widely available. The data structure consists of two independent samples:

Source Study: Contains treatment assignment ( $A$ ), baseline covariates ( $X$ ), post-treatment surrogates ( $S$ ) for all units, and a primary outcome ( $Y$ ) observed only for a validation subset ( $M=1$ ).
Target Population: Contains only baseline covariates ( $X$ ).

The goal is to estimate the transported counterfactual distribution function $\psi_a(y) = P(Y^a \le y \mid R=0)$ , its quantiles $q_a(\tau)$ , and the quantile treatment effect $\Delta(\tau) = q_1(\tau) - q_0(\tau)$ . Crucially, the surrogate $S$ is not treated as a replacement endpoint, nor are strict Prentice-type surrogacy conditions imposed. Instead, $S$ serves as auxiliary information to improve efficiency under missing-at-random (MAR) sampling of the primary outcome.

Methodology
The authors develop a semiparametric framework based on the following identification strategy:

Identification: Under assumptions of consistency, source exchangeability, conditional transportability of the treatment-specific surrogate-primary outcome process, and MAR validation sampling, the target CDF is identified as $\psi_a(y) = E_0[g_a(y, X)]$ , where $g_a(y, x) = E[m_a(y, X, S) \mid R=1, A=a, X=x]$ and $m_a$ is the conditional CDF of $Y$ given $(X, S)$ in the validation subset.
Efficient Influence Function (EIF): The paper derives the canonical gradient for the observed-data functional. The EIF, $\phi^y_a(O)$ $ϕ_{a}^{y} (O)$ , decomposes into three orthogonal components:
- A target covariate sampling term.
- A source surrogate-process residual term ( $m_a - g_a$ ).
- A primary-outcome validation residual term ( $Z_y - m_a$ ).
  This decomposition reflects the three sources of information used for identification.
Estimation: A closed-form, cross-fitted one-step estimator is proposed. It combines the plug-in estimate of the target expectation with the empirical averages of the EIF components. The nuisance functions ( $m_a, g_a, e_a, \rho_a, \omega$ ) are estimated using convex or normal-equation-based methods (e.g., ridge logistic regression, kernel ridge regression, entropy balancing).
Inference: The paper establishes pointwise and uniform asymptotic linearity for the CDF estimator. Using the delta method and inverse-CDF mapping, it derives Bahadur representations for quantiles and QTEs. Simultaneous confidence bands are constructed via multiplier bootstrap, relying on explicit conditions for the entropy of the estimated process classes.

Key Contributions

Identification and Efficiency: The paper extends the line of work on surrogate-assisted average treatment effects (e.g., Kallus and Mao, 2025) to the full counterfactual distribution and quantiles. It derives the exact efficient influence function for this transported setting, showing that the gradient has three orthogonal residual components corresponding to the target covariates, the source surrogate process, and the missing primary outcomes.
Exact Drift Identity and Robustness: The authors establish an exact drift identity for the estimator. This identity demonstrates "ratio-level robustness," showing that the first-order bias vanishes if either the outcome regressions ( $m_a, g_a$ ) are correct or the ratio-level weights (density ratio $\omega$ and validation propensity ratio $\rho$ ) are correct. This yields product-rate conditions for nuisance estimation, allowing for slower convergence rates than $n^{-1/4}$ provided the product of errors is $o(n^{-1/2})$ .
Quantile-Specific Efficiency Gains: The paper quantifies the efficiency gain from observing surrogates. It derives a formula showing that the variance reduction depends on the conditional variance of the surrogate-predicted outcome given covariates and the scarcity of validation data. Crucially, the gain is quantile-specific; a surrogate may substantially improve inference at one quantile while offering little gain at another, depending on the local density of the target distribution and the predictive power of the surrogate at that threshold.
Low-Level Verification: The authors provide a "deliberately analyzable" implementation path. They verify the high-level nuisance rate and entropy conditions for a specific class of bounded finite-dimensional or finite-rank learners, including ridge logistic regression, finite-rank kernel ridge regression, and entropy balancing under exponential-tilt models. This bridges the gap between abstract semiparametric theory and practical, convex optimization-based implementation.

Results

Asymptotic Theory: The proposed estimator is shown to be semiparametrically efficient at fixed $y$ . Under uniform regularity conditions (including Donsker properties and entropy bounds), the estimator achieves uniform asymptotic linearity over the outcome support, enabling valid simultaneous inference for the entire CDF and QTE curves.
Simulation Evidence: Simulation studies confirm the theoretical findings. The surrogate-assisted one-step estimator (SA) significantly reduces Mean Squared Error (MSE) compared to inverse-probability weighting (IPW) and transported plug-in estimators, particularly in the tails of the distribution. The SA estimator maintains low bias and achieves coverage rates close to nominal levels, whereas IPW suffers from high variance and plug-in estimators from approximation bias.
Efficiency Validation: The simulations verify the theoretical efficiency gain formula, demonstrating that the variance reduction correlates with the surrogate's predictive power and the scarcity of validation data.

Significance and Claims
The paper claims to provide a rigorous theoretical foundation for estimating distributional and quantile treatment effects in the presence of missing primary outcomes and available surrogates, specifically when transporting results from a source study to a target population.

Methodological Novelty: It moves beyond mean-based effects to the full distribution, handling the complexities of inverse-CDF mapping and the coupling of transport weights with surrogate residuals.
Practical Utility: By proposing a closed-form, cross-fitted one-step estimator that relies on convex subproblems, the paper offers a computationally feasible solution that does not require non-convex optimization.
Theoretical Precision: The derivation of the exact drift identity and the quantile-specific efficiency gain formula provides a precise characterization of when and how surrogates improve inference, avoiding the need for strong surrogacy assumptions.

The authors emphasize that their contribution lies in the canonical gradient theory for a two-population surrogate-assisted model and the verification of these conditions for specific, analyzable nuisance learners, rather than in proposing arbitrary flexible machine learning algorithms. The work is presented as a working manuscript, inviting comments on the theoretical derivations and implementation details.