📈 economics

Design-Based Inference under Random Potential Outcomes

This paper demonstrates that by shifting from fixed potential outcomes to stochastic mechanisms with suitably sparse local dependence, aggregate design-based estimators can consistently recover mechanism-level causal effects and their variances from a single randomized experiment, overcoming the limitations of classical finite-population inference.

Original authors: Yukai Yang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yukai Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist trying to understand how a new fertilizer affects plant growth.

The Old Way: The "Frozen World"

In the traditional way of doing experiments (called the Fixed Potential Outcome framework), researchers imagine a "frozen world." They assume that if they had planted the same seeds in the same soil yesterday, today, and tomorrow, the plants would have reacted exactly the same way every time. The only thing that changes is which plants get the fertilizer and which get water.

In this frozen world, the goal is to measure the difference between the plants that got fertilizer and those that didn't in this specific experiment. If you want to know what would happen in a different world (maybe with slightly different weather or soil), the old method says, "We can't know that. We only have data from this one frozen day."

Furthermore, when the old method calculates how "sure" we are about our results (the variance), it has to be very pessimistic. It assumes the worst-case scenario because it can't account for the fact that the plants might have reacted differently in other possible worlds. It gives you a "safe" answer, but it's often too wide to be very useful.

The New Idea: The "Living, Breathing World"

This paper, by Yukai Yang, proposes a different way of thinking. Instead of a frozen world, imagine the experiment happens in a living, breathing world where the environment is slightly different every time you run it.

The Analogy: Imagine you are testing a new recipe for bread.
- Old Way: You bake one loaf today. You assume that if you baked it again, it would taste exactly the same. You only care about this loaf.
- New Way: You acknowledge that every time you bake, the humidity, the flour batch, and the oven temperature might vary slightly. You want to know the average quality of the bread you would get if you baked it a million times in this kitchen, not just the quality of the one loaf you have right now.

The paper calls this the Random Potential Outcomes (RPO) framework. It treats the "environment" (weather, soil, oven temp) as a random variable that changes.

The Big Problem: One Snapshot vs. The Whole Movie

Here is the tricky part: Even though we want to know the average result across all possible worlds (the whole movie), we can only observe one single experiment (one snapshot).

If every single plant in your garden was affected by the exact same weird weather event (a single global shock), looking at 1,000 plants wouldn't help you figure out what the "average" weather effect is. You'd still only know about that one specific day.

The Solution: "Local Neighborhoods"

The paper's breakthrough is a condition called Local Dependence.

The Analogy: Imagine a city where people influence their immediate neighbors, but not people across town.
- If you live in a neighborhood, your mood might be affected by your neighbor's loud party.
- But your mood is not affected by a party happening in a different city.
- If you have a huge city with many small neighborhoods, and you look at enough people, you can figure out the "average" effect of parties on mood, even if you only observed the city on one specific night.

The paper proves that if the "influence" (dependence) between units is sparse (limited to small local groups) and doesn't spread to the whole group at once, you can use the average of the single experiment to accurately guess the average of all possible worlds.

It's like listening to a crowd. If everyone is shouting the exact same random noise (global dependence), you can't tell what the average conversation is. But if people are only chatting in small, separate groups (local dependence), and you have enough groups, you can accurately estimate the average conversation of the whole crowd just by listening to one moment in time.

The Magic Result: Better Confidence

Because the paper allows us to treat the outcome as a "living mechanism" rather than a "frozen schedule," it solves a major headache in statistics: Variance Estimation.

Old Way: "We are 95% sure the effect is between 0 and 10." (This is a huge, useless range because the method is forced to be conservative).
New Way: "We are 95% sure the effect is between 4.2 and 4.8."

By understanding the local structure of how units depend on each other, the paper shows we can calculate a precise measure of uncertainty from just one experiment. We don't have to guess the worst-case scenario anymore; we can actually measure the true variability of the mechanism.

Summary

The Shift: Move from thinking of experiments as "frozen snapshots" to "one realization of a living, random process."
The Goal: We want to know the average effect of a mechanism across all possible environments, not just the one we saw.
The Catch: Usually, one experiment isn't enough to guess the average of all possibilities.
The Fix: If the "randomness" is local (neighbors affect neighbors, but not the whole world), then averaging across the people in your single experiment is enough to guess the global average.
The Benefit: This allows researchers to get much more precise and accurate confidence intervals from a single experiment, rather than being stuck with overly cautious, wide guesses.

The paper uses math (specifically "Riesz representers" and "dependency graphs") to prove this works, and simulations show that in practice, it gives the right answers about 95% of the time, just like a good statistical method should.

Technical Summary: Design-Based Inference under Random Potential Outcomes

Problem Statement
The paper addresses a fundamental limitation in classical design-based causal inference, specifically the Fixed Potential Outcome (FPO) framework established by Neyman (1923/1990) and Rubin (1974). In the FPO framework, potential outcomes are treated as fixed quantities, and randomness arises solely from the treatment assignment mechanism. Consequently, causal estimands are defined conditionally on a single realized experimental environment. While this allows for identification of treatment effects within that specific realization, it does not guarantee the recoverability of mechanism-level quantities—defined as expectations over latent stochastic environments—from a single realized experiment.

The central inferential problem is that identification alone is insufficient for recoverability. If all units share a common latent shock, increasing the sample size provides no additional information about the expectation over the latent environment. The paper seeks to determine whether, and under what conditions, cross-sectional averaging across units in a single experiment can consistently recover expectations defined over the underlying stochastic mechanism generating the outcomes.

Methodology and Framework
The author proposes a Random Potential Outcome (RPO) framework where potential outcomes are modeled as random elements indexed by a latent stochastic environment $\omega$ .

Model Specification:
- Potential outcomes are represented as measurable mappings $\tilde{y}_i(z, \omega) = y_i(z, x_i(\omega), \epsilon_i(\omega))$ , where $z$ is the treatment vector and $\omega$ is a latent random element on a common probability space.
- Causal estimands are defined as expectations over $\omega$ (e.g., $\tau_{RPO} = E_\omega[\frac{1}{n}\sum (\tilde{y}_i(1, \omega) - \tilde{y}_i(0, \omega))]$ ), rather than conditional on a fixed schedule.
- Treatment effects are formalized as continuous linear functionals on a model space $M_i \subset L^2(Z \times \Omega)$ . By the Riesz Representation Theorem, these functionals admit a unique representer $\psi_i$ , allowing the treatment effect to be expressed as an inner product $\theta_i(\tilde{y}_i) = \langle \tilde{y}_i, \psi_i \rangle$ .
Estimation Strategy:
- The paper utilizes Riesz estimators ( $\hat{\theta}_i = \tilde{y}_i \psi_i$ ) which are unbiased for the mechanism-level estimand under the joint randomness of treatment assignment and the latent environment.
- An Aggregate Riesz Estimator is constructed as a weighted sum of individual Riesz estimators.
Recoverability Conditions (Local Dependence):
- To bridge the gap between cross-sectional averaging and averaging over latent environments, the paper introduces local dependence conditions modeled via dependency neighborhoods (following Chen and Shao, 2004; Ross, 2011).
- Assumption 6 & 7: The maximum and average dependency neighborhood sizes must grow sufficiently slowly relative to the sample size $n$ (specifically $D_n = o(n^d)$ with $d \leq 1/4$ for asymptotic normality).
- These conditions induce an ergodic-type property, ensuring that cross-sectional averaging asymptotically recovers the expectation over the latent stochastic environment from a single realized experiment.
Variance Estimation:
- Unlike the FPO framework, where Neyman-type variance estimators are structurally limited to conservative upper bounds, the RPO framework permits consistent variance estimation.
- The proposed variance estimators exploit the sparsity of the dependency structure. By summing cross-products of centered residuals only over pairs within the dependency neighborhoods (or correlated pairs), the estimator consistently recovers the true sampling variance associated with the stochastic mechanism.

Key Results

Consistency and Recoverability: Under local dependence and boundedness assumptions, the aggregate Riesz estimator is consistent in mean square for the mechanism-level estimand. Cross-sectional averaging asymptotically substitutes for averaging over the latent environment.
Asymptotic Normality: The estimator satisfies a Central Limit Theorem ( $\sigma_n^{-1}(\hat{\tau}_n - \tau_n) \xrightarrow{d} N(0,1)$ ) provided the dependency neighborhoods are sufficiently sparse ( $d \leq 1/4$ ) and finite fourth moments exist.
Consistent Variance Estimation: The paper establishes that feasible variance estimators, constructed based on local dependency structures, are consistent for the true sampling variance. This overcomes the structural limitation of the FPO framework where only conservative bounds are available.
Simulation Evidence: Monte Carlo simulations confirm that the proposed procedure maintains accurate size control and empirical coverage close to nominal levels (e.g., 95%) under various dependency growth rates and sample sizes, provided the theoretical sparsity conditions are met. The method also demonstrates high statistical power.

Significance and Claims
The paper claims to reframe design-based inference under RPO as a recoverability problem for mechanism-level causal quantities. Its primary contributions are:

Theoretical Bridge: It demonstrates that suitably sparse local dependence induces an ergodic property, allowing a single realized experiment to yield asymptotically valid inference for mechanism-level estimands.
Variance Estimation: It resolves the issue of inconsistent variance estimation in design-based settings by shifting the inferential target from fixed schedules to stochastic mechanisms. This allows for consistent variance estimation rather than relying on conservative upper bounds.
Distinction from Existing Frameworks: The work distinguishes itself from model-assisted approaches (which rely on outcome modeling) and sampling-based formulations (which address population-level quantities under sampling uncertainty). Instead, it relies solely on the randomization design and the structure of the latent environment to achieve recoverability.

The paper concludes that while identification is a prerequisite, recoverability from a single experiment is only possible under specific probabilistic structures (local dependence) induced by the experimental design. The proposed framework enables asymptotically valid inference for mechanism-level quantities without requiring repeated experimental replications.