📊 statistics

Untangling Sample and Population Level Estimands in Bayesian Causal Inference

This paper clarifies the critical distinctions between sample and population-level causal estimands in Bayesian inference to prevent common implementation errors, demonstrating through four detailed Stan examples how to correctly apply first-principles thinking and Bayes' theorem across identification, modeling, and computation.

Original authors: Arman Oganisian

Published 2026-03-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Arman Oganisian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Who" Matters

Imagine you are a doctor trying to figure out if a new medicine works. You have a group of patients in your clinic (your Sample). You want to know two things:

The Specifics: "How does this drug work for these specific 50 people sitting in my waiting room right now?"
The General Rule: "How does this drug work for all people in the world who might have this condition?"

The paper argues that many researchers are accidentally mixing these two questions up. They think they are answering the "General Rule" question, but they are actually only answering the "Specifics" question. This leads to wrong conclusions and overconfident predictions.

The author, Arman Oganisian, says: "Stop guessing. Be precise about who you are talking about."

The Core Confusion: The "Missing Puzzle Piece"

To understand why this is hard, imagine a puzzle.

The Factual Piece: You see what happened to a patient who took the drug. (We have this piece).
The Counterfactual Piece: You don't see what would have happened to that same patient if they had taken the sugar pill instead. (This piece is missing).

In statistics, we call this missing piece the Counterfactual.

Sample-Level Inference (The "Who's in the Room" approach): You want to know the average effect for the specific people you have. To do this, you have to guess (impute) the missing puzzle piece for every single person in your room. You have to imagine, "If Person A took the pill, they got better. But what if they took the sugar pill? Would they have gotten better? Worse?"
- The Trap: To guess this, you have to make a wild assumption about how the "Pill World" and the "Sugar Pill World" are connected. The paper calls this a "Cross-World Assumption." It's like assuming that if you jumped off a cliff in one universe, you would land in a specific spot in a parallel universe. It's impossible to prove, but you have to assume it to do the math.
Population-Level Inference (The "General Rule" approach): You want to know the average effect for the whole world. You don't care about the specific missing piece for Person A. You only care about the average shape of the puzzle pieces for the whole population.
- The Benefit: You don't need to guess the missing piece for every single person. You just need to understand the general rules of the population. You can ignore the "Cross-World" connection because you are averaging it out.

The Four Main Mistakes People Make

The paper highlights four ways people get tripped up:

1. The "One Size Fits All" Mistake (Interpretation)

Analogy: Imagine you measure the height of 10 specific students in a classroom.
- Sample Effect: "The average height of these 10 kids is 5'4"."
- Population Effect: "The average height of all kids in the city is 5'2"."
The Error: If you calculate the average of your 10 kids but claim it represents the whole city, you might be wrong. The paper shows that the "uncertainty" (how much you might be wrong) is totally different for the 10 kids vs. the whole city. The city estimate is usually fuzzier because the city is bigger and more diverse.

2. The "Magic Assumption" Mistake (Cross-World Modeling)

Analogy: To guess what Person A would have done if they took the sugar pill, you have to assume a magical link between their "Drug Self" and their "Sugar Self."
The Error: For the "Specifics" (Sample), you must make this magical assumption. For the "General Rule" (Population), you usually don't have to. Many researchers try to calculate the "General Rule" but accidentally use the "Magical Assumption" meant for the "Specifics." This makes their results look more certain than they really are.

3. The "Blind Spot" Mistake (Uncertainty)

Analogy: If you guess the missing puzzle piece for one person, you are very unsure. If you guess the average for a million people, the errors cancel out, and you are more sure.
The Error: If you treat the "Specifics" (which are very uncertain) as if they are the "General Rule" (which should be more certain), you will draw the wrong conclusions. Your "Confidence Interval" (your safety net) will be too narrow, and you might think you know more than you do.

4. The "Shortcut" Mistake (Computation)

Analogy: Imagine you want to know the average weight of all apples in a warehouse.
- The Right Way (Population): You take a few samples, build a model of the warehouse, and simulate the average weight of new imaginary apples based on that model.
- The Wrong Way (Sample/Shortcut): You just take the 50 apples you have, weigh them, and say, "This is the average for the whole warehouse."
The Error: The paper critiques a popular method (the Bayesian G-Formula) where researchers take their specific sample, simulate new outcomes for those same people, and call it a "Population Average." It's not. It's just a fancy way of describing the specific people you already have.

The "Stan" Code and the "Black Box"

The author uses a computer program called Stan to show exactly how to do the math correctly.

The Old Way: Researchers would put the missing puzzle pieces (counterfactuals) into a "black box" at the end of the code, pretending they were just numbers to be averaged.
The New Way: The author says, "If you want to know about the specific people, you must declare those missing pieces as unknown variables right at the start of the code." If you want to know about the population, you must integrate them out (mathematically remove them) and focus only on the population rules.

The "Recipe" for Success

The paper ends with a simple recipe for anyone doing this kind of research:

Ask "Who?": Before you write a single line of code, ask: "Am I trying to learn about these specific people or everyone like them?"
Check the Ingredients:
- If it's Specific People: You must model the missing pieces for everyone. Be ready to make "Cross-World" assumptions (and admit they are assumptions).
- If it's Everyone: You don't need to guess the missing pieces for individuals. You just need to model the population distribution.
Follow the Math: Don't take shortcuts. If you want the population average, don't just average the specific people you have. Simulate new people based on your model.

Summary

Think of this paper as a warning label on a complex machine. It says: "Don't confuse the specific parts you are holding with the whole machine they came from."

If you want to know how a machine works for everyone, don't just tinker with the one you have in your hand. Build a model of the factory that made it. If you mix them up, your machine might work for you, but it will explode for everyone else.

1. Problem Statement

The paper addresses a critical and frequently overlooked source of error in Bayesian causal inference: the conflation of sample-level estimands (e.g., Individual Treatment Effect [ITE], Sample Average Treatment Effect [SATE]) and population-level estimands (e.g., Conditional Average Treatment Effect [CATE], Population Average Treatment Effect [PATE]).

While these concepts appear similar, they differ fundamentally in:

Target of Inference: Sample-level estimands focus on the specific units observed in the dataset ( $D_O$ ), whereas population-level estimands target the average effect over a super-population distribution from which the sample was drawn.
Identification Assumptions: Sample-level inference often requires cross-world assumptions (assumptions about the dependence between potential outcomes $Y(1)$ and $Y(0)$ for the same unit, which are never jointly observed). Population-level inference often does not require these assumptions if the counterfactuals are integrated out.
Computational Implementation: Errors arise when researchers use posterior draws intended for one estimand (e.g., SATE) to make inferences about another (e.g., PATE), or when they fail to propagate uncertainty in the covariate distribution correctly (e.g., confusing the Mixed Average Treatment Effect [MATE] with the PATE).

2. Methodology and Framework

The author utilizes the Rubin Causal Model (RCM) within a Bayesian framework. The core methodology involves a rigorous derivation of the joint posterior distribution of all unknowns (parameters $\omega$ and missing counterfactuals $Y^M$ ) given observed data $D_O$ .

Key Theoretical Components:

Joint Posterior: $f(Y^M, \omega | D_O) \propto f(D_C; \omega)f(\omega)$ , where $D_C$ is the complete data (observed + counterfactuals).
Marginalization Logic:
- Sample-level (ITE/SATE): Requires the marginal posterior of the counterfactuals $f(Y^M | D_O)$ . This necessitates imputing missing counterfactuals from the joint distribution, which depends on the non-identifiable cross-world dependence parameter (e.g., correlation $\rho$ ).
- Population-level (CATE/PATE): Requires the marginal posterior of the structural parameters $f(\omega | D_O)$ . The missing counterfactuals are integrated out. For PATE, this further requires integrating the CATE over the covariate distribution $f(L)$ .
Cross-World Assumptions: The paper highlights that sample-level inference typically requires assuming a specific dependence structure (e.g., $\rho$ ) between $Y(1)$ and $Y(0)$ . Population-level estimands like PATE often do not require specifying this dependence if the goal is only the marginal mean, provided the model for the marginals is correct.

Computational Approach:

Stan Implementation: The paper provides detailed Stan code examples to demonstrate correct implementation.
Bayesian Bootstrap (BB): To avoid strong parametric assumptions about the covariate distribution $f(L)$ , the author advocates for the Bayesian Bootstrap. This propagates uncertainty in the covariate distribution, unlike the empirical distribution (which fixes weights at $1/n$ ) and avoids the "MATE" trap.
Monte Carlo Integration: For complex population-level estimands (e.g., probabilities of joint outcomes), the paper demonstrates using Monte Carlo simulation to approximate integrals over the joint potential outcome distribution, distinct from simple posterior imputation.

3. Key Contributions

The paper makes four primary contributions:

Conceptual Clarification: It explicitly delineates the mathematical and conceptual differences between sample and population estimands, emphasizing that they are functionals of different unknowns (counterfactuals vs. parameters).
Identification of Common Errors:
- The MATE vs. PATE Confusion: The paper critiques the practice of using the empirical distribution (fixed weights) to estimate PATE. It shows this yields the Mixed Average Treatment Effect (MATE), which underestimates uncertainty because it ignores the variability of the covariate distribution.
- Cross-World Confusion: It demonstrates that treating posterior draws of ITEs as draws of CATEs leads to incorrect uncertainty quantification (ITE uncertainty is typically much larger).
- Keil et al. (2018) Critique: The author provides a detailed critique of the "Bayesian G-Formula" implementation by Keil et al., arguing that their computational procedure (sampling covariates from the empirical distribution and averaging differences) actually targets a finite-sample quantity (SATE-like) rather than the population-level PATE they claimed, and fails to account for covariate distribution uncertainty.
Flexible Modeling with Nonparametrics: The paper extends these concepts to Bayesian Nonparametric (BNP) models, specifically Truncated Dirichlet Process Mixtures (DPM). It shows how to correctly derive CATE and PATE from DPMs, highlighting that while CATEs can be derived directly from mixture weights, PATEs require an additional Monte Carlo integration step over the mixture covariate distribution.
Handling Multi-Treatment and Time-Varying Scenarios: The paper illustrates that as the number of treatments ( $K$ ) increases, the number of non-identifiable cross-world parameters grows polynomially ( $K(K-1)/2$ ). This makes sample-level inference intractable without strong, unfalsifiable assumptions, whereas population-level inference remains more feasible.

4. Results and Illustrative Examples

The paper uses four detailed examples (implemented in Stan) to demonstrate these points:

Example 1 (Simple Parametric): A bivariate normal model comparing SATE and PATE.
- Result: The posterior means for SATE and PATE are similar, but the posterior uncertainty (variance) for PATE is significantly larger because it includes uncertainty from the covariate distribution. Using SATE draws for PATE inference yields overly narrow credible intervals.
Example 2 (Flexible Covariate Models): Comparing "Exact" PATE (parametric), Bayesian Bootstrap PATE, and MATE (empirical distribution).
- Result: The MATE (using fixed empirical weights) has a similar center to the true PATE but severely underestimates uncertainty. The Bayesian Bootstrap correctly propagates covariate distribution uncertainty.
Example 3 (Flexible BNP & Cross-World): Using Truncated DPMs to estimate CATE vs. ITE.
- Result: Demonstrates that even with flexible models, one must not confuse draws of the ITE (which require imputing $Y^M$ ) with draws of the CATE (which rely on parameter means).
Example 4 (Population Estimands Requiring Cross-World): Estimating the probability of joint failure $P(Y(1)>\tau, Y(0)>\tau)$ $P (Y (1) > τ, Y (0) > τ)$ .
- Result: Shows that some population-level estimands do require cross-world assumptions. The correct approach involves Monte Carlo integration over the joint distribution using the posterior of parameters, rather than simple imputation of observed units.
Example 5 (Multi-Treatment): Extending to $K > 2$ $K > 2$ treatments.
- Result: Highlights the intractability of sample-level inference due to the explosion of non-identifiable correlation parameters, reinforcing the preference for population-level estimands in complex settings.

5. Significance and Conclusion

The paper serves as a technical sequel to previous works by Ding & Li (2018) and Li et al. (2023), moving from conceptual overviews to implementation-level rigor.

Practical Impact: It provides a checklist for practitioners to avoid common implementation errors in Bayesian causal inference, particularly regarding the choice of estimand and the propagation of uncertainty.
Methodological Guidance: It advocates for "first-principles thinking":
1. Define the substantive estimand (Sample vs. Population).
2. Identify the unknowns it depends on.
3. Derive the specific marginal posterior required.
4. Implement strictly according to Bayes' theorem, ensuring cross-world assumptions are made explicit and only when necessary.
Recommendation: The author concludes that while sample-level inference is sometimes necessary (e.g., for specific finite populations like a single city), population-level inference is generally preferred for generalizability. However, if population-level inference is chosen, researchers must use methods (like the Bayesian Bootstrap) that properly account for covariate distribution uncertainty and avoid conflating SATE/MATE with PATE.

In summary, the paper argues that correct implementation is not just about model flexibility, but about correctly identifying which marginal of the joint posterior distribution corresponds to the scientific question. Failure to do so leads to invalid inferences, even when using sophisticated Bayesian models.