📊 statistics

Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?

The paper proposes a new framework called the Treatment Reactive Average Causal Effect (TRACE) to identify and estimate the causal impact of a treatment on specific subgroups defined by their potential post-treatment responses, thereby avoiding the biases and indefensible assumptions associated with conditioning on observed post-treatment variables.

Original authors: Chad Hazlett, Nina McMurry, Tanvi Shinkre

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chad Hazlett, Nina McMurry, Tanvi Shinkre

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Half-Baked Cake" Dilemma

Imagine you are a baker testing a new recipe for a chocolate cake. You give the new recipe to one group of people (the Treatment Group) and the old recipe to another (the Control Group).

After the bake, you notice the new cake tastes... okay. Not amazing, but not terrible. You wonder: "Was the recipe actually bad, or did some people just forget to add the cocoa powder?"

If you only look at the people who actually added the cocoa (the "Implementers"), you run into a scientific trap. Maybe the people who forgot the cocoa were also the ones who didn't preheat their ovens. If you only study the "good" bakers, you might accidentally conclude the recipe is magic, when really, you’ve just ignored all the people who messed up the process. This is called "Post-Treatment Bias."

In the real world, this happens in politics, policing, and medicine. If a government starts a new community program, but half the towns don't actually follow the instructions, a standard study will say, "The program didn't work." But that might be a lie! The program might be amazing—it just wasn't actually done in most places.

The Solution: The TRACE Method

The authors of this paper propose a new way to measure success called TRACE (Treatment Reactive Average Causal Effect).

Instead of asking, "Did the program work on average?" (which includes all the people who messed up), TRACE asks: "If we could go back in time and ensure that everyone who was supposed to follow the instructions actually did, what would the effect have been?"

Think of it like a "What If" simulator.

How it works (The "Two-Room" Analogy)

Imagine two rooms.

Room A is the "Success Room" (people who would react to the treatment).
Room B is the "Non-Reactive Room" (people who wouldn't change even if they got the treatment).

The researchers know the total average (the "Intent-to-Treat"). They also know how many people ended up in the Success Room. The only thing they don't know is exactly what happened in the Non-Reactive Room.

Since they can't see into the future, they use "Bounded Inference." This is like saying: "I don't know exactly how much the Non-Reactive group changed, but I'm 99% sure they didn't change by more than 5%. Based on that, the real effect of the treatment must be somewhere between X and Y."

Real-World Examples from the Paper

To prove this works, the authors applied it to three very different scenarios:

1. The Police Stop (The "Necessary Condition")
If you want to study if a police officer's perception of race leads to violence, you have a problem: violence can only happen during a traffic stop. If there is no stop, there is no violence.

The TRACE approach allows researchers to say: "Since violence is impossible if no stop occurs, we can mathematically 'zoom in' on the effect specifically during the stops, giving us a much clearer picture of racial bias."

2. The Liberia Policing Study (The "Implementation Gap")
A program was launched in Liberia to reduce mob violence. On average, it looked like it did nothing.

The TRACE approach showed that in communities where the program was actually implemented (where people formed security groups), the violence dropped significantly! The "failure" of the program wasn't because the idea was bad, but because the implementation was patchy.

3. The Transgender Rights Study (The "Mechanism")
Researchers wanted to know if a 10-minute conversation could change support for transgender rights. They noticed the conversation worked best on people who already felt a "warmth" toward the topic.

The TRACE approach helped them calculate the effect specifically for those "reactive" people, helping them understand the mechanism of how empathy changes policy support.

The Bottom Line

The TRACE method is like a magnifying glass for researchers.

Instead of looking at a blurry, averaged-out mess of "people who tried" and "people who didn't," it allows scientists to say: "Let's look at the people who actually engaged with the treatment. Even if we can't be 100% certain of the exact number, we can give you a very reliable range of how much this actually works when it's done right."

Technical Summary: Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?

1. The Problem: Post-Treatment Bias and Sub-group Interest

In many experimental settings, researchers are not interested in the average effect across the entire population (Intent-to-Treat), but rather in the effect among a specific sub-group. This sub-group is often defined by a post-treatment variable ( $M$ )—a variable that is influenced by the treatment ( $D$ ) and subsequently affects the outcome ( $Y$ ). Examples include:

Implementation/Compliance: The effect among those who actually implemented a program or complied with a protocol.
Mechanisms: The effect among those who demonstrated a specific psychological or behavioral response (e.g., increased warmth toward a group).
Necessary Conditions: The effect among those for whom a specific event (like a police stop) occurred, which is a prerequisite for the outcome (like violence during a stop).

The core challenge: Simply conditioning on the observed value of $M$ (e.g., comparing treated vs. untreated units where $M=1$ ) introduces two major issues:

Selection Bias/Collider Bias: Since $M$ is a consequence of both $D$ and potentially unobserved confounders ( $U$ ), conditioning on $M$ creates a non-causal association between $D$ and $U$ , biasing the estimate.
Identification Failure: In randomized trials, we cannot observe the counterfactual $M$ for the control group. We do not know which control units would have responded to the treatment if they had received it.

2. Methodology: The TRACE Estimand

The authors propose a new estimand: the Treatment Reactive Average Causal Effect (TRACE).

Definition:
The TRACE is the total effect of treatment on the outcome for the sub-group that, if treated, would realize a specific value of the post-treatment variable ( $M=1$ ).
$\text{TRACE} \equiv E[Y_i(1) - Y_i(0) \mid M_i(1) = 1]$

Identification Strategy:
The paper acknowledges that TRACE is generally not point-identified due to the unobserved $M(0)$ for the treated group and the unobserved $M(1)$ for the control group. They propose a two-step identification approach:

Step 1: No-Assumption Trimming Bounds: Using only the properties of randomization, the authors derive "sharp" bounds. The upper and lower bounds are determined by the most extreme possible distributions of the control group's outcomes.
Step 2: Leveraging Assumptions on $\text{TRACE}(0)$ : To narrow these bounds, the authors allow researchers to make defensible, qualitative assumptions about $\text{TRACE}(0)$ $TRACE (0)$ —the effect among those who would not respond to the treatment ( $M(1)=0$ $M (1) = 0$ ). Common assumptions include:
- $\text{TRACE}(0) = 0$ (The treatment has no effect if the mechanism isn't activated).
- $|\text{TRACE}(0)| < |\text{TRACE}|$ (The effect is smaller in the non-reactive group).
- $\text{TRACE}(0)$ has the opposite sign of $\text{TRACE}$ (e.g., a "backlash" effect).

By algebraically linking $\text{TRACE}$ and $\text{TRACE}(0)$ via the Law of Iterated Expectations, any reasoned bound on the non-reactive group's effect translates into a tighter, informative bound on the TRACE.

3. Key Contributions

New Estimand: Distinguishes TRACE from existing quantities like LATE (which requires the exclusion restriction), SACE (which focuses on always-takers), and CDE (which focuses on direct effects).
Relaxing the Exclusion Restriction: Unlike Instrumental Variable (IV) approaches, TRACE targets the total effect, meaning it explicitly includes the "direct effect" of the treatment that might bypass the mediator.
Principled Partial Identification: Provides a framework for "bounded inference" that moves beyond the binary of "identifiable vs. non-identifiable." It allows researchers to be transparent about how much they must believe about the non-reactive group to support a specific conclusion about the reactive group.

4. Results and Applications

The authors demonstrate the utility of TRACE through three diverse cases:

Police Violence: In studies of police-perceived race, violence can only occur if a stop ( $M=1$ ) occurs. By assuming $\text{TRACE}(0)=0$ (no violence if no stop occurs), the authors show that the effect of perceived race on violence can be point-identified, solving a long-standing problem in administrative data bias.
Community Policing in Liberia: In a trial where implementation was uneven, the ITT was null. However, by assuming the intervention only works when implemented ( $\text{TRACE}(0)=0$ ), the TRACE revealed a significant reduction in mob violence among "implementing" communities.
Transgender Rights Advocacy: Re-analyzing a famous field experiment, the authors showed that while the average effect on policy support was modest, the effect among "reactive" individuals (those whose feelings improved) was substantially larger, providing a clearer picture of the mechanism.

5. Significance

This paper is significant for social science and policy evaluation because it provides a mathematical way to handle "failed" or "diluted" treatments. Instead of dismissing a null result as a failure of the intervention, researchers can use TRACE to determine if the intervention was effective for those it actually reached, while simultaneously providing a rigorous way to test whether a null result is truly due to a lack of effect or merely a lack of implementation.