📊 statistics

Treatment effect: a critique

This paper critiques the distinction between model-based and counterfactual definitions of treatment effects, arguing that while counterfactual frameworks offer idealized clarity, model-free definitions may be unsuitable for generalizing scientific conclusions beyond the observed sample, echoing concerns raised by Cox.

Original authors: Heather Battey, Charlotte Edgar

Published 2026-01-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heather Battey, Charlotte Edgar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to figure out if adding a secret spice (the "treatment") makes a soup taste better. You want to know exactly how much better it is. This paper is a debate between two groups of statisticians about the best way to measure that "betterness."

Here is the breakdown of the paper's argument, using simple analogies.

The Two Competing Recipes

The authors, Battey and Edgar, are comparing two different ways to define a "treatment effect" (the impact of the spice).

1. The "Model-Based" Recipe (The Fisherian Approach)

The Idea: This approach assumes there is a hidden, stable rule governing how the soup changes. Think of it like a physics law. If you add the spice, the soup's flavor shifts by a specific, consistent amount (like adding exactly 2 grams of salt).
The Goal: To find that stable "rule" or parameter. Even if you cook the soup with different base ingredients (different people), the change caused by the spice remains constant according to the rule.
The Analogy: Imagine a conveyor belt where every box gets a sticker added. The "treatment effect" is the size of the sticker. It's the same size for every box, regardless of what's inside the box.

2. The "Model-Free" Recipe (The Counterfactual Approach)

The Idea: This approach doesn't assume a hidden rule. Instead, it asks a "What if?" question for every single person. "What would Person A have tasted if they didn't get the spice, compared to what they did taste?"
The Goal: To calculate the average difference between the "real world" and the "imaginary world" for everyone in the group.
The Analogy: Imagine you have 100 clones of yourself. You give the spice to one clone and not the other. Then you do it again for a different set of clones. You take the average of all those differences. This is the "Model-Free" method.

The Big Problem: The "Chameleon" Effect

The authors argue that the "Model-Free" approach has a major flaw: It changes depending on who you ask.

To prove this, they use a thought experiment (a "fictitious idealization"). They pretend we have a magic machine that lets us see both the "spiced" and "unspiced" versions of every person at the same time. Even with this perfect information, the Model-Free average behaves strangely.

The Analogy of the Unstable Ruler:
Imagine you are measuring the height of a group of people to see how much a growth hormone (the treatment) helps them.

The Model-Based view says: "The hormone makes everyone grow exactly 2 inches." This is a stable fact.
The Model-Free view calculates the average growth.
- If your group consists of very short people, the hormone might make them grow 2 inches, which is a huge percentage increase.
- If your group consists of very tall people, the same 2 inches might be a tiny percentage increase.
- The Result: The "average effect" calculated by the Model-Free method changes completely just because you picked a different group of people to measure.

The authors show that in many real-world scenarios (like binary outcomes like "sick" vs. "healthy"), the Model-Free average depends entirely on the specific mix of people in your sample. If you change the sample, the "answer" changes.

Why This Matters

The authors argue that if you want to make scientific conclusions that apply to the world at large (not just the specific people in your study), you need a stable definition.

The Critique: The Model-Free approach is like trying to measure the "average speed" of a car by looking at how fast it goes on different roads. If the road is steep, it goes slow; if it's flat, it goes fast. The "speed" isn't a property of the car; it's a property of the specific road you happened to drive on.
The Consequence: If you use the Model-Free method, your conclusion about the treatment might be true for your specific group of patients, but it might be completely wrong for the next group of patients you see. It doesn't generalize well.

Addressing the Counter-Arguments

The paper also tackles three common defenses of the Model-Free approach:

"But people are different!"
- Defense: The Model-Free method is better because it lets different people have different effects.
- Rebuttal: The Model-Based method can handle this too! It just treats the differences as "interactions" (like saying the spice works differently on spicy soup vs. bland soup) while keeping the core rule stable.
"We don't want to assume a model!"
- Defense: We shouldn't guess the rule; we should just measure the difference.
- Rebuttal: If you don't assume a rule, you lose the ability to say anything stable about the future. You end up with a number that only works for the specific people you measured today.
"It has 'Double Robustness'!"
- Defense: The math is clever; it works even if one part of the calculation is wrong.
- Rebuttal: This is only useful if the thing you are measuring (the average difference) is actually a meaningful, stable target. If the target itself is wobbly (like the unstable ruler), being "robust" doesn't help you hit the bullseye.

The Bottom Line

The authors conclude that while the "Model-Free" (counterfactual) approach is popular and mathematically clever, it often fails to provide a stable definition of a treatment effect that can be generalized to new people.

They argue that we should return to the "Model-Based" approach (associated with R.A. Fisher), where we look for a stable rule or parameter that explains how the treatment works, rather than just averaging up the differences in a specific, potentially unrepresentative group.

In short: Don't just average the differences you see in your specific group. Find the underlying rule that explains the change, so your conclusions hold true for the next group you meet.

Technical Summary: Treatment Effect: A Critique

Problem Statement
The paper addresses a fundamental divergence in the definition of "treatment effect" within statistics. One camp, rooted in the Fisherian tradition, defines treatment effect as a stable parameter within a statistical model (e.g., a regression coefficient). The opposing camp, associated with the Rubin causal model and counterfactual frameworks (Rubin, 1974; Rosenbaum and Rubin, 1983), defines treatment effect as a model-free population-level difference in potential outcomes (or counterfactuals). While the literature has extensively debated the identifiability and estimation of counterfactual effects, this paper questions whether the model-free definition (specifically the average difference in potential outcomes) constitutes an appropriate target for inference when the goal is scientific generalization beyond the observed sample. The authors argue that model-free definitions may lack stability and scientific coherence even under idealized conditions where counterfactuals are observable.

Methodology
The authors employ a comparative theoretical analysis using "fictitious idealization." To isolate the issue of what is being measured from the issue of how it is estimated (i.e., untestable assumptions about unobservables), they suppose a scenario where the counterfactual process is completely known, and both potential outcomes ( $Y_i(0)$ and $Y_i(1)$ ) are observable for every individual. In this idealized setting, the problem reduces to a matched-pair design where each individual is twinned with a perfect copy.

The authors then compare the "model-free" treatment effect, defined as the sample average of individual differences:
$\tau_n := \frac{1}{n} \sum_{i=1}^n E_i(Y_i(1) - Y_i(0))$
against the "model-based" treatment effect, defined as a stable parameter $\theta$ (or $\Delta$ ) governing a transformation of the probability distribution of the outcome (following McCullagh, 2002, 2022).

They analyze five specific examples to test the compatibility of these two definitions:

Linear Model: Additive effects on the mean.
Exponential Model: Multiplicative effects on the rate.
Weibull Model: Multiplicative effects on the rate with a shape parameter $\alpha$ .
Exponential Model (Additive Rate): Additive effects on the rate.
Logistic Model: Additive effects on the log-odds (binary outcomes).

Key Contributions and Results
The analysis yields the following technical findings:

Compatibility in Transformation Families: In cases where the treatment effect acts as a group transformation on the baseline distribution parameters (e.g., additive shifts in a linear model or multiplicative shifts in an exponential rate), the model-free average $\tau_n$ recovers the model-based parameter (or a simple transformation of it).
Instability in Non-Transformation Families: In cases where the treatment effect is defined stably in the model (e.g., an additive shift $\Delta$ $Δ$ to an exponential rate, or an additive shift $\Delta$ $Δ$ to a logistic log-odds), the model-free average $\tau_n$ $τ_{n}$ fails to recover a stable parameter. Instead, $\tau_n$ $τ_{n}$ becomes a complex function of the specific baseline parameters ( $\gamma_i$ $γ_{i}$ ) of the sampled individuals.
- Example: In the additive exponential rate model, $\tau_n = \frac{1}{n} \sum \log((\gamma_i + \Delta)/\gamma_i)$ . This value varies drastically depending on the distribution of $\gamma_i$ in the sample. If $\gamma_i$ are small, $\tau_n$ is large; if $\gamma_i$ are large, $\tau_n$ is tiny.
- Example: In the logistic model, $\tau_n$ depends on the specific values of the linear predictors $\gamma_i$ for the sampled units.
Necessary Conditions for Equivalence: The authors derive three criteria under which the model-free and model-based definitions yield identical answers for any sample:
1. The treatment effect acts on the baseline distribution via the baseline parameter space.
2. The group action is consistent on the observations.
3. No nuisance parameters other than the baseline parameter are needed to characterize the relevant expectation.
  Violation of these conditions (as seen in the last two examples) renders the model-free target dependent on the specific sample chosen, violating the principle of stable parameters.

Significance and Claims
The paper claims that the instability of the model-free treatment effect $\tau_n$ is a critical flaw when scientific conclusions are intended to generalize. The authors echo concerns raised by Cox (1958b, 2012) regarding the suitability of model-free averages as targets of inference.

Generalizability: Because $\tau_n$ depends on the specific distribution of baseline characteristics in the sample, conclusions drawn from one study may not generalize to a different population with a different distribution of $\gamma_i$ , even if the underlying causal mechanism (the model parameter) is identical.
Scientific Practice: The authors argue that if treatment effects vary systematically across units, this variation should be explained via interaction terms within a stable model (as in McCullagh's formalism), rather than absorbed into an unstable, sample-dependent average.
Rebuttal to Common Defenses: The paper addresses three common arguments for model-free definitions:
- Heterogeneity: The model-based framework can accommodate individual-specific effects through interaction components while maintaining parameter stability.
- Model Misspecification: While model-free approaches avoid misspecification, they sacrifice stability and interpretability. If a model is misspecified, it too loses stability, underscoring the need for model adequacy checks.
- Double Robustness: The property of double robustness is only relevant if the target parameter $\tau_n$ is accepted as a valid scientific quantity; the authors challenge this acceptance.

Conclusion
The paper concludes that the model-free definition of treatment effect is not a universally appropriate target for inference. In many standard scientific settings (such as logistic or additive exponential models), the model-free average is an unstable quantity that changes with the sample composition, whereas the model-based parameter remains a stable, scientifically meaningful quantity. The authors suggest that the proliferation of methods to handle conditional validity or sub-population averages in causal inference literature is a symptom of this underlying instability in the model-free definition.

The Two Competing Recipes

The Big Problem: The "Chameleon" Effect

Why This Matters

Addressing the Counter-Arguments

The Bottom Line

More like this