📊 statistics

Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation

This paper reveals a critical disconnect between methodological research and practical application in treatment effect estimation, demonstrating that counterfactual metrics and semi-simulated benchmarks fail to reliably predict model performance on real-world data, thereby advocating for a shift toward observable metrics and real-data validation.

Original authors: George Panagopoulos

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: George Panagopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which of two different fertilizers makes plants grow the biggest. You have a team of scientists (the researchers) and a team of farmers (the real-world practitioners).

This paper argues that the scientists and the farmers are playing two completely different games, using different rulebooks, and they don't realize they are talking about different things.

Here is the breakdown of the paper's findings using simple analogies:

1. The Two Different Rulebooks

In the world of "Causal Machine Learning" (using AI to figure out the effect of a treatment, like a drug or a marketing email), there are two ways to judge if a model is good:

The Scientist's Rulebook (Semi-Simulated Benchmarks):
Imagine the scientists are in a lab. They grow plants in a controlled environment where they know exactly what would have happened if they used Fertilizer A and what would have happened if they used Fertilizer B. Because they know both outcomes, they can calculate the "perfect" difference. They use a metric called PEHE (Precision in Estimation of Heterogeneous Effects).
- The Analogy: It's like a video game where the developers know the "cheat code" for the perfect score. They judge the AI based on how close it gets to that perfect, known score.
The Farmer's Rulebook (Real-World Data):
The farmers are out in the actual field. They can only see what happened to the plants they actually treated. They never see what would have happened if they had chosen the other fertilizer. They can't measure the "perfect difference." Instead, they judge the AI based on observable results: "Did the AI help us pick the best 20% of plants to treat?" or "Did the total harvest increase?"
- The Analogy: It's like a real-life sports match. You don't know what would have happened if the player had taken a different shot; you only know if the goal was scored. You judge the player by the win/loss record, not by a theoretical "perfect play."

2. The Big Disconnect (The "Metric Mismatch")

The paper ran a massive experiment (the largest of its kind) to see if the "Lab Winners" were also the "Field Winners."

The Shocking Finding:
The models that won the "Lab Game" (using the perfect cheat codes) were often not the models that won the "Field Game."

The Analogy: Imagine a chess computer that is the undisputed world champion in a simulation where it knows the opponent's next 10 moves. But when you put that same computer in a real tournament against a human, it loses badly.
The Paper's Proof: When the researchers looked at the data, they found that the "best" model according to the Lab's perfect score (PEHE) was often ranked very poorly by the Farmer's real-world metrics (like "Up@20" or "Qini"). In fact, picking the Lab Winner often led to a "regret" (a loss in performance) when applied to real data.

3. The "Fake Field" Problem

The scientists often use "Semi-Simulated" data. This is real data where they pretend to know the counterfactuals (the "what ifs") by making up the missing numbers based on mathematical assumptions.

The Finding:
Even when the scientists used the same real-world metrics (like ranking) on these "Fake Fields," the results didn't match the results from the actual real fields.

The Analogy: It's like testing a car on a perfect, smooth, computer-generated track. The car looks amazing. But when you drive that same car on a real road with potholes and rain, it handles completely differently. The "Fake Field" creates a ranking of cars that doesn't translate to the "Real Road."

4. The Surprise Winner: The "Simple" Models

The paper looked at many complex, fancy AI models (like specialized neural networks designed just for this).

The Finding:
The fancy, specialized models often lost. The winners were usually simple, robust models (like standard "Meta-Learners" paired with strong, basic tools like XGBoost).

The Analogy: In the race between a high-tech, aerodynamic Formula 1 car (the specialized causal models) and a sturdy, reliable pickup truck (the simple meta-learners), the pickup truck kept winning in the real world. The F1 car was great on the test track, but the pickup truck handled the real terrain better.

5. The Conclusion: Stop Cheating, Start Testing

The authors conclude that the field of research is currently "broken" because it rewards models that are good at solving a theoretical puzzle (the Lab Game) rather than models that actually work in the real world (the Field Game).

The Takeaway: We cannot just look at the "Perfect Score" (PEHE) on a simulated dataset and say, "This is the best model." We must also test these models on real data using real-world goals (like ranking or total profit).
The Call to Action: Researchers need to stop treating the "Lab Game" as the only truth. They need to include "Field Tests" (real data and observable metrics) to ensure their models actually work when deployed.

In short: Just because a model looks perfect in a simulation where we know the answer, doesn't mean it will be the best choice when we don't know the answer in the real world. The paper urges us to stop relying solely on the simulation and start testing in the real world.

Technical Summary: Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation

Problem Statement

The field of Causal Machine Learning (Causal ML) faces a critical misalignment between research evaluation practices and real-world deployment requirements. While academic research predominantly relies on semi-simulated benchmarks and counterfactual metrics (e.g., $\sqrt{PEHE}$ ) that require ground-truth potential outcomes, industrial applications operate under conditions where only observable outcomes are available. In real-world settings, the counterfactual outcome (what would have happened under a different treatment) is unobserved by definition. Consequently, practitioners must rely on observable metrics such as ranking performance (e.g., uplift curves) or factual test outcomes.

The core problem addressed is whether the evaluation regimes used in methodological papers reliably predict model performance in deployment scenarios. Specifically, the authors investigate if models selected as "best" by counterfactual metrics on semi-simulated data transfer effectively to real datasets when evaluated using observable metrics.

Methodology

The authors conduct a large-scale empirical study involving 117 benchmark instances, comprising 112 semi-simulated datasets and 5 real-world datasets. The sample sizes range from hundreds to nearly 14 million (Criteo-ITE).

Estimators Evaluated

The study evaluates a comprehensive pool of estimators, including:

Meta-learners: S-, T-, X-, R-, and Doubly Robust (DR) learners [Künzel et al., 2019; Nie and Wager, 2021].
Base Learners: Linear/Logistic Regression (LR), XGBoost (XGB), and two-layer Multi-Layer Perceptrons (NN).
Specialized Architectures: TARNet, CFR, DragonNet, and Causal Forests.

Evaluation Metrics

The authors compare three families of metrics:

Counterfactual-based (Oracle): $\sqrt{PEHE}$ , Policy Value, and $\Delta$ ATE(Sim). These require access to both potential outcomes and are restricted to semi-simulated settings.
Observable Outcome-based: Factual loss and $\Delta$ ATE (measuring discrepancy between predicted and empirical average treatment effects).
Observable Ranking-based: Up@20, Qini score, and RATE AUTOC. These assess the model's ability to prioritize units for treatment based on observed data.

Experimental Design

The study addresses two primary questions:

Q1 (Metric Transfer): Do counterfactual metrics on semi-simulated data induce the same model rankings as observable metrics?
Q2 (Dataset Transfer): Do rankings obtained on semi-simulated benchmarks transfer to real datasets?

The authors utilize Kendall's $\tau$ and Wilcoxon signed-rank tests to quantify the agreement between metric-induced rankings. They also perform a matched case study using the Criteo dataset (real) and Criteo-ITE (semi-simulated variant derived from the same source) to isolate the effect of the evaluation regime from covariate differences.

Key Results

1. Metric Disconnect (Q1)

The study finds that counterfactual metrics do not reliably recover the estimators preferred by observable metrics, even on the same semi-simulated benchmarks.

Ranking Divergence: Models selected as "best" by $\sqrt{PEHE}$ often incur substantial regret when evaluated on ranking metrics like Up@20 or Qini. For instance, the estimator selected by $\sqrt{PEHE}$ ranked 13th on average under Up@20.
Statistical Significance: The agreement between ranking metrics is significantly higher (mean $\tau = 0.355$ ) than the agreement between ranking and counterfactual metrics (mean $\tau = 0.256$ ).
Implication: Treating counterfactual metrics as the gold standard for model selection may lead to selecting estimators that are suboptimal for deployment-oriented goals.

2. Dataset Non-Transferability (Q2)

Rankings obtained on semi-simulated benchmarks do not transfer to real datasets, even when using identical observable metrics.

Inconsistent Winners: Methods that dominate semi-simulated benchmarks (e.g., DragonNet, DR-LR on IHDP) often fail to appear in the top ranks of real-world datasets.
Rank Shifts: There is significant dispersion in rank shifts ( $\Delta$ Rank) between semi-simulated and real data. For example, R- and DR-based estimators often improve in real-data rankings, while T- and X-learners may deteriorate.
Matched Case Study: Even in the Criteo matched study (same covariates and treatment assignments, different outcome generation), the overall ordering of estimators changed between the counterfactual (Criteo-ITE) and observable (Criteo) settings.

3. Practical Observations

Base Learner Dominance: Simple meta-learners paired with strong predictive backbones (specifically XGBoost) are consistently competitive across regimes. T(XGB), X(XGB), and S(XGB) dominate the overall rankings.
Specialized Models: Specialized neural architectures (e.g., DragonNet, CFR) generally underperform compared to simple meta-learners with tabular backbones, except when evaluated specifically under counterfactual semi-simulated metrics.

Significance and Claims

The paper argues that the field of treatment effect estimation faces both a metric problem and a dataset problem.

Re-evaluating Evaluation Protocols: The authors claim that progress in Causal ML should not be assessed solely through oracle-style counterfactual metrics and semi-simulated benchmarks. These regimes can reward methods with inductive biases that align with simulation mechanisms rather than real-world treatment assignment or outcome noise.
Need for Real-Data Validation: The study advocates for incorporating observable metrics and real-data validation to close the gap between research and deployment.
Benchmarking Standards: The authors suggest that methodological papers should include simple meta-learners with strong predictive backbones (like XGBoost) as baselines, as specialized causal models often fail to outperform them in observable, deployment-oriented settings.

The paper concludes that while semi-simulated benchmarks are useful for stress-testing and controlled analysis, they are insufficient as standalone guides for practical applications. Future work should focus on developing evaluation protocols that better connect oracle-style counterfactual evaluation with observable, deployment-oriented criteria.