📈 economics

Reevaluating Causal Estimation Methods with Data from a Product Release

This paper evaluates modern causal machine learning methods using a unique dataset from a product release at a large technology company, finding that while recovering ground truth causal effects is feasible, it requires careful modeling choices to ensure credible treatment effect estimation in high-dimensional settings.

Original authors: Justin Young, Eleanor Wiske Dillon

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Justin Young, Eleanor Wiske Dillon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to figure out if a new secret ingredient makes your soup taste better.

The Perfect Test (The Experiment):
In a perfect world, you would cook two identical batches of soup. In one batch, you add the secret ingredient. In the other, you don't. You taste both, and the difference tells you exactly what the ingredient does. This is what scientists call a randomized experiment. It's the "gold standard" because you know the only thing that changed was the ingredient.

The Real-World Mess (The Observation):
But in the real world, you can't always run a perfect experiment. Maybe you can't force people to eat the soup. Instead, you have to look at people who chose to add the ingredient themselves.
Here's the problem: The people who chose to add the secret ingredient might be different from those who didn't. Maybe they are more adventurous cooks, or they have better kitchens. If their soup tastes better, is it because of the ingredient, or because they are just better cooks? This is called selection bias.

The Paper's Mission:
This paper is like a cooking competition where the judges have a secret "perfect test" result (the gold standard) and they want to see if the chefs can guess that result just by looking at the messy "real-world" data of people who chose their own ingredients.

The authors, working with Microsoft, created a special dataset. They had:

The Experiment: A group of computers where Microsoft randomly turned a new feature on or off. They knew the true effect of this feature.
The Observation: A group of computers where users chose to turn the feature on.

They asked: Can we use fancy math and computer learning (Machine Learning) to look at the "choosers" and accurately guess what the "random" group found?

The Big Findings

1. It's Possible, But You Need the Right Tools
The paper found that yes, you can recover the true answer from the messy data, but only if you are extremely careful.

The "Naive" Approach: If you just compare the average performance of users who chose the feature vs. those who didn't, you get it wrong. It's like assuming the adventurous cooks made better soup just because of the ingredient, ignoring that they were better cooks to begin with.
The "Best Practice" Approach: When the authors used advanced methods (like Doubly Robust Estimators) and followed strict rules, they could successfully "undo" the bias and find the true answer for one of their tests.

2. The "Recipe" Matters More Than the Ingredients
The authors discovered that how you build your model is just as important as what data you feed it.

Analogy: Imagine trying to predict the weather. You have a fancy supercomputer (Machine Learning), but if you don't tune it correctly (setting the right "hyperparameters"), it might just guess "sunny" every day because that's what happened most often in its training data.
The Lesson: The authors found that if you don't carefully tune these computer models and check them against new data, they can be just as wrong as a simple guess. But if you tune them right, they can see complex patterns that simple math misses.

3. The "Trimming" Rule
Sometimes, the people who chose the feature are so different from those who didn't that you can't compare them fairly.

Analogy: Imagine trying to compare the speed of a Ferrari to a bicycle. It's a bad comparison. The authors found they had to "trim" the data—throwing out the extreme cases (the Ferraris and the bicycles) and only comparing the sports cars to the fast motorcycles. This made the comparison fair and the results accurate.

4. The "Binary" Problem (The Hard Case)
The paper tested two things:

Outcome A (Continuous): A smooth, sliding scale (like a speedometer). Here, the fancy math worked perfectly. They found the true answer.
Outcome B (Binary): A simple Yes/No question (like "Did the computer crash?"). Here, even the best math failed to find the true answer.
Why? The authors suspect there was a "hidden ingredient" (an unobserved factor) that influenced the Yes/No outcome but wasn't measured in their data. No amount of math can fix a missing piece of the puzzle. This highlights a hard limit: if you don't measure the right things, you can't find the truth, no matter how smart your computer is.

5. Checking Your Work (Sensitivity Analysis)
The authors also tested how to know if you are missing a hidden ingredient.

The Test: They asked, "How strong would a hidden factor have to be to change our conclusion?"
The Surprise: For the smooth outcome (where they got the right answer), the test said, "Hey, a tiny hidden factor could ruin this!" (Because the effect was small).
For the Yes/No outcome (where they got it wrong), the test said, "You'd need a massive hidden factor to change this!" (Because the effect was huge).
The Takeaway: These tests are useful, but they can be tricky. Just because a test says your result is "robust" doesn't mean it's correct; it might just mean the effect is so big that only a giant error could hide it.

The Bottom Line

This paper is a reality check for anyone trying to find cause-and-effect in messy, real-world data.

Good News: If you use modern, flexible computer tools and follow strict "best practices" (like tuning your models and trimming your data), you can often get results that match a perfect experiment.
Bad News: If you skip the careful steps, or if you are missing key data points, even the smartest computer cannot find the truth.
Final Thought: Statistics and computer science are powerful tools, but they are not magic wands. They can't fix bad data or missing information. To get the right answer, you need both the right math and a deep understanding of the real world you are studying.

Technical Summary: Reevaluating Causal Estimation Methods with Data from a Product Release

Problem Statement

The reliability of observational causal inference methods remains a critical open question in econometrics and data science. Unlike prediction tasks, where models can be validated against realized outcomes, causal effect estimates typically lack a "ground truth" for comparison. While recent advancements in causal machine learning (e.g., doubly robust estimators, meta-learners) offer flexible ways to model relationships between confounders, treatments, and outcomes, it is unclear how successfully these methods recover true causal effects in real-world, high-dimensional settings. Previous validation efforts, notably starting with LaLonde (1986), have yielded mixed results, often hindered by limited covariate overlap between observational and experimental samples or insufficient similarity between the two populations.

This paper addresses the gap by evaluating observational causal inference methods against a new, high-quality benchmark dataset derived from a product release at a large technology company. The study aims to determine under what conditions observational methods can recover experimental ground truth, identify best practices for estimation design, and assess the limits of these methods when unobserved confounding is present.

Methodology and Data

The authors utilize a unique paired dataset involving a new product feature rollout on Windows devices:

Experimental Sample: A randomized controlled trial (RCT) where a subset of devices was exogenously assigned the new feature.
Observational Sample: A simultaneous sample of devices where users endogenously opted into the feature.
Outcomes: Two device performance metrics are analyzed: a continuous outcome and a binary outcome.
Covariates: A rich set of user and device characteristics (e.g., technical specifications, usage patterns) is available, though some features are masked for public release.

The study employs a comprehensive suite of estimators, including:

Outcome Modeling: Linear regression and Oaxaca-Blinder estimators.
Propensity Score Methods: Propensity Score Matching (PSM) and Inverse Propensity Weighting (IPW).
Doubly Robust (DR) Methods: Augmented Inverse Probability Weighting (AIPW).
Conditional Average Treatment Effect (CATE) Estimators: Meta-learners (S-, T-, X-, R-, and DR-learners) and Q-aggregation ensembles.

Key Implementation Details:

Trimming: The authors apply the optimal trimming rule (Crump et al., 2009) to ensure common support, removing units with extreme propensity scores to mitigate extrapolation bias.
Nuisance Modeling: They utilize AutoML (specifically FLAML and Optuna) to tune hyperparameters for flexible machine learning models (e.g., Light Gradient Boosted Machines, Random Forests) used to estimate nuisance functions (propensity scores and outcome regressions).
Ensembling: To reduce model uncertainty, the authors ensemble 50 tuned models for each nuisance function rather than relying on single runs.
Cross-Fitting: Sample splitting is used to construct valid confidence intervals and prevent overfitting in the nuisance estimation stages.

Key Results

1. Recovery of Ground Truth (Continuous Outcome)

For the continuous performance outcome, the study demonstrates that observational methods can successfully recover the experimental ground truth, but only when strict best practices are followed.

Success Conditions: When using sample trimming, AutoML-tuned flexible machine learning models for nuisance functions, and model ensembling, doubly robust estimators (AIPW) produce estimates indistinguishable from the experimental benchmark.
Failure Modes: Deviating from these practices leads to significant bias.
- No Trimming: Estimates remain biased upward.
- Poor Nuisance Tuning: Using untuned or poorly performing models (e.g., default hyperparameters) results in estimates as biased as naive difference-in-means. The study finds a near-monotonic relationship between the predictive quality ( $R^2$ , AUC) of the nuisance models and the accuracy of the causal estimate.
- Model Instability: Single runs of cross-fitting introduce variation; ensembling across multiple runs is necessary to stabilize estimates, though this variation is smaller than standard sampling error.

2. Failure to Recover Ground Truth (Binary Outcome)

For the binary performance outcome, no observational method successfully recovers the experimental benchmark, even with the best practices applied.

Bias: All methods overstate the treatment effect (estimating a 54% decline vs. the true 43% decline).
Cause: The available observed confounders fail to predict the binary outcome (AUC $\approx$ 0.54), suggesting the presence of unobserved confounders specific to this outcome.
Sensitivity Analysis Paradox: Sensitivity analysis (Chernozhukov et al., 2022) suggests the binary outcome estimate is "robust" to unobserved confounding (requiring a confounder to explain ~75% of residual variation to nullify the effect). However, this is an artifact of the large magnitude of the true treatment effect. Conversely, the continuous outcome, which does match the ground truth, appears highly sensitive to unobserved confounding in sensitivity tests because the true effect size is small. This highlights that sensitivity analysis alone cannot validate the unconfoundedness assumption or guarantee accuracy.

3. Heterogeneous Treatment Effects (CATE)

The study evaluates the ability of CATE estimators to detect and rank treatment effect heterogeneity.

Detection: The DR-score test correctly identifies heterogeneity for the binary outcome but not for the continuous outcome in both experimental and observational samples.
Ranking vs. Magnitude: While the observational sample fails to recover the magnitude of the CATE (due to the systemic bias noted above), it successfully recovers the ordering of treatment effects across subgroups. Q-aggregation ensembles in the observational sample correctly identify the same user/device features driving heterogeneity (e.g., high usage frequency correlates with larger declines in the binary outcome) as the experimental benchmark.

4. Re-analysis of LaLonde Data

Applying the proposed workflow (trimming, AutoML tuning, ensembling) to the classic LaLonde (1986) datasets (LDW-CPS and LDW-PSID) yields results consistent with recent re-analyses (Imbens and Xu, 2025). The modern best practices improve the alignment of observational estimates with experimental benchmarks, particularly in the LDW-CPS sample, reinforcing the generalizability of the findings.

Significance and Claims

The paper claims to be the first real-world application quantifying the critical importance of precise estimation of nuisance functions (the effects of confounders on treatment and outcome) in observational studies, supplementing previous simulated illustrations.

Core Contributions:

Best Practices Validation: The study provides empirical evidence that observational causal inference can be credible, but it is highly sensitive to design choices. Specifically, the choice of functional form (via flexible, tuned ML) is as quantitatively important as the selection of confounders.
Practical Guidance: It offers concrete, verifiable steps for practitioners:
- Trim samples to ensure overlap.
- Use sample splitting and cross-fitting.
- Tune hyperparameters using AutoML to maximize nuisance model predictive performance.
- Ensemble multiple models to reduce variance.
Limits of Sensitivity Analysis: The paper cautions that sensitivity analysis metrics (like robustness values) can be misleading. A "robust" estimate may still be biased if the true effect is large, while a "sensitive" estimate may be accurate if the true effect is small and aligns with an experimental benchmark.
CATE Utility: While CATE estimators may fail to recover absolute effect magnitudes in the presence of unobserved confounding, they can still reliably identify the ranking of subgroups, which is valuable for targeted policy interventions.

Ultimately, the paper concludes that while careful design and modern statistical techniques can greatly enhance the credibility of observational causal inference, they do not eliminate the fundamental reliance on substantive assumptions and the untestable unconfoundedness assumption.