A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling

1. Problem Statement

In large-scale digital marketing, standard A/B testing provides only Average Treatment Effects (ATE), which fail to distinguish between individual customer responses (positive, negative, or null). This limitation leads to three costly targeting errors:

• Wasted Spend: Targeting "always-converters" who would have purchased regardless of the intervention.
• Negative Returns: Targeting "sleeping dogs" who react adversely to the treatment.
• Missed Revenue: Failing to prioritize "high-responders" who are most likely to convert due to the intervention.

The paper addresses the need for Precision Marketing by estimating the Conditional Average Treatment Effect (CATE), denoted as $\tau(x) = E[Y(1) - Y(0) | X = x]$. While various CATE estimation methods exist, there is a lack of systematic, industrial-scale benchmarking to guide practitioners on method selection, particularly regarding the trade-offs between meta-learners and Causal Forests on massive datasets.

2. Methodology

Dataset

The study utilizes the Criteo Uplift v2.1 dataset, a public benchmark containing 13.98 million customer records.

• Features: 12 anonymized covariates ($f_0$–$f_{11}$).
• Treatment Assignment: Near-random (Propensity AUC = 0.509), with an 85% treated / 15% control split.
• Outcome: The primary outcome modeled is "visit" (4.7% positive rate); "conversion" (0.29% rate) is noted but left for future work due to sparsity.

Pipeline Architecture

The authors developed UpliftBench, a reproducible pipeline involving:

1. Preprocessing: 80/20 stratified train-test split; StandardScaler normalization.
2. Propensity Estimation: Logistic Regression to estimate propensity scores, appended as a feature for meta-learners.
3. CATE Estimation: Training four distinct estimators.
4. Evaluation: Qini coefficient, cumulative gain curves, policy simulation, and uncertainty quantification.
5. Interpretability: SHAP value analysis for feature attribution.

Estimators Compared

All meta-learners utilize LightGBM as the base learner for speed and performance on tabular data:

1. S-Learner: Fits a single outcome model $\mu(x, t)$ including the treatment indicator $t$ as a feature. CATE is estimated as $\hat{\tau}(x) = \mu(x, 1) - \mu(x, 0)$.
2. T-Learner: Fits separate models $\mu_1(x)$ and $\mu_0(x)$ for treated and control groups, respectively. CATE is the difference between predictions.
3. X-Learner: A two-stage cross-fitting approach designed to handle treatment imbalance. It imputes individual treatment effects by applying group models to the opposite group, then re-fits on these imputed effects weighted by propensity scores.
4. Causal Forest (EconML): A non-parametric approach using honest splitting to provide point estimates and 95% bootstrap confidence intervals. Note: Due to computational constraints, this was evaluated on a 10% stratified subsample (~280k records).

Evaluation Metrics

• Qini Coefficient: The primary metric, measuring the area between the model's cumulative gain curve and the random targeting baseline.
• Policy Simulation: Translates model performance into business value (incremental conversions vs. contact cost).
• Uncertainty Quantification: Uses Causal Forest confidence intervals to segment customers into "confident persuadables" (Lower CI > 0) and "confident sleeping dogs" (Upper CI < 0).

3. Key Results

Model Performance

• S-Learner Dominance: The S-Learner achieved the highest Qini score (0.3759), outperforming the X-Learner (0.3615), T-Learner (0.3503), and Causal Forest (0.2524).
• Targeting Efficiency: Targeting the top 20% of customers based on S-Learner predictions captured 77.7% of all incremental conversions. This represents a 3.9× improvement over random targeting.
• Variance Stability: The S-Learner exhibited the lowest CATE variance (Std = 0.0223) compared to T-Learner (0.0267) and Causal Forest (0.0377), indicating more stable ranking behavior.
• X-Learner Surprise: Contrary to theoretical expectations that X-Learner excels under treatment imbalance (85/15 split), it did not outperform the S-Learner. The authors attribute this to the S-Learner's implicit regularization reducing overfitting in large-sample, low-base-rate scenarios.
• Causal Forest Limitations: The lower Qini score for Causal Forest is likely due to the necessary subsampling (10% of data) rather than a fundamental algorithmic deficit.

Business Impact (Policy Simulation)

In a hypothetical campaign of 1,000,000 customers at $1 per contact:

• Targeting the top 20% via S-Learner achieves the same conversion volume as targeting 100% of the population randomly, but at 80% lower contact costs.

Uncertainty & Attribution

• Customer Segmentation: Using Causal Forest confidence intervals on the subsample:
  • Confident Persuadables: 1.9% (Lower CI > 0).
  • Confident Sleeping Dogs: 0.1% (Upper CI < 0).
  • Uncertain: 98.0% (CI overlaps 0).
  • Insight: The high uncertainty suggests that probabilistic targeting (using the full distribution) is superior to binary rule-based approaches.
• Feature Attribution (SHAP): Analysis identified feature $f_8$ as the dominant driver of Heterogeneous Treatment Effects (HTE), followed by $f_6$ and $f_2$.

4. Key Contributions

1. UpliftBench: A reproducible, large-scale benchmark (13.98M records) comparing meta-learners and Causal Forests, filling a gap in industrial-scale literature.
2. Empirical Evidence on Regularization: Demonstrates that at massive scales, the S-Learner's implicit regularization can outperform the theoretically superior X-Learner, challenging the default assumption that X-Learner is always best for imbalanced data.
3. Uncertainty Quantification at Scale: Provides a practical framework for segmenting customers based on confidence intervals, revealing that the vast majority of users have uncertain treatment effects.
4. Actionable Guidance: Offers evidence-based recommendations for practitioners, specifically validating the use of LightGBM-based S-Learners for high-volume uplift modeling.

5. Significance and Limitations

Significance:
The paper provides critical guidance for marketing data scientists, suggesting that for large-scale datasets with low base rates, simpler models (S-Learner) with strong base learners (LightGBM) may outperform complex meta-learners due to better variance control. It also highlights the computational trade-offs of Causal Forests in production environments.

Limitations:

• Subsampling: Causal Forest results are based on a 10% subsample; full-population performance might differ.
• Outcome Scope: Only "visit" was modeled; conversion (revenue) modeling was deferred.
• Feature Anonymization: The use of anonymized features ($f_0$–$f_{11}$) limits direct business interpretation of why certain features drive uplift, though SHAP analysis still identified dominant drivers.
• External Validity: Findings are specific to the Criteo dataset and may not generalize to all domains or data distributions.

The full pipeline and code are available at the provided GitHub repository, enabling further replication and extension by the research community.