Technical Summary: Synthesizing Evidence: Data-Pooling as a Tool for Treatment Selection in Online Experiments

1. Problem Statement

Randomized controlled trials (RCTs) are the gold standard for causal inference in business applications, particularly within online platforms. However, the increasing scale of experimentation—where companies like Microsoft, Google, and Booking run thousands of concurrent experiments—has created a "data scarcity" problem. While the total user base is large, the traffic allocated to any single experiment is often constrained by operational limits or risk management.

This scarcity leads to high variance in Average Treatment Effect (ATE) estimates, making it difficult to distinguish effective policies from ineffective ones. Traditional "Individual Treatment Roll-out" (ITR) methods, which analyze each experiment in isolation using difference-in-means or Ordinary Least Squares (OLS), suffer from low statistical power when sample sizes are small. This issue is exacerbated in targeted experimentation (subgroup analysis) and orthogonal designs where specific treatment combinations may be underrepresented. The central research question is: How can platforms improve treatment selection and policy roll-out decisions when facing a large number of experiments with limited data per experiment?

2. Methodology: The DPTR Framework

The authors propose the Data-Pooling Treatment Roll-out (DPTR) framework. Unlike ITR, which treats experiments independently, DPTR aggregates information across multiple experiments to improve decision-making.

Core Mechanism

DPTR employs a shrinkage estimator that combines an individual experiment's estimator ($\hat{\tau}_k$) with an "anchor" estimator ($\hat{\tau}_0$), defined as the average of all individual estimators. The new estimator $\bar{\tau}_k$ is a weighted average:
$$\bar{\tau}_k = \frac{N}{N + \beta} \hat{\tau}_k + \frac{\beta}{N + \beta} \hat{\tau}_0$$
where $N$ is the sample size and $\beta \geq 0$ is a data-driven scale parameter.

Decision-Aware Design

A critical distinction of DPTR is that it is decision-aware. Traditional shrinkage often aims to minimize Mean Squared Error (MSE). In contrast, DPTR optimizes the roll-out reward (the expected benefit of implementing policies). The scale parameter $\beta$ is explicitly derived to maximize the expected reward of the roll-out decision, incorporating the significance level ($\alpha$) and the threshold for policy implementation. This results in "shrunken decision bounds" rather than standard confidence intervals.

Implementation Scenarios

The framework is designed to be flexible across four distinct experimentation scenarios:

1. Non-overlapping experiments with linear specifications: Uses Difference-in-Means (DM) or OLS. The scale parameter $\hat{\beta}$ is derived based on the ratio of within-experiment variance to between-experiment heterogeneity.
2. Non-overlapping experiments with nonlinear specifications: Utilizes Double Machine Learning (DML) and cross-fitting to handle partial linear models.
3. Overlapping experiments with linear specifications: Handles cases where users are simultaneously exposed to multiple treatments using OLS on the joint treatment vector.
4. Overlapping experiments with nonlinear specifications: Extends the DML approach to overlapping traffic structures.

In all cases, the framework provides formulas for a data-driven scale parameter ($\hat{\beta}$) that can be either shared across experiments or personalized ($\hat{\beta}_k$) based on covariate information.

3. Key Contributions

The paper makes three primary contributions:

1. A General Data-Pooling Framework: The authors introduce a novel framework that integrates data across multiple experiments to enhance treatment selection. It is explicitly decision-aware, aligning estimation with downstream roll-out decisions by incorporating significance levels into the shrinkage procedure. The framework is applicable to both non-overlapping and overlapping traffic scenarios, as well as linear and nonlinear model specifications.
2. Theoretical Guarantees: Under non-overlapping experiments with linear specifications, the authors provide formal theoretical proofs showing that DPTR weakly improves upon the ITR method and strictly improves it in the "positive-anchor regime" (where the expected ATE across experiments is positive). They derive the optimal scale parameter $\beta^*$ that maximizes expected reward and demonstrate that a consistent data-driven estimator for $\beta$ exists.
3. Empirical Validation: The framework is validated through:
   • Synthetic Simulations: Demonstrating robustness across varying sample sizes, numbers of experiments, noise levels, and model misspecifications (linear vs. nonlinear).
   • Real-World Applications: Application to two datasets: (a) Criteo's ad-targeting data (non-overlapping) and (b) Ye et al. (2025) short-video platform data (overlapping).

4. Results

Synthetic Experiments

• Performance Metrics: DPTR consistently outperforms ITR and Bayesian benchmarks in terms of Optimality Ratio (OR) and Value of Data Pooling (VDP).
• Trade-offs: DPTR significantly improves Recall (identifying positive ATEs) compared to ITR, often at the cost of slightly lower Specificity and Precision. This trade-off is intentional, as the framework prioritizes maximizing total reward over minimizing classification error.
• Sensitivity: The benefits of DPTR are most pronounced when:
  • The number of experiments ($K$) is large.
  • The sample size per experiment ($N$) is small.
  • The expected ATE ($\tau_0$) is small.
• Robustness: DPTR remains effective even under model misspecification (e.g., using DM when the true model is nonlinear), whereas ITR performance degrades sharply in such scenarios.

Real-World Applications

• Criteo Dataset (Non-overlapping): DPTR significantly outperformed ITR and Bayesian methods in selecting personalized recommendations for user subgroups, particularly when sample sizes were small. The Bayesian method performed similarly to ITR in this context because individual group variances were small, leading to negligible shrinkage, whereas DPTR's use of aggregate variance enabled effective pooling.
• Short-Video Platform (Overlapping): In a setting with three overlapping experiments, DPTR consistently yielded higher OR values than ITR across various estimation methods (DM, OLS, DML) and sample sizes. Notably, DPTR enhanced the performance of DML, which typically suffers from high variance in small samples, by effectively reducing that variance through pooling.

5. Significance and Claims

The paper claims that DPTR provides a scalable, practical solution for online platforms to better leverage experimental data in complex business environments. Its significance lies in:

• Addressing Data Scarcity: It offers a principled way to improve decision-making when traffic allocation is limited, a common constraint in modern A/B testing.
• Decision-Centric Optimization: By shifting the objective from pure estimation accuracy (MSE) to decision quality (reward), DPTR aligns statistical inference with business goals.
• Robustness: The framework is shown to be robust against model misspecification and overlapping experimental designs, which are prevalent in real-world platforms but often difficult to handle with traditional methods.

The authors modestly note that the method is designed for settings with multiple parallel experiments without capacity constraints. They acknowledge that in environments with tight capacity constraints (limiting the number of roll-outs) or strong sign-reversing interactions between treatments, the unconstrained roll-out rule studied may not be optimal. However, for the vast majority of online experimentation scenarios characterized by data scarcity and high-dimensional treatment spaces, DPTR offers a theoretically grounded and empirically validated improvement over traditional isolated analysis.