Multi-Experiment Analysis

1. Problem Statement

Online controlled experiments (A/B tests) are the gold standard for data-driven decision-making in technology. However, as organizations increase experimentation velocity, they face a critical bottleneck: concurrent overlapping experiments on shared user traffic.

• The Conflict: Traditional approaches to handle overlaps include traffic splitting (layering), sequential execution, or factorial designs.
  • Traffic splitting/layering requires complex coordination and reduces the available traffic for each test, slowing down velocity.
  • Sequential execution delays feature launches.
  • Factorial designs require pre-planned coordination and balanced allocation, which is impractical when different teams launch independent experiments with different triggering conditions and timelines.
• The Consequence: When experiments run concurrently without restrictions, two issues arise:
  1. Collisions: Simultaneous changes break the user experience (often prevented by blocking).
  2. Interactions: Experiments run together without breaking UX, but their effects interact. Analyzing them in isolation (univariate analysis) produces biased estimates that do not correspond to any actionable launch scenario (e.g., launching Feature A alone vs. Feature A + Feature B). Naive analysis fails to identify optimal feature combinations or detect negative synergies.

2. Methodology: Multi-Experiment Analysis (MEA)

The paper proposes MEA, a post-hoc framework for consistent joint estimation of overlapping experiments without requiring traffic restrictions or pre-planned factorial designs.

Core Concept: Triggering-Based Partitioning

Instead of treating the population as a single block, MEA partitions users based on joint triggering logic.

• Let $S_i$ be the indicator that a user triggers Experiment $i$.
• Users are partitioned into mutually exclusive regions (strata) based on the vector $S = (S_1, ..., S_k)$.
  • Example: For two experiments, regions include: Only $E_1$ triggered ($R_{10}$), Only $E_2$ triggered ($R_{01}$), Both triggered ($R_{11}$), and Neither ($R_{00}$).
• Key Insight: The impact of a launch scenario is a weighted average across all triggered regions, not just the intersection ($R_{11}$). For instance, the effect of launching $(t_1, t_2)$ includes users who only saw $t_1$ (in $R_{10}$) and those who only saw $t_2$ (in $R_{01}$).

Estimation Algorithm

1. Stratification: Group users by their trigger states ($R_{10}, R_{01}, R_{11}$).
2. Within-Stratum Comparison: Compare the outcome of the target variant combination against the baseline within each specific region.
   • In $R_{11}$: Compare $(t_1, t_2)$ vs. $(c_1, c_2)$.
   • In $R_{10}$: Compare $(t_1, \text{nan})$ vs. $(c_1, \text{nan})$.
3. Weighted Aggregation: Compute the final effect estimate as a weighted sum of region-specific differences:
   $$\hat{\tau} = \sum_{r} w_r (\bar{Y}_{t,r} - \bar{Y}_{c,r})$$
   Where weights $w_r$ are proportional to the size of the triggered population in region $r$ (excluding $R_{00}$).

Critical Assumptions

For MEA to yield consistent causal estimates, it relies on Arm-Trigger Invariance:

• Definition: The probability of a user triggering an experiment must not change based on the treatment assignment of other experiments.
• Mathematical Form: $P(S=s | do(A=a)) = P(S=s)$.
• Violation: If Experiment 1's treatment changes user behavior such that they are more likely to trigger Experiment 2 (e.g., adding a link that leads to the trigger condition), the assumption is violated, and estimates become biased.
• Solution: The paper introduces automated diagnostics (Chi-squared tests and Cramér's $V$) to detect violations using only experiment data.

Extensions

• Ratio Metrics: MEA extends to ratios (e.g., CTR, Conversion Rate) by applying weighted estimation to numerators and denominators separately, then combining via the Delta method or bucketed jackknife.
• Scenario-Based Analysis: Allows estimating the conditional effect of one experiment given a specific launch decision for another (e.g., "What is the effect of $E_2$ if $E_1$ is already live?").

3. Key Contributions

1. Methodological Innovation: First systematic application of post-stratification and weighted averaging to causal estimation across overlapping online experiments with complex, uncoordinated triggering patterns.
2. Three Types of Estimates:
   • Corrected Individual Effects: Estimates of a single experiment's effect accounting for the concurrent state of others.
   • Combined Effects: Estimates of launching any desired combination of variants.
   • Conditional Effects: Estimates of an experiment's impact given specific launch decisions for other experiments.
3. Statistical Guarantees:
   • Proves consistency under Arm-Trigger Invariance.
   • Demonstrates that MEA's variance is at most that of a coordinated factorial design, often offering exponential efficiency gains as the number of experiments ($k$) increases (due to fewer variant cells in partial overlaps).
4. Production System: A scalable, self-serve system implemented at LinkedIn supporting hundreds of millions of units, utilizing distributed computing (Trino/Spark) and bucketed jackknife for variance estimation.

4. Results and Validation

• Simulations: Comprehensive simulations confirmed that MEA estimates converge to the true ground-truth delta as sample size increases. Coverage of 95% confidence intervals was correct (~95.2%).
• Comparison with Alternatives:
  • Univariate Analysis: Often leads to incorrect decisions. In a simulation with strong negative interaction, univariate analysis recommended the worst possible combination (launching both), while MEA correctly identified the optimal single-feature launch.
  • Regression: Standard interaction regression on the intersection ($R_{11}$) ignores singly-triggered users, leading to massive overestimation of launch impact (e.g., estimating +11% when the true launch impact is +1.7%).
• Real-World Case Study (LinkedIn):
  • Scenario: A content test (nested) and a page redesign test (broad) overlapped asymmetrically.
  • Finding: Univariate analysis suggested a +26.2% lift for the redesign. MEA revealed this was an artifact of the concurrent content test. The scenario-specific effect ranged from +48.6% to other values depending on the content variant.
  • Outcome: MEA identified a specific combination $(v4, v2)$ with a +96% lift, which univariate analysis would have missed. The team launched this combination, reducing time-to-market.
• Adoption: Deployed across 8+ product verticals. 80–85% of audited analyses led to changed or informed decisions. 4% of analyses triggered independence flags, successfully catching design flaws (e.g., treatment-induced triggering).

5. Significance

• Velocity vs. Validity: MEA resolves the trade-off between experimentation velocity and statistical validity. It allows teams to run thousands of concurrent experiments without traffic fragmentation or sequential delays, while still recovering accurate causal insights.
• Actionable Insights: It moves beyond "did this work?" to "what is the best combination to launch?" and "what happens if we launch this alongside that?", directly supporting product strategy.
• Broad Applicability: While focused on online tech, the methodology applies to clinical trials, public health, and manufacturing where multiple interventions overlap on populations.
• Practical Tooling: The inclusion of automated diagnostics and a scalable system design makes the complex statistical theory accessible to data scientists and product managers in production environments.

In summary, Multi-Experiment Analysis provides a robust, non-parametric framework to untangle the effects of overlapping experiments, enabling organizations to maximize the value of their concurrent testing infrastructure without sacrificing statistical rigor.