Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation

Technical Summary: Can AI Agents Simulate A/B Test Outcomes?

Problem Statement
Online controlled experimentation (A/B testing) is the standard for product iteration in the technology industry, yet it incurs substantial costs in engineering effort, wall-clock time, and the exposure of real customer traffic to potentially underperforming treatments. The paper posits that a sufficiently accurate behavioral simulator could allow teams to vet dozens of candidate treatments cheaply before committing live traffic. While recent work has explored using AI agents to simulate user behavior, existing systems lack an explicit decomposition of estimation error, making systematic diagnosis and reduction difficult. Furthermore, current approaches often fail to distinguish between errors caused by the behavioral model's approximation capabilities and those caused by finite-sample subsampling.

Methodology: The Simulated Randomized Controlled Trial (S-RCT)
The authors formalize agentic A/B prediction as a Simulated Randomized Controlled Trial (S-RCT). In this framework, live traffic is replaced by computational surrogates (agents) that map a user's persona ($\Pi_j$), context, and task to a simulated outcome ($\tilde{Y}_j$).

• Error Decomposition: A core contribution is the derivation of a two-layer error decomposition for the total estimation error:
  $$\widehat{\text{ATE}} - \text{ATE} = (\widehat{\text{ATE}}_\theta - \text{ATE})_{\text{approximation error}} + (\widehat{\text{ATE}} - \widehat{\text{ATE}}_\theta)_{\text{subsampling error}}$$

  • Approximation Error: The gap between the simulator's population-level prediction and the true Average Treatment Effect (ATE), driven by the behavioral model and persona completeness.
  • Subsampling Error: The finite-sample gap between the realized agent estimate and the simulator-population ATE.

• Instantiation: The framework is engine-agnostic but instantiated here using a general-purpose foundation model. Agents are constructed via "agentic-twin pairing," where each agent corresponds to a specific real customer from historical data. The design allows for within-subject estimation, where each agent is exposed to both treatment and control arms, enabling the calculation of paired individual treatment effects ($\tilde{\Delta}_j$).

Key Contributions

1. Formalization: The paper formalizes agentic A/B prediction as an S-RCT and derives the two-layer error decomposition separating approximation error from subsampling error.
2. Validation: The framework is validated against a benchmark of 67 historical marketing A/B tests (measuring click-through rates on product pages).
3. Improvement Protocols: The authors propose specific techniques to address each error layer independently:
   • Principled Subsampling: Using Neyman allocation to optimize agent budget distribution across strata based on historical variance heterogeneity.
   • Two-Phase Calibration: A lightweight protocol using pre-period user data to fit a calibration function (e.g., Platt scaling) that corrects systematic biases before the main prediction phase.
   • Within-Subject Estimation: A design where agents serve as their own controls to eliminate between-agent composition variance.

Results

• Baseline Performance: Using an off-the-shelf foundation model without calibration, the S-RCT captures directional signal with a sign overlap of 0.70. However, it systematically overshoots effect magnitudes (Mean Absolute Error = 0.0893), resulting in poor launch alignment (0.41) near the random floor.
• Impact of Calibration: Applying the two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible noise) by approximately 77×, bringing estimates close to the historical noise floor.
• Impact of Within-Subject Design: Switching from a between-subject to a within-subject design reduces standard errors by approximately 2.4× and improves sign accuracy from 65% to 70%.
• Subsampling Efficiency: The paper demonstrates that smart subsampling (Neyman allocation) yields significant efficiency gains for continuous metrics like revenue (2–5× improvement) but offers limited gains for binary metrics with low base rates (a few percent).

Significance and Limitations
The paper concludes that while current general-purpose foundation models exhibit a "behavioral amplification" (predicting larger effects than observed in reality), the S-RCT framework offers a viable path for pre-screening experiments. The authors emphasize that the immediate practical value lies not in replacing live experiments, but in:

• Directional Screening: Flagging likely "losers" before live deployment (relying on sign accuracy).
• Interpretability: Providing reasoning traces and "flipper analysis" (identifying personas that change decisions) to guide segment-level analysis.
• Bayesian Priors: Using noisy directional signals to accelerate go/no-go decisions when combined with early real-experiment data.

The authors acknowledge limitations, including the systematic magnitude overshoot, the reliance on a single-domain benchmark (marketing CTR), and the retrospective nature of the evaluation. They note that predicting the triggering population (who enters the experiment) remains a critical upstream challenge for prospective deployment. The ultimate goal is to make the experimentation pipeline faster and more informed, allowing customers to benefit from improvements sooner, rather than to replace real-world validation.