Regression Adjustments for Double Randomization in Two-Sided Marketplaces

1. Problem Statement

In two-sided marketplaces (e.g., ride-sharing, e-commerce), standard A/B testing often fails due to interference (spillovers), where the treatment of one unit (e.g., a seller) affects the outcomes of others (e.g., buyers). To address this, Multiple Randomization Designs (MRDs) have been proposed, where both sides of the market (buyers and sellers) are randomized independently.

While MRDs allow for the estimation of direct, spillover, and total effects under a "local interference" assumption, a critical gap exists in the literature: how to improve the precision of these estimators using covariate adjustments.

• The Challenge: Standard regression adjustments (like ANCOVA or CUPED used in simple A/B tests) are not directly applicable or optimal in MRDs. In fact, naive regression adjustments can sometimes increase asymptotic variance compared to unadjusted estimators, particularly in imbalanced experimental designs.
• The Goal: Develop a regression adjustment methodology specifically for MRDs that minimizes asymptotic variance without relying on strong parametric assumptions about the outcome model.

2. Methodology

The authors propose a model-robust, finite-population framework where potential outcomes are fixed, and randomness arises solely from the double randomization of buyers ($W^B$) and sellers ($W^S$).

A. Estimators and Setup

• Design: Simple Multiple Randomization Designs (SMRDs) assign binary treatments to buyers and sellers. A pair $(i,j)$ receives treatment only if both are treated ($W^B_i=1, W^S_j=1$).
• Potential Outcomes: Under local interference, outcomes depend on the treatment status of the specific pair and the fraction of treated units on the opposite side. This creates four groups:
  • $tr$: Both treated.
  • $ib$: Buyer treated, Seller control.
  • $is$: Buyer control, Seller treated.
  • $cc$: Both control.
• Target Estimands: Linear combinations of group means ($\bar{y}_\gamma$), such as the Direct Effect ($\bar{y}_{tr} - \bar{y}_{ib} - \bar{y}_{is} + \bar{y}_{cc}$) and Total Effect.

B. Optimal Regression Adjustment Strategy

The core methodology involves non-interacted imputation estimators:
$$\hat{\tau}_c(\beta) = \sum_{\gamma \in \Gamma} c_\gamma \frac{1}{I_\gamma J_\gamma} \sum_{(i,j) \in \gamma} (y_{ij} - X_{ij}^\top \beta)$$
where $X_{ij}$ are covariates and $\beta$ is a coefficient vector.

1. Deriving the Optimal $\beta$:
   The authors derive the $\beta$ that minimizes the asymptotic variance of $\hat{\tau}_c(\beta)$. Unlike classical settings where the optimal $\beta$ might depend on unobservable potential outcomes, they prove that the optimal $\beta$ is estimable from observed data.

   • The optimal $\beta$ is the solution to a generalized least squares problem involving specific weights derived from the variance structure of the MRD.
   • The variance of the estimator is decomposed into row-variance, column-variance, and interaction-variance components.

2. Interpretation of the Optimal Estimator:

   • Direct Effect: The optimal plug-in estimator corresponds to a Weighted Two-Way Fixed Effects (TWFE) regression with interacted terms. The weights are inversely proportional to the squared group sizes ($1/I_\gamma^2 J_\gamma^2$), resembling the "Tyranny-of-the-Minority" (ToM) regression found in cross-sectional literature.
   • Total/Spillover Effects: The optimal adjustments are generally not representable as simple linear regressions (unlike the Direct Effect), but they are still computable via the derived moment conditions.

3. Inference:
   The paper establishes Central Limit Theorems (CLTs) for these estimators. Crucially, they provide conservative variance estimators that yield valid confidence intervals without assuming the linear model is correctly specified.

C. Theoretical Improvements to MRD Theory

To prove these results, the authors improve the theoretical foundation of MRDs:

• They establish a new CLT for doubly randomized sums in the 1-Wasserstein distance.
• This new CLT relaxes assumptions required by previous work (e.g., [MVR+24]), specifically removing the need for bounded outcomes and allowing for variance scaling regimes where the variance decays faster than $O(1/I + 1/J)$ (e.g., $O(1/IJ)$).

3. Key Contributions

1. Optimal Regression Adjustments for MRDs:

   • Derivation of the minimum asymptotic variance estimator for general estimands in MRDs.
   • Proof that these optimal adjustments are estimable and do not require knowledge of unidentifiable potential outcome contrasts.
   • Demonstration that the optimal adjustment for the direct effect is a weighted TWFE regression, which is distinct from standard OLS or ANCOVA.

2. No-Harm Principle:

   • The proposed plug-in estimators are model-robust. Even if the linear covariate model is misspecified, the adjusted estimator is guaranteed to be at least as efficient as the unadjusted estimator (and often significantly better), unlike naive ANCOVA which can degrade performance in imbalanced designs.

3. Advanced Theoretical Tools:

   • Development of a new CLT for MRDs using Stein's method and concentration inequalities for sampling without replacement.
   • Proof of consistency for conservative variance estimators under broader scaling regimes (including sparse interaction settings).

4. Empirical Validation:

   • Simulations show that the optimal adjustment significantly reduces variance and shortens confidence intervals compared to unadjusted estimators and standard ANCOVA, particularly in imbalanced experiments (where treatment ratios are small).

4. Results

• Efficiency Gains: Numerical simulations demonstrate that the optimal adjustment (Opt) consistently outperforms the unadjusted estimator and ANCOVA. In imbalanced settings, ANCOVA can actually have higher variance than the unadjusted estimator, whereas Opt maintains or improves efficiency.
• Coverage: The constructed confidence intervals maintain the desired coverage probability (e.g., 95%) while being significantly shorter than those from unadjusted methods.
• Direct Effect Interpretation: The paper confirms that for the direct effect, the optimal strategy is a weighted regression that upweights smaller groups, effectively balancing the "tyranny of the minority" in the randomization design.

5. Significance

This paper resolves a major barrier to the practical adoption of Multiple Randomization Designs in industry (e.g., at companies like Amazon).

• Practical Impact: It provides a "plug-and-play" methodology for data scientists running marketplace experiments to increase statistical power without increasing sample size.
• Theoretical Advancement: It bridges the gap between finite-population causal inference and complex experimental designs with interference, offering a rigorous framework that handles the unique scaling properties of two-sided markets.
• Robustness: By proving that these adjustments are model-robust, the authors alleviate concerns that regression adjustments might introduce bias or variance inflation in complex interference settings.

In summary, the paper establishes that optimal regression adjustments are not only possible but essential for efficient inference in two-sided marketplaces, providing a theoretically grounded, empirically superior alternative to standard A/B testing practices in the presence of interference.