📈 economics

Randomization Inference in Two-Sided Market Experiments

This paper develops a randomization inference framework for two-sided market experiments that ensures finite-sample validity under sharp null hypotheses, addresses the limitations of standard asymptotic methods under weak nulls by proposing a two-way variance estimator, and enhances testing power by leveraging the unique two-sided market structure.

Original authors: Jizhou Liu, Azeem M. Shaikh, Panos Toulis

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jizhou Liu, Azeem M. Shaikh, Panos Toulis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive online marketplace, like a digital version of a bustling flea market. You have thousands of Buyers looking for goods and thousands of Sellers offering them.

You want to test a new policy: "Free Shipping."

In a normal experiment, you might just pick a few people and give them free shipping. But in a two-sided market, things get messy. If you give free shipping to Buyer A, but their favorite Seller B doesn't get it, does Buyer A still buy? If you give it to Seller B but not Buyer A, does the deal happen?

This is the problem of interference: The outcome of one transaction depends on the status of both the buyer and the seller. Traditional statistical tools, designed for simple "one-sided" experiments (like testing a new drug on patients), break down here because they assume everyone is independent.

This paper by Liu, Shaikh, and Toulis is like a new rulebook for running experiments in these complex, interconnected markets. Here is the breakdown in simple terms:

1. The Setup: The "Double-Check" Experiment

The authors propose a specific way to run these tests called Multiple Randomization Design.

The Old Way: You might try to randomize every single transaction (Buyer A + Seller B). This is a nightmare to manage.
The New Way: You randomly pick a group of Buyers to get the "Free Shipping" badge, and independently, you randomly pick a group of Sellers to get the badge.
The Magic: A transaction only gets the "Free Shipping" treatment if BOTH the buyer and the seller have the badge. If only one has it, the deal is "untreated."

This creates a grid (a matrix) of possibilities, allowing researchers to see how the market reacts when only one side changes, or when both change.

2. The Problem: The "Blind Spot"

When you run these experiments, you want to know:

Spillover: Does treating the Buyer change how they interact with untreated Sellers?
Total Effect: Does treating both sides create a magic boost?

The problem is that standard math assumes you know exactly what would have happened in every scenario (the "Sharp Null"). But in a two-sided market, you can't know that. You can't know what Buyer A would have done with Seller B if neither had the badge, because you only observed them when one of them had it.

It's like trying to guess the score of a soccer game you didn't watch, but you only have the stats from the games where one team wore red jerseys.

3. The Solution: "Conditional Randomization" (The Detective's Trick)

The authors use a clever trick called Conditional Randomization Tests.

Imagine you are a detective trying to solve a crime. You can't re-enact the whole city, but you can focus on a specific neighborhood where the rules are clear.

The Trick: To test if "Buyers" affect the market, the researchers say, "Okay, let's pretend the Sellers are frozen in time. We will only look at the transactions where the Sellers were not treated."
By freezing the Sellers, the "interference" disappears for that specific group. Now, the experiment looks like a simple, one-sided test again.
They then shuffle (permute) the Buyers around to see if the results change. If the results stay the same no matter how you shuffle the Buyers, then the Buyers didn't actually cause a spillover effect.

They do this for Sellers too (freezing Buyers) and for the "Total Effect" (using a block-based shuffling method).

4. The Twist: When the Math Breaks (Weak Nulls)

The paper also tackles a harder problem: Weak Null Hypotheses.

Sharp Null: "Nothing happened at all." (Easy to test).
Weak Null: "On average, nothing happened." (Harder).

In the real world, we usually care about averages. The authors found that the standard math tools used by statisticians (called Studentization, which is like adding a "safety margin" to your calculations) often fail in two-sided markets.

The Analogy:
Imagine you are weighing a bag of apples.

Standard Tool: You assume the scale is perfect.
The Reality: In a two-sided market, the scale is wobbly because the Sellers are also moving around randomly. The standard tool ignores this wobble, leading to false alarms (thinking you found a difference when you didn't).

The Fix: The authors invented a "Two-Way Variance Estimator."
Think of this as adding a second shock absorber to your scale.

The first shock absorber handles the wobble from the Buyers.
The new, second shock absorber handles the wobble from the Sellers.
By accounting for both sources of randomness, their new math tool stops giving false alarms and gives accurate results.

5. Real-World Test: The Micro-Lending Experiment

To prove it works, they tested their method on real data from Nepal.

The Scenario: A program offered savings accounts to households.
The Twist: They treated households as "Buyers" (those in financial shock) and "Sellers" (those not in shock) to see if offering accounts changed how they lent money to each other.
The Result: Using their new methods, they found no significant change in lending behavior. This suggests that just giving people access to savings accounts isn't enough to change how they help each other in a crisis.

Summary

This paper is a toolkit for scientists and data analysts working in the digital age.

It acknowledges that in marketplaces, everyone affects everyone else.
It provides a method to isolate these effects by "freezing" one side of the market to test the other.
It fixes the math that was previously broken, ensuring that when companies (like Amazon, Uber, or Airbnb) run experiments, they don't get fooled by the complex web of interactions between buyers and sellers.

In short: It teaches us how to run fair experiments in a world where everyone is connected.

1. Problem Statement

Randomized experiments are increasingly used in two-sided markets (e.g., buyer-seller platforms, streaming services) to evaluate policy interventions. However, standard causal inference methods often fail in these settings due to market interference (spillovers). In a two-sided market, the outcome of a specific transaction (a buyer-seller pair) depends not only on the treatment status of that specific pair but also on the treatment status of other buyers and sellers in the market.

The paper addresses three core challenges:

Non-Sharp Null Hypotheses: Traditional Fisherian randomization tests require "sharp" null hypotheses (where all potential outcomes are known under the null). In two-sided markets, spillover hypotheses (e.g., "no buyer spillover") are not sharp because they do not specify outcomes for all treatment assignments, making standard imputation impossible.
Asymptotic Validity of Studentization: While studentized statistics (like t-tests) are commonly used to test "weak" null hypotheses (average effects) in single-sided experiments, the authors demonstrate that standard Neyman-style studentization fails to control Type I error rates in two-sided designs under global average nulls due to unaccounted variance from the second side of randomization.
Power Optimization: There is a trade-off between sample size and the support of the randomization distribution when designing tests for total treatment effects in these complex structures.

2. Methodology

The authors develop a Fisherian-style Randomization Inference (RI) framework tailored for independent two-sided randomized designs (where buyers and sellers are randomized independently).

A. Setup and Assumptions

Design: An independent two-sided design where a buyer-seller pair $(i, j)$ is treated only if both the buyer ( $w^B_i=1$ ) and the seller ( $w^S_j=1$ ) are treated.
Local Interference (Assumption 2.1): Potential outcomes for a pair depend only on their own treatment status and the fraction of treated units on the opposite side. This reduces the infinite treatment space to four distinct exposure cases: $(0,0), (1,0), (0,1), (1,1)$ .
Null Hypotheses:
- Sharp Nulls: No spillover from buyers ( $H_0^{buyer}$ ), no spillover from sellers ( $H_0^{seller}$ ), or no total effect ( $H_0^{total}$ ).
- Weak Nulls: Average spillover effects or average total effects across the population.

B. Conditional Randomization Tests (Sharp Nulls)

To handle non-sharp nulls, the authors employ Conditional Randomization Tests (CRT). They restrict the randomization to a subset of "focal units" and assignments where potential outcomes can be imputed under the null.

Spillover Effects (Buyer/Seller):
- Conditioning: To test for buyer spillovers, the test conditions on the set of control sellers (fixing $w^S_j=0$ ). This ensures that for any permutation of buyer treatments, the seller side remains untreated, making the potential outcomes imputable under $H_0^{buyer}$ .
- Procedure: Permute buyer assignments while keeping seller assignments fixed.
- Symmetry: If the design is symmetric (exchangeable), simple permutations are valid. If not, the authors propose sampling from the conditional distribution $p(W|C)$ , which may require rejection sampling.
Total Effects:
- k-Block Conditioning: To test the total effect ( $H_0^{total}$ ), the authors introduce a k-Block Conditioning Event. The data is partitioned into $k \times k$ diagonal blocks where units within a block share the same treatment status (either all $(0,0)$ or all $(1,1)$ ).
- Procedure: Permute the treatment status of entire blocks rather than individual units.
- Power Trade-off: The authors analyze the trade-off between block size $k$ and test power. Smaller $k$ increases the support of the randomization distribution (more permutations) but reduces the sample size per block. Larger $k$ increases sample size but reduces permutations. They provide a heuristic to select $k$ to maximize power (e.g., targeting a specific maximum power threshold).

C. Weak Null Hypotheses and Studentization

The paper investigates testing average effects using studentized statistics.

Failure of Standard Studentization:
- The authors show that for a Seller-Specific Average Null (where the average effect is zero for every seller), standard Neyman-style studentization is asymptotically valid.
- However, for a Population Average Null (global average is zero), standard studentization is invalid. The sampling distribution of the test statistic includes an additional variance component induced by the randomization of the other side (sellers), which the standard variance estimator fails to capture. This leads to over-rejection (inflated Type I error).
Two-Way Variance Estimator:
- To restore validity for the global weak null, the authors propose a Two-Way Studentized Statistic.
- Mechanism: They augment the standard buyer-side variance estimator ( $V^B$ ) with a seller-side term ( $V^S$ ).
- $V^S$ estimates the sampling variability of the focal mean effect induced by seller randomization. It is constructed using the variance of within-seller difference-in-means across control sellers.
- Result: The new statistic $T^{TW} = T^B / \sqrt{V^B + V^S}$ is asymptotically valid under the global weak null because the denominator correctly accounts for the total variance (buyer + seller randomization).

3. Key Contributions

Novel Inference Framework: The paper provides the first rigorous finite-sample valid randomization inference framework specifically for two-sided market experiments with interference, utilizing conditional randomization tests.
Diagnosis of Studentization Failure: It identifies a critical limitation in applying standard Neyman-style studentization to two-sided experiments under global weak nulls, a problem previously overlooked in the literature.
Two-Way Variance Correction: The proposal of a two-way variance estimator that absorbs the additional sampling variation from the second side of the market, ensuring asymptotic validity where standard methods fail.
Power-Optimized Design: The introduction of the $k$ -block conditioning event for total effects and the theoretical analysis of the trade-off between block size and randomization support, offering practical guidance for experimental design.
Empirical Validation: Extensive simulation studies and an application to a micro-lending network experiment (Comola and Prina, 2021) demonstrating the methods' performance.

4. Results

Simulations (Sharp Nulls): The proposed Fisher Randomization Tests (FRT) maintain exact finite-sample size control (rejection rates $\approx 5\%$ ) under sharp nulls. They generally outperform conservative Neymanian methods in power for spillover effects.
Simulations (Weak Nulls):
- Under seller-specific nulls, standard studentization works.
- Under population-average nulls, standard studentization fails (rejection rates up to 22% at a 5% nominal level).
- The proposed Two-Way Studentization successfully controls the size (rejection rates $\approx 5\%$ ) in both scenarios.
- The paper also shows that the two-way method relies on Assumption B.3 (non-degeneracy of buyer-level averages); if this is violated, even the two-way method may fail.
Empirical Application: Applied to a savings account experiment in Nepal. The authors tested whether offering savings accounts to "shocked" households (buyers) or "non-shocked" households (sellers) affected financial lending links.
- Finding: The randomization tests failed to reject the null hypotheses for buyer spillovers, seller spillovers, or total effects. This suggests that access to savings accounts did not significantly alter the financial network behavior in this specific context, complementing previous findings that effects are context-dependent.

5. Significance

This paper is significant for the growing field of marketplace experimentation. As platforms (Uber, Airbnb, Amazon, etc.) increasingly rely on randomized trials to optimize pricing, matching, and incentives, the interference between users is a major source of bias.

Theoretical Rigor: It bridges the gap between Fisherian exact tests and Neymanian asymptotic inference in complex interference settings.
Practical Utility: The methods are computationally feasible (using permutations) and provide a "plug-in" solution (the two-way variance estimator) for researchers currently using standard t-tests in two-sided markets, potentially preventing false positives.
Design Guidance: The insights on block sizes and conditioning events provide actionable advice for designing future marketplace experiments to maximize statistical power while maintaining validity.