📈 economics

Randomization Tests in Switchback Experiments

This paper proposes a finite-sample valid, distribution-free randomization-test framework for switchback experiments that addresses temporal interference and serial dependence by utilizing conditional randomization tests based on ex ante design pooling, while providing diagnostics and power analyses to guide robust experimental design and analysis.

Original authors: Jizhou Liu, Liang Zhong

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jizhou Liu, Liang Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, bustling digital marketplace. You want to know if a new feature (like a "Buy Now" button in a different color) actually makes people spend more money.

In a perfect world, you'd flip a coin for every single user: heads, they see the new button; tails, they see the old one. You'd compare the two groups, and the answer would be clear.

But here's the problem: In the real world, users don't exist in a vacuum. If User A sees the new button and buys a shirt, User B (who is their friend) might see that shirt and want one too. This is called interference. If you randomize user-by-user, the "control" group gets contaminated by the "treatment" group, and your experiment fails.

So, what do companies do? They run Switchback Experiments.

The Switchback: A Traffic Light for the Whole City

Instead of changing the button for some people, you change it for everyone for a while, then switch it back for everyone.

Monday 9 AM – 10 AM: Everyone sees the New Button (Treatment).
Monday 10 AM – 11 AM: Everyone sees the Old Button (Control).
Monday 11 AM – 12 PM: Everyone sees the New Button again.

It's like a traffic light for the entire city. You turn the light green for the whole city, then red for the whole city, then green again.

The Three Big Headaches

While this solves the interference problem, it creates three new nightmares for statisticians trying to measure the results:

The Hangover Effect (Carryover): If you show the new button at 10 AM, people might still be excited about it at 10:15 AM, even if you switch back to the old button. The effect "lingers."
The Crystal Ball Effect (Anticipation): If people know the button changes every hour, they might start acting differently before the switch happens because they are "anticipating" the change.
The Short Memory: Companies need answers fast. They can't wait for years of data. They have small sample sizes, but the data is messy, full of sudden spikes (like a viral trend) that break standard math formulas.

The Paper's Solution: A "Smart" Game of "What If"

The authors of this paper (Liu and Zhong) developed a new way to analyze these experiments using Randomization Tests.

Think of a standard statistical test as trying to predict the weather using a complex climate model. If your model is slightly wrong, your prediction is garbage.
The authors say: "Don't model the weather. Just replay the game."

Here is how their method works, using a simple analogy:

1. The "Section" Strategy (Cutting the Cake)

Imagine your experiment is a long loaf of bread. Because of the "Hangover Effect," you can't just slice it anywhere; the crust of one slice might contaminate the crumb of the next.

The Fix: The authors say, "Let's only look at the middle of the bread slices."
They divide the timeline into big chunks called Sections.
They ignore the very beginning and very end of each section (the "burn-in" period) where the hangover effect might still be happening.
They only analyze the "focal" middle parts where the treatment history is clean.

2. The "What If" Replay (The Monte Carlo Simulation)

Once they have their clean "focal" data, they don't use a formula. Instead, they use a computer to play a game of "What If" thousands of times.

The Rule: "We know the rules of how we switched the lights (the assignment mechanism)."
The Game: The computer simulates thousands of alternative timelines where the lights were switched differently, but only in ways that respect the rules and the "clean" sections.
The Comparison: "In 95% of these fake worlds, the result was worse than what we actually saw. Therefore, our result is real."

This gives them a p-value (a measure of certainty) that is mathematically perfect, even with small data and messy shocks, without needing to assume the data follows a "bell curve."

3. The Detective Work (Diagnostics)

The paper also gives tools to check if their assumptions are true:

The Memory Test: "How long does the hangover last?" They have a method to test different time windows (1 hour? 2 hours?) to find the exact point where the effect dies out.
The Crystal Ball Test: "Are people cheating by looking ahead?" They have a test to see if people are changing their behavior before the switch happens.

Why This Matters

In the past, if you ran a switchback experiment with messy data, you might get a result that looks great but is actually a statistical fluke. Or, you might be too conservative and miss a great product feature.

This paper provides a robust, "plug-and-play" toolkit for data scientists. It says:

"You don't need to know the exact shape of the noise in your data."
"You don't need to wait for infinite time."
"You just need to know how you randomized the switches, and we can tell you if your product change actually works."

The Bottom Line

The authors have built a statistical seatbelt for fast-paced tech experiments. It ensures that when a company decides to roll out a new feature based on a switchback test, they aren't just guessing—they are making a decision backed by math that holds up even when the real world gets messy, short, and unpredictable.

1. Problem Statement

Switchback experiments are widely used in online platforms (e.g., marketplaces, ride-sharing) where unit-level randomization is infeasible due to interference, or where outcomes are aggregated. In these designs, treatment and control are alternated over time in blocks. However, reliable inference in these settings faces three major challenges:

Limited Sample Sizes: Operational constraints often force short experiment durations, making asymptotic approximations (which require large $T$ ) unreliable.
Temporal Dependence: Outcomes exhibit serial correlation, seasonality, and heavy-tailed shocks, rendering standard normal approximations poorly calibrated.
Temporal Interference: Outcomes may depend on past treatments (carryover) or future scheduled treatments (anticipation). This violates the "sharp null" assumption required for standard Fisher Randomization Tests (FRT), making naive FRTs invalid or non-imputable.

The paper aims to develop a finite-sample valid, distribution-free inference framework for switchback experiments that does not rely on parametric assumptions about the outcome process.

2. Methodology

The authors propose a Conditional Randomization Test (CRT) framework tailored for time-indexed designs. The core innovation lies in constructing conditioning events that render specific null hypotheses "sharp" (imputable) despite temporal interference.

A. Core Assumptions

The framework relies on two primitive conditions on potential outcomes:

Non-anticipation: Outcomes at time $t$ do not depend on future treatment assignments ( $w_{t+1:T}$ ).
Finite Carryover Horizon ( $m$ ): Outcomes at time $t$ depend only on the treatment history within the last $m+1$ periods ( $w_{t-m:t}$ ).

B. The Conditional Randomization Test (CRT) Framework

Standard FRTs fail in switchbacks because changing an assignment at time $t$ affects outcomes at $t+1, \dots, t+m$ , making potential outcomes unobservable under alternative assignments. The authors solve this by:

Predetermined Sections: They define a fixed, ex-ante partition of time into "sections" by pooling consecutive design blocks.
Conditioning on Constant Sections: They condition on the event that specific sections remain constant (all treatment or all control) under the realized assignment.
Focal Units: Within these constant sections, they identify "focal periods" (the last $n$ periods of the section) where the treatment history is entirely contained within the section. Under the null hypothesis, outcomes in these focal periods are imputable because they depend only on the constant history of that section.
Resampling: The test resamples treatment labels only for the non-focal sections (or specific pairs of sections) while holding the focal outcomes fixed, generating a valid reference distribution.

C. Specific Tests Developed

Total Treatment Effect Test: Tests the null that sustained treatment has no effect ( $Y_t(1^T) = Y_t(0^T)$ ). The CRT pools blocks into sections, conditions on constant sections, and uses a weighted difference-in-means statistic on focal periods.
Carryover Horizon Diagnostic: A sequential testing procedure to estimate the maximum carryover horizon $m$ . It uses an "alternating holdout" strategy (testing even sections against odd predecessors) to check if outcomes depend on assignments beyond $m$ periods. It controls the Family-Wise Error Rate (FWER) by exploiting the nested structure of the hypotheses.
Non-anticipation Diagnostic: Adapts the Pairwise Imputation-based Randomization Test (PIRT) to switchbacks. It restricts randomization to schedules sharing a prefix with the observed assignment to ensure sufficient overlap of "imputable times," testing for dependence on future assignments.
Weak-Null Inference (Studentized CRTs): For testing average effects (where the null is not sharp), the authors introduce studentized CRTs. These use Horvitz-Thompson estimators with conservative variance bounds. They establish asymptotic validity under mild moment conditions, accommodating within-session seasonality (e.g., day-of-week effects).

D. Power Analysis

The paper derives power approximations under a superpopulation model with distributed-lag effects and AR(1) noise. These approximations clarify the trade-offs between:

Block Length: Longer blocks reduce noise via averaging but reduce the number of independent sections.
Burn-in: Necessary to eliminate carryover contamination.
Pooling: Aggregating blocks into sections increases the signal-to-noise ratio but reduces the number of usable randomization units.

3. Key Contributions

Finite-Sample Validity: The proposed CRTs provide exact, distribution-free p-values for switchback experiments without requiring large sample sizes or Gaussian assumptions. This is a significant improvement over existing asymptotic methods which often fail in realistic, short-horizon settings.
Handling Temporal Interference: The paper provides a rigorous solution to the non-imputability problem caused by carryover and anticipation, which has historically invalidated standard randomization tests in time-series experiments.
Diagnostic Tools: It introduces the first finite-sample valid randomization tests for diagnosing the validity of the assumptions themselves (i.e., testing for the existence of carryover or anticipation), rather than just assuming them.
Studentized Weak-Null Tests: It extends randomization inference to weak null hypotheses (average treatment effects) in the presence of carryover, establishing asymptotic validity for studentized statistics.
Design Guidance: The power analysis provides concrete formulas to guide practitioners on choosing block lengths, burn-in periods, and pooling strategies to maximize statistical power.

4. Results

Simulation Evidence: Monte Carlo simulations demonstrate that the proposed CRT maintains near-nominal size (Type I error control) under both Gaussian and heavy-tailed (t-distribution) noise.
Comparison with Alternatives:
- Fisher Randomization Tests (FRT): Under misspecified sharp nulls (common in switchbacks), standard FRTs exhibit severe over-rejection (inflated Type I error), especially with heavy-tailed data.
- Asymptotic Tests: Horvitz-Thompson asymptotic tests are often conservative (low power) at realistic sample sizes.
- Proposed CRT: Matches the power of asymptotic tests under Gaussian noise but dominates them under heavy-tailed disturbances while maintaining valid size.
Diagnostics: The carryover and non-anticipation diagnostics show accurate size control and increasing power as the magnitude of the violation or sample size increases.

5. Significance

This paper bridges a critical gap between theoretical causal inference and the practical realities of online experimentation.

Operational Reliability: It enables product teams to make faster, more reliable decisions in short-duration experiments where traditional statistical methods fail.
Robustness: By being distribution-free, the framework is robust to the heavy-tailed shocks and complex temporal dependencies common in real-world platform data.
Generalizability: While focused on switchbacks, the conditioning-based approach extends naturally to other time-indexed designs, including crossover designs and time-series experiments with interference.
Methodological Advancement: It moves the field beyond asymptotic approximations in interference settings, providing a unified, exact inference framework that respects the temporal structure of the data.

In summary, Liu and Zhong provide a rigorous, practical, and robust toolkit for causal inference in switchback experiments, solving the long-standing problem of valid inference in the presence of temporal interference and limited data.