📈 economics

Randomization Tests in Randomized Saturation Designs

This paper develops finite-sample valid and asymptotically justified randomization tests for various null hypotheses regarding spillover effects in randomized saturation designs, including individual-level, average, and monotonicity assumptions, and demonstrates their practical utility through simulations and a real-world application.

Original authors: Jizhou Liu, Azeem M. Shaikh, Liang Zhong

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jizhou Liu, Azeem M. Shaikh, Liang Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a new teaching method works. But there's a twist: you can't just test it on individual students in isolation. If Student A learns the method, they might help Student B learn it too, even if Student B wasn't officially assigned the method. This is called a spillover effect.

To study this, researchers use a Randomized Saturation Design. Think of this like a school district experiment:

The Clusters: Instead of testing students one by one, you test whole classrooms (clusters).
The Saturation: You assign different classrooms to have different "densities" of the new method.
- Classroom A gets 0% of students using the method.
- Classroom B gets 33% of students using it.
- Classroom C gets 66%.
- Classroom D gets 100%.
The Goal: You want to see if the students who didn't use the method (the untreated ones) still learned more just because their classmates did.

The Problem:
Usually, statisticians use big, complex math formulas to guess if these results are real or just luck. But these experiments often have very few classrooms (maybe only 10 or 20). When you have such a small group, those big formulas often break down and give wrong answers. Also, the classrooms are often different sizes, making the math even messier.

The Solution: The "What If" Game (Randomization Tests)
The authors of this paper propose a smarter way to play the "What If" game. Instead of guessing with formulas, they simulate thousands of alternative realities based only on the rules of the experiment.

Here is how they do it, broken down by the three main tools they invented:

1. The "Focal Unit" Filter (For Specific Comparisons)

Imagine you want to compare the 33% classroom against the 0% classroom.

The Trick: You don't look at every student in the room. You pick a few specific "focal" students who didn't use the method in both classrooms.
The Game: You pretend to swap the labels of the classrooms. You imagine, "What if Classroom A was actually the 0% room and Classroom B was the 33% room?"
The Result: Because you only looked at students who didn't use the method, and you know the rules of the game, you can calculate exactly how likely it is that the results you saw happened by pure chance. This gives a perfectly accurate answer for small groups, no matter how weird the data looks.

2. The "Studentized" Score (For Average Effects)

Sometimes, you don't care if every single student is affected; you just want to know if the average student in the 33% room did better than the average in the 0% room.

The Challenge: The "perfect" math trick above doesn't work for averages because the "What If" game gets messy when you only care about the average.
The Fix: The authors created a special "scorecard" (a studentized statistic) that adjusts for how much the data naturally varies.
The Result: While this isn't a "perfect" answer for a tiny group, it becomes extremely accurate as you add more classrooms. It bridges the gap between the small-group "perfect" math and the big-group "approximate" math.

3. The "Monotone" Check (For Multiple Levels)

What if you have four levels (0%, 33%, 66%, 100%) and you want to know: "Does the effect get stronger as we add more students using the method?"

The Old Way: You would have to run three separate tests (0 vs 33, 33 vs 66, 66 vs 100) and hope they all line up. This is like checking if a ladder is straight by measuring each rung separately; it's prone to errors.
The New Way: The authors built a "Global Monotone Test." It looks at the entire ladder at once. It asks, "Is there any way to rearrange these classrooms that would make the results look like a straight, upward-sloping line?"
The Result: It gives a single, perfectly accurate answer for the whole trend, even with small groups, without needing to run multiple separate tests.

Real-World Test: The Zomba Experiment

To prove their methods work, the authors applied them to a real experiment in Malawi called the Zomba Cash Transfer Program.

The Setup: They looked at schools where different percentages of girls were offered cash to stay in school.
The Question: Did the girls who weren't offered cash still benefit (or suffer) because their classmates were?
The Outcome: They ran their new tests. The results showed that, for the specific questions they asked, the data did not provide strong evidence that the spillover effects were happening in the way they hypothesized. Crucially, they showed that their new methods could handle the messy, small-group data of this real-world experiment much better than old methods could.

The Big Takeaway

This paper is like giving researchers a new, more reliable toolkit for studying how people influence each other in groups.

If you have a small group, they give you a method that is exact (no guessing).
If you care about averages, they give you a method that gets more accurate as you get more data.
If you have many levels of treatment, they give you a way to test the whole pattern at once.

They didn't invent new drugs or new teaching methods; they invented a better way to measure if those things work when people are influencing each other.

Technical Summary: Randomization Tests in Randomized Saturation Designs

Problem Statement

Randomized saturation designs are a primary tool for estimating spillover effects in clustered populations. In these designs, clusters are first assigned to different treatment saturation levels (the proportion of treated units within a cluster), and units are subsequently randomized within clusters conditional on the assigned saturation. While these designs effectively separate own treatment effects from peer treatment intensity, valid statistical inference remains challenging.

Two primary obstacles hinder inference in this setting:

Small Sample Sizes: The number of clusters is often too small to justify large-sample approximations, and cluster sizes are frequently heterogeneous. This necessitates exact finite-sample methods rather than asymptotic normal approximations.
Partial Sharpness: Many hypotheses of interest regarding spillover effects (e.g., no spillover, bounded spillover, or average spillover) are only partially sharp. They restrict potential outcomes under specific exposure conditions but do not impute the full schedule of missing potential outcomes across the entire assignment space. Consequently, naive randomization tests that resample the full assignment vector fail to generate valid reference distributions.

Existing literature has focused largely on large-sample estimation or conditional randomization tests under sharp nulls. There is a gap in developing finite-sample valid randomization tests for partially sharp, bounded, and weak average null hypotheses within multi-saturation designs.

Methodology

The paper develops a suite of randomization tests tailored to the structure of randomized saturation designs under the assumption of homogeneous partial interference (where outcomes depend only on own treatment status and the number of treated peers in the cluster). The methodology is divided into three main components based on the nature of the null hypothesis and the number of saturation levels.

1. Conditional Tests for Two Saturation Levels

For a fixed pair of saturation levels $s$ and $s'$ , the authors propose a conditional randomization framework that conditions on "focal units."

Focal Unit Selection: From clusters where the observed saturation is either $s$ or $s'$ , a fixed number of untreated units are sampled. These units serve as the basis for comparison.
Relabeling Distribution: The test generates a reference distribution by permuting the saturation labels ( $s$ and $s'$ ) across the selected clusters while keeping the focal units and their observed outcomes fixed. This reduces the problem to a cluster-level relabeling problem.
Null-Specific Statistics:
- Partially Sharp Null ( $H_{0,PS}$ ): Assumes $Y_i(0, s) = Y_i(0, s')$ for all units. Under this null, the focal outcomes are invariant to relabeling, allowing any focal statistic (e.g., difference in means) to yield a finite-sample valid test.
- Bounded Null ( $H_{\delta,B}$ ): Assumes $Y_i(0, s) - Y_i(0, s') \leq \delta$ . The test uses the equality boundary ( $=\delta$ ) as a "least-favorable" imputation. By shifting the observed outcomes by $\delta$ , the test constructs a conservative statistic that maintains finite-sample validity for the inequality null.

2. Asymptotic Tests for Weak Average Nulls

For hypotheses concerning average spillover effects (e.g., unit-average or cluster-average spillovers), the null does not determine missing potential outcomes, rendering exact finite-sample conditional tests impossible.

Approach: The authors utilize the same conditional relabeling distribution but pair it with a studentized statistic (using Neyman variance estimators).
Validity: While the conditional distribution is not an exact finite-sample null distribution, the authors prove that the studentized relabeling statistic delivers asymptotically valid inference under regularity conditions (e.g., stable design fractions and bounded moments). This bridges the Fisherian randomization perspective with Neymanian large-sample inference.

3. Unconditional Tests for Global Monotonicity

For designs with multiple ordered saturation levels, the paper addresses the global null hypothesis that untreated outcomes are monotone in the saturation level.

Pairwise Imputation: Instead of decomposing the problem into pairwise contrasts (which requires multiple testing corrections), the authors develop an unconditional randomization test based on pairwise imputation (PIRT).
Mechanism: The test compares the observed assignment against all other feasible assignments in the design support. It utilizes a statistic that is "pairwise-imputable," meaning it depends only on units that are untreated under both the observed and the reference assignment and whose exposures lie within the ordered set covered by the null.
Validity: This construction yields a finite-sample valid unconditional test for the global monotone null, avoiding the need to combine p-values from local contrasts.

Key Results

The paper establishes the following theoretical results:

Finite-Sample Validity (Conditional): For partially sharp and bounded nulls involving two saturation levels, the proposed conditional randomization tests (Procedure 3.1) control the Type I error rate exactly in finite samples, conditional on the selected focal set.
Asymptotic Validity (Weak Nulls): For weak average nulls, the studentized conditional randomization test controls the asymptotic size, providing valid inference even when the null is not sharp.
Finite-Sample Validity (Global Monotonicity): The Pairwise-Imputable Randomization Test (PIRT) for the global monotone null (Procedure 5.1) is proven to be finite-sample valid unconditionally. The test is conservative but directly targets the global shape of the spillover response.
Empirical Performance: Simulations calibrated to the Zomba Cash Transfer experiment demonstrate that:
- The conditional CRT for bounded nulls maintains size close to nominal levels (average size ~0.051) and offers better power than regression-based benchmarks in small samples.
- The global monotone PIRT is more conservative than combining adjacent pairwise tests via Bonferroni but offers superior power for detecting large violations of monotonicity, particularly as the number of saturation levels increases.

Significance and Contributions

The paper makes three primary contributions to the literature on causal inference under interference:

Extension of Conditional Randomization Tests: It extends the conditional randomization framework (previously applied to binary first-stage designs with single treated units) to multi-saturation designs with multiple treated units per cluster. This addresses the more complex assignment spaces and partial imputation problems common in empirical applications.
Inference for Weak Nulls: It provides a pathway for randomization-based inference on weak average spillover nulls. By combining conditional relabeling with studentization, the paper connects Fisherian exact testing with Neymanian asymptotic inference, allowing researchers to test scientifically meaningful average effects even when exact finite-sample validity is unattainable.
Global Monotonicity Testing: It moves beyond equality nulls to test bounded and monotonic restrictions. The development of the PIRT for global monotonicity allows researchers to test the entire shape of the spillover response in a single framework, rather than relying on local pairwise comparisons. This is particularly relevant for applications where the direction of spillovers (e.g., increasing or decreasing) is the primary question.

The authors emphasize that the core insight is the separation of the conditioning step (selecting imputable units) from the statistic (tailored to the specific null). This modular approach allows for valid inference across a spectrum of hypotheses, from sharp individual effects to weak average effects and global monotonicity, within the challenging context of randomized saturation designs.