📈 economics

Causal clustering: design of cluster experiments under network interference

This paper proposes a framework for designing cluster experiments under network interference by formulating optimal clustering as a penalized min-cut problem solvable via semi-definite programming to minimize the worst-case mean-squared error of the global treatment effect.

Original authors: Davide Viviano, Lihua Lei, Guido Imbens, Brian Karrer, Okke Schrijvers, Liang Shi

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Davide Viviano, Lihua Lei, Guido Imbens, Brian Karrer, Okke Schrijvers, Liang Shi

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist trying to figure out if a new fertilizer makes plants grow taller. You have a giant garden with thousands of plants, and you want to test the fertilizer on some and leave others alone.

In a perfect world, you could just pick a few random plants, give them fertilizer, and compare them to the rest. But in the real world, plants don't live in isolation. They share soil, water, and even pests. If you put fertilizer on Plant A, the nutrients might leak over to Plant B next to it. Plant B might grow taller not because it got the fertilizer, but because its neighbor did. This is what the paper calls "spillover effects" or "network interference."

If you ignore this, your experiment will give you the wrong answer. You might think the fertilizer works great, when actually, you're just measuring the effect of the plants helping each other.

The Problem: How to Group the Plants?

To fix this, scientists often use "Cluster Randomization." Instead of picking individual plants, they group them into clusters (like rows or patches). They treat the entire row as a unit: either the whole row gets fertilizer, or the whole row gets nothing. This stops the fertilizer from leaking between the "treated" and "untreated" groups.

But here is the catch: How big should the rows be?

If the rows are too small: The fertilizer still leaks out of the treated row into the untreated row next door. Your results will be biased (wrong).
If the rows are too big: You end up with very few rows. If one row happens to have naturally healthier soil, your results will be noisy (unreliable). You can't tell if the fertilizer worked or if that one row was just lucky.

This is a balancing act. You want to minimize the "messiness" (bias) and the "uncertainty" (variance) at the same time.

The Paper's Solution: A "Smart Map" for Experiments

The authors of this paper have built a new mathematical tool to solve this puzzle. They treat the garden not just as a collection of plants, but as a map of connections (a network). Some plants are close friends (neighbors), others are strangers.

They propose a method called "Causal Clustering." Think of it as a GPS for your experiment that doesn't just look at geography, but looks at relationships.

Here is how their method works, using a simple analogy:

1. The "Cut" and the "Penalty"

Imagine you have a giant piece of fabric with a complex pattern of threads connecting different spots. You want to cut this fabric into separate patches (clusters) to run your experiment.

The Goal: You want to make cuts that separate the "treated" patches from the "untreated" patches as cleanly as possible.
The Penalty: Every time you cut a thread that connects a treated plant to an untreated plant, you pay a "bias penalty."
The Size Penalty: If you make the patches too uneven (one giant patch and many tiny ones), you pay a "variance penalty" because your data becomes unreliable.

The paper's algorithm finds the perfect way to cut the fabric. It solves a complex math problem (called a "penalized min-cut") to find the grouping that results in the lowest total penalty. It's like finding the path through a maze that avoids the most traps while taking the shortest route.

2. The "Magic Number" (The Tuning Knob)

The method uses a "tuning knob" (a parameter the researchers call $\xi$ ). This knob decides how much you care about bias versus variance.

If you turn the knob to care mostly about bias, the algorithm will make many small, tight groups to ensure no spillover happens.
If you turn it to care mostly about variance, it will make fewer, larger groups to get more stable data.
The paper shows you how to set this knob based on how strong you think the "spillover" effect is (e.g., "Do you think the fertilizer leaks a little, or a lot?").

Real-World Tests: Facebook and Chinese Villages

The authors tested their idea in two very different places:

Facebook (The Digital Garden): They looked at the massive network of friendships on Facebook. They compared their "Causal Clustering" against standard ways Facebook already groups people (like "Louvain" or "Balanced Partitioning").
- The Result: Their method showed that existing Facebook groupings were often too messy for experiments. Their new method could find better groupings that would give more accurate results for things like ad campaigns. They found that for many online experiments, grouping people by these smart clusters is actually better than just picking random individuals.
Rural China (The Physical Garden): They used data from a real experiment in 185 villages in China, where they tried to sell insurance.
- The Problem: The villages were the "natural" groups. But people in Village A were friends with people in Village B. The "natural" village borders didn't match the "friendship" borders.
- The Result: Their algorithm ignored the village borders and created new groups based on who was actually friends with whom. This new grouping was much better at measuring the true effect of the insurance than sticking to the official village lines.

The Bottom Line

This paper doesn't just say, "Group people together." It says, "Group people together intelligently based on how they are connected."

It provides a recipe for researchers to:

Look at the network of connections.
Decide how much they fear "leaking" effects (bias) vs. "noisy" data (variance).
Run a calculation that spits out the perfect groups to use for their experiment.

By doing this, they ensure that when they say, "This treatment works," they are actually right, and they aren't just measuring the ripple effects of their neighbors.

Technical Summary: Causal Clustering: Design of Cluster Experiments under Network Interference

1. Problem Statement

The paper addresses the design of cluster experiments in settings where units interact via a known network, leading to spillover effects (interference). In such environments, standard individual-level randomization (Bernoulli designs) often fails to identify the Global Average Treatment Effect (GATE) due to bias induced by cross-cluster exposure. Conversely, cluster randomization eliminates this bias by assigning entire clusters to treatment or control but introduces variance due to the correlation of assignments within clusters and potential heterogeneity in cluster sizes.

The core challenge is that pre-specified administrative clusters (e.g., villages) or standard community detection algorithms may not be optimal for causal inference. They might be too coarse (introducing high variance) or too fine (failing to capture all relevant spillovers, thus introducing bias). The authors frame the choice of clustering as a statistical design problem: selecting a partition of the network that minimizes the worst-case Mean Squared Error (MSE) of the difference-in-means estimator for the GATE.

2. Methodology and Framework

2.1 Local Asymptotic Framework

The authors formalize the problem using a local asymptotic framework where spillover effects are "local to zero" but non-negligible for inference.

Assumption 1 (First-order local interference): Potential outcomes depend only on an individual's treatment and the treatments of their immediate neighbors.
Assumption 2 (Boundedness and Lipschitz continuity): Potential outcomes are uniformly bounded, and the magnitude of spillovers is bounded by a parameter $\bar{\phi}_n$ . Specifically, the change in potential outcomes is linear in the share of treated neighbors, with a slope bounded by $\bar{\phi}_n$ .
Assumption 3 (Spillover local asymptotics): The product of the maximum spillover magnitude and the squared maximum degree is bounded ( $\limsup \bar{\phi}_n N_{n,max}^2 \leq \epsilon$ ). This allows for settings where direct effects and spillover effects are of similar magnitude, a common scenario in online platforms where effects are small but economically significant.

2.2 Characterization of Bias and Variance

The paper derives sharp worst-case characterizations for the bias and variance of the difference-in-means estimator $\hat{\tau}_n(C_n)$ under a given clustering $C_n$ :

Worst-Case Bias: The bias is governed by the "between-cluster connectedness," defined as the average fraction of an individual's neighbors lying outside their assigned cluster ( $b_n(C_n)$ ). The worst-case bias is proportional to $\bar{\phi}_n b_n(C_n)$ .
Worst-Case Variance: Under the local asymptotic framework, the variance is asymptotically driven by within-cluster correlations induced by the clustering design. It is proportional to the sum of squared cluster sizes ( $\sum n_k^2$ ), scaled by the outcome range $\bar{\psi}$ .
Worst-Case MSE: Combining these, the objective function to minimize is:
$B^*_n(C_n, \lambda) = \bar{\psi} \frac{1}{n^2} \sum_{k=1}^{K_n} n_k^2 + \lambda \bar{\phi}_n^2 b_n(C_n)^2$
where $\lambda$ is a user-specified weight (typically 1 for MSE).

2.3 Optimization Algorithm

The authors show that finding the optimal clustering reduces to a novel penalized min-cut problem.

Reformulation: The problem is reformulated as a trace-optimization program with a semi-definite constraint.
Convex Relaxation: Since the exact problem is combinatorial, the authors propose a Semi-Definite Programming (SDP) relaxation. The objective involves minimizing a weighted sum of the trace of the cluster matrix (variance component) and the square of a normalized cut (bias component).
Rounding: A feasible clustering is retrieved by applying K-means clustering to the eigenvectors of the SDP solution.
Algorithm 1: The procedure iterates over a range of cluster numbers ( $K$ ) and a calibration parameter $\xi_n = (\lambda \bar{\phi}_n^2 / \bar{\psi})^{-1}$ , which balances the bias-variance trade-off. The algorithm provides a provable certificate of performance (approximation bound) relative to the optimal clustering.

3. Key Contributions

Decision-Theoretic Framework for Clustering: Unlike existing literature that treats clustering as given or focuses on community detection, this paper derives clustering directly from the causal objective of minimizing worst-case MSE for the GATE.
Sharp Worst-Case Characterizations: The paper provides explicit, sharp bounds for bias and variance under network interference, showing that bias depends on cross-cluster exposure and variance depends on cluster size dispersion.
Novel Optimization Problem: The authors identify that optimal clustering solves a penalized min-cut problem that internalizes both cross-cluster exposure (bias) and cluster-size dispersion (variance). This differs from standard spectral clustering or normalized cuts which do not account for the variance component induced by cluster sizes.
Practical Design Rules:
- Bernoulli vs. Cluster: A rule of thumb is derived to determine when a cluster design dominates a Bernoulli design based on the ratio of spillover magnitude to outcome variance and the number of clusters.
- Unknown Spillovers: The paper extends the framework to handle unknown spillover magnitudes using a minimax regret approach, optimizing for the worst-case ratio of MSE relative to an oracle.
Empirical Validation: The method is applied to unique Facebook network data and a field experiment in rural China (Cai et al., 2015).
- In the Facebook data, the proposed method ranks existing clustering algorithms (Louvain, Balanced Partitioning), finding that Louvain with many clusters often performs best.
- In the field experiment, the method constructs alternative partitions that outperform administrative village boundaries and standard clustering algorithms (Spectral, $\epsilon$ -net) across various signal-to-noise ratios.

4. Results and Significance

The paper claims that the proposed "Causal Clustering" method offers a robust procedure for experimental design in the presence of network interference.

Performance: In simulations calibrated to the China field experiment, Causal Clustering achieves near-zero regret (0.0% - 0.2%) in first-order interference settings, significantly outperforming benchmarks like Spectral clustering and Louvain, which show high regret (up to 50%+) depending on the signal-to-noise ratio.
Robustness: Even under misspecified global interference models (where outcomes depend on neighbors' outcomes rather than just treatments), the method remains competitive, avoiding the significant underperformance seen in other methods.
Practical Utility: The framework allows researchers to quantify the trade-off between bias and variance explicitly. It provides a data-driven way to choose the number and structure of clusters, rather than relying on administrative boundaries or heuristic community detection.

The authors emphasize that their approach is particularly relevant for online platforms (e.g., Meta/Facebook) and field experiments where spillovers are small but non-negligible, and where the goal is to estimate the total effect of a policy at scale. The method bridges the gap between graph clustering literature and causal inference, providing a computationally tractable (via SDP) solution to a complex design problem.