📈 economics

Finite Population Inference for Factorial Designs and Panel Experiments with Imperfect Compliance

This paper establishes a finite population framework with nonparametric estimators and asymptotic distributions to analyze causal effects in factorial designs and panel experiments under imperfect compliance, demonstrating their utility through simulations and a re-examination of a voter mobilization study.

Original authors: Pedro Picchetti

Published 2026-01-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pedro Picchetti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out what makes a plant grow the tallest. You have a garden with thousands of plants, and you want to test two things: giving them water (Treatment A) and giving them fertilizer (Treatment B).

In a perfect world, you would tell half the plants "Get water" and the other half "Don't," and do the same for fertilizer. But in the real world, things get messy. Some plants you told to get water ignore you and stay dry. Some plants you told not to get water sneakily find a puddle and drink anyway. In research terms, this is called imperfect compliance.

This paper, by Pedro Picchetti, is like a new, super-smart rulebook for figuring out the true cause-and-effect in these messy situations, specifically when you are testing multiple things at once (like water and fertilizer) or testing things over time (like watering on Monday, then Tuesday, then Wednesday).

Here is the breakdown of the paper's ideas using simple analogies:

1. The Problem: The "Messy Garden"

Standard methods for analyzing experiments often break down when:

Multiple treatments are involved: You aren't just testing one thing; you are testing combinations (Water only, Fertilizer only, Both, or Neither).
People don't follow orders: Just because you assigned a treatment doesn't mean the subject actually received it.
Time is involved: In a panel experiment, you are watching the same people over time. Did yesterday's treatment affect today's result?

The author argues that if you try to use standard "quick fixes" (like standard statistical tools) in these messy scenarios, you might get the wrong answer. It's like trying to measure the height of a plant while ignoring that some plants drank extra water they weren't supposed to.

2. The Solution: The "Magic Filter" (The Estimator)

The author invents a new mathematical tool (a nonparametric estimator) that acts like a magic filter.

How it works: Imagine you have a list of every single person in your experiment. The tool looks at who was assigned to get the treatment and who actually took it.
The "Complier" Concept: The tool focuses only on the people who did exactly what they were told (the "Compliers"). It ignores the people who refused the treatment or took it when they weren't supposed to.
The "Multiple-Difference" Trick: This is the paper's secret sauce.
- If you want to know the effect of one thing, you compare the "told to do it" group vs. the "told not to" group.
- If you want to know the effect of two things (or a sequence over time), the tool uses a "Difference-in-Differences" approach. It's like a game of "Spot the Difference" but with four or more groups. It compares:
  - (Both assigned) vs. (Neither assigned)
  - (Only A assigned) vs. (Only B assigned)
- By subtracting these differences from each other, the tool cancels out the noise and isolates the true effect of the specific sequence of treatments.

3. The "Finite Population" Perspective

Most statistics books assume you are pulling a sample from an infinite ocean of data. This paper takes a different view: The garden is fixed.

The author says, "Let's assume these specific 1,000 people are the only people that exist for this experiment." The randomness comes only from how we assigned the treatments (who got the phone call, who didn't), not from the people themselves. This allows for a very precise way of calculating confidence intervals (how sure we are of our answer) without making up extra assumptions about how the world works.

4. Real-World Test: The "Phone Call" Experiment

To prove the tool works, the author applied it to a famous real-world experiment about getting young people to vote (Get-Out-The-Vote).

The Setup: Researchers called young voters. Some got a call from a volunteer, some from a professional, some got both, and some got none.
The Mess: Not everyone answered the phone (imperfect compliance).
The Result: Using this new tool, the author confirmed that phone calls did increase voting. Interestingly, they found that getting two calls wasn't twice as effective as getting one; the second call had "diminishing returns." The tool successfully untangled the effects of the volunteer call vs. the professional call, even though the data was messy.

5. Why This Matters

The paper provides a rigorous, "design-based" way to analyze complex experiments where:

There are multiple treatments (Factorial designs).
Treatments happen over time (Panel experiments).
People don't always follow the rules (Imperfect compliance).

It offers a new way to calculate the "true" effect of a treatment sequence, ensuring that researchers don't get fooled by the chaos of real-world data. The simulations in the paper show that this new tool is accurate and reliable, whereas older tools often give biased (wrong) answers in these specific scenarios.

In short: The paper gives researchers a better set of glasses to see the true cause-and-effect in complex, multi-step experiments where people don't always do what they are told.

Technical Summary: Finite Population Inference for Factorial Designs and Panel Experiments with Imperfect Compliance

1. Problem Statement

Standard Instrumental Variables (IV) methods, specifically the Local Average Treatment Effect (LATE) framework established by Imbens & Angrist (1994), are well-suited for cross-sectional settings with a single binary treatment and a single binary instrument. However, many empirical applications deviate from this canonical setting. Two prominent examples are factorial designs (where multiple treatments are assigned simultaneously) and panel experiments (where treatments and outcomes are observed over time). In both settings, imperfect compliance is common: units assigned to treatment do not always take it, and units assigned to control do not always remain untreated.

In these complex settings, standard IV estimands (such as two-stage least squares, 2SLS) often lack a straightforward causal interpretation. Standard methods may fail to account for the specific sequences of treatments or the dynamic nature of treatment uptake, potentially leading to misleading conclusions regarding causal effects. Furthermore, existing literature on design-based inference in IV settings has largely focused on single-treatment scenarios, leaving a gap in finite population inference procedures for multi-factor and dynamic settings with imperfect compliance.

2. Methodology and Framework

The paper develops a finite population, design-based framework for causal inference. This approach treats the potential outcomes and potential treatments as fixed quantities for a finite population of $N$ units, with uncertainty arising solely from the randomization of assignments.

2.1 Core Assumptions

The identification strategy relies on a set of standard assumptions adapted for multi-factor and dynamic settings:

No-Spillovers (SUTVA): Potential outcomes and treatments for a unit depend only on that unit's own assignments and treatments, ruling out interference between units.
Exclusion Restrictions:
- Outcome Exclusion: Assignments affect outcomes only through their effect on treatment uptake.
- Treatment Exclusion: In factorial designs, the uptake of treatment for factor $k$ depends only on the assignment for factor $k$ . In panel experiments, the treatment uptake at time $t$ depends only on the assignment at time $t$ .
Monotonicity: Units assigned to treatment are at least as likely to take the treatment as those assigned to control. This allows for the classification of units into Always-takers, Never-takers, and Compliers.
Individualistic/Sequential Randomization: Assignments are independent of potential outcomes and treatments, and independent across units (or sequentially randomized in panel settings).
Common Support: The propensity scores for all possible assignment sequences are bounded away from zero and one.

2.2 Identification Strategy

The paper defines causal effects for sequences of treatments (vectors of factors or paths over time).

Target Parameters: The focus is on the average causal effect for the subpopulation of compliers for a specific sequence of assignments. This includes joint effects of multiple factors (in factorial designs) and lag- $p$ dynamic effects (in panel experiments).
The Multiple-Differences Approach: The core identification insight is that causal effects for a sequence of treatments can be identified by exploiting variations in the corresponding sequence of assignments using a multiple-differences format.
- For a single factor, this reduces to a standard Wald estimator.
- For two factors (or two time periods), it takes the form of a difference-in-differences.
- For three factors (or periods), it takes the form of a triple-differences, and so on.
Horvitz-Thompson Estimators: The author utilizes Horvitz-Thompson estimands, weighted by the inverse of the assignment probabilities (propensity scores), to identify local response functions. The causal effect is then derived as the difference between these identified response functions.

2.3 Estimation and Inference

Nonparametric Estimators: The paper proposes nonparametric estimators for the identified causal effects. These estimators are constructed by plugging in sample analogs of the Horvitz-Thompson quantities.
Asymptotic Properties: The paper derives the finite population asymptotic distributions of these estimators as the population size $N$ increases. The estimators are shown to be consistent over the randomization distribution and asymptotically normal.
Inference: To handle the unobservability of individual potential outcomes (which prevents exact variance calculation), the paper proposes conservative confidence intervals based on the upper bound of the variance of the reduced form, following the approach of Bloom (1984). These intervals are valid under the stated assumptions, though they may be conservative in the presence of heterogeneous causal effects.

3. Key Contributions

Novel Identification for Multi-Factor/Dynamic Settings: The paper provides a nonparametric identification approach for causal effects in factorial designs and panel experiments with imperfect compliance. It defines a rich class of causal effects (joint and dynamic) that capture the relative effectiveness of alternative treatment sequences.
Design-Based Finite Population Inference: Unlike previous works that often rely on super-population assumptions, this paper develops a purely design-based perspective. It derives finite population asymptotic distributions for IV estimators in settings with multiple treatments and instruments.
Extension of Dynamic IV Literature: The paper extends the results of Bojinov et al. (2021) to panel experiments with imperfect compliance. It demonstrates that standard IV estimands (like period-specific 2SLS) generally do not have a straightforward causal interpretation in the presence of dynamic causal effects, even when exclusion restrictions hold.
New Estimators: It introduces specific Horvitz-Thompson type estimators (single-stage for single factors, multiple-differences for sequences) and provides valid inference procedures (conservative confidence intervals) for these estimators.

4. Results

4.1 Monte Carlo Simulations

The paper conducts extensive Monte Carlo simulations to evaluate the finite-sample properties of the proposed estimators.

Factorial Designs: In simulations with $N=1,000$ and $K=2$ factors, the proposed estimators exhibit negligible bias and confidence intervals with coverage rates close to the nominal 95% level.
Panel Experiments: In simulations with $N=1,000$ and $T=2$ periods, the proposed Horvitz-Thompson estimators remain consistent and unbiased. In contrast, standard period-specific 2SLS estimators and multivariate 2SLS estimators exhibit severe bias and poor coverage (far below 95%) when dynamic effects are present. The simulations highlight that as the number of time periods increases, standard 2SLS methods perform increasingly poorly due to time-varying heterogeneity.

4.2 Empirical Application: GOTV Experiment

The paper applies the factorial design estimator to a famous voter mobilization experiment by Nickerson (2007), which studied the effects of volunteer and professional phone bank calls on youth voter turnout.

Findings: The analysis confirms that both volunteer and professional calls have positive causal effects on turnout.
Interaction Effects: The results suggest diminishing returns to GOTV efforts. The effect of receiving both calls is not significantly larger than receiving just one, and the interaction term implies a negative effect of combining treatments.
Inference: The estimates are statistically significant under the proposed conservative inference framework. The paper notes that while previous studies (Nickerson 2007, Blackwell 2017) found similar point estimates, the interpretation of uncertainty differs: this paper treats the population as fixed and assignments as stochastic, providing valid finite population confidence intervals.

5. Significance and Claims

The paper claims to make a significant contribution to the causal inference literature by bridging the gap between standard IV methods and complex experimental designs involving multiple treatments and time dynamics.

Addressing Misleading Conclusions: It addresses the risk of misleading conclusions from standard IV methods in factorial and panel settings with imperfect compliance.
Theoretical Rigor: By providing a finite population framework, it offers a rigorous foundation for inference that does not rely on super-population assumptions, which is particularly relevant for randomized experiments where the population is fixed.
Practical Utility: The proposed estimators and inference procedures are shown to perform well in simulations and provide a robust tool for re-analyzing existing experimental data, as demonstrated in the GOTV application.

The paper concludes that under standard IV assumptions and treatment exclusion restrictions, causal effects associated with sequences of treatments can be identified and estimated using a two-stage multiple-differences approach, offering a reliable alternative to standard methods in complex experimental settings.