Randomization Inference For the Always-Reporter Treatment Effect

1. Problem Statement

The paper addresses the challenge of conducting statistical inference for treatment effects in Randomized Controlled Trials (RCTs) where sample attrition occurs. Specifically, outcomes are unobserved for a subset of units after randomization.

• The Core Issue: Standard analysis often suffers from selection bias if attrition is related to potential outcomes.
• The Framework: The authors adopt the monotonicity assumption (following [18]), which posits that treatment assignment affects reporting behavior in only one direction (e.g., treatment never causes a unit to stop reporting if they would have reported under control).
• Target Parameter: Under monotonicity, the paper focuses on the Average Treatment Effect for Always-Reporters (AR-ATE). This is the effect for the subpopulation of units who would report their outcomes regardless of treatment assignment ($r_i(1)=1$ and $r_i(0)=1$).
• The Challenge: The set of "always-reporters" is unobservable. While the monotonicity assumption allows for partial identification (bounds) of the AR-ATE, it does not reveal exactly which units belong to this group. Existing methods for inference on AR-ATE rely on asymptotic approximations, which may not hold in finite samples or under specific data configurations (e.g., when the differential attrition rate is negligible).

2. Methodology

The authors propose a worst-case randomization inference procedure that provides strong finite-sample guarantees.

A. The Worst-Case Approach

Since the true set of always-reporters ($A$) is unknown, the method considers the set of all possible reporting tables (configurations of $A$) consistent with the observed data ($D_{obs}, R$).

• Worst-Case P-value: The procedure calculates the randomization p-value for every possible configuration of always-reporters consistent with the data. The final p-value is the maximum (worst-case) of these p-values.
• Rejection Rule: The sharp null hypothesis ($H_0: y_i(1) = y_i(0)$ for all $i \in A$) is rejected if this worst-case p-value is less than the significance level $\alpha$.

B. Two-Balance Condition

A key theoretical insight is that under the sharp null hypothesis, randomization implies two distinct balance conditions:

1. Outcome Balance: The distribution of outcomes among always-reporters should be balanced between treatment and control.
2. Count Balance: The number of always-reporters should be balanced between the treatment and control groups (since assignment is random).
   The proposed test statistics incorporate both conditions, unlike standard tests that only check outcome balance.

C. Test Statistics

The paper introduces three studentized statistics to test the null hypotheses:

1. Studentized Hajek Statistic ($T^0_n$): Based on the difference in means of outcomes, normalized by a variance estimator.
2. Chi-Square Type Statistics ($T^1_n, T^2_n$): These combine the squared Hajek statistic with a term measuring the imbalance in the count of always-reporters.
   • $T^1_n$ uses a standard Wald-type term for count imbalance.
   • $T^2_n$ uses a floor-function adjusted term to handle discrete distributions more robustly.

D. Computational Implementation

Calculating the worst-case p-value involves optimizing over a massive space of possible reporting tables. The authors propose efficient algorithms based on the data type:

• Discrete Outcomes (Finite Support): For binary or categorical data, they exploit the symmetry of the randomization distribution. The problem reduces to enumerating outcome count vectors, allowing for exhaustive search with polynomial time complexity relative to sample size.
• Continuous Outcomes: They reformulate the optimization problem using Integer Linear Programming (ILP).
  • They decompose the problem by the number of always-reporters ($n_A$).
  • They use a bisection search and "matching variables" to transform the non-linear test statistics into quadratic constraints solvable by standard ILP solvers.
  • A pre-testing step (pruning) is used to eliminate implausible configurations of $n_A$ early, reducing computational burden.

3. Key Contributions

1. Dual Guarantees: The procedure offers finite-sample validity for the sharp null hypothesis and asymptotic validity for the weak null hypothesis (where the average treatment effect is zero but individual effects may vary). This is a significant improvement over existing asymptotic methods which often fail in finite samples or specific parameter regions.
2. Handling "Negligible" Attrition: The authors prove uniform asymptotic validity even in parameter regions where the differential attrition rate between treatment and control is negligible (a case often ruled out by previous literature). This is achieved through a refined conditional analysis of the number of always-reporters.
3. Computational Feasibility: They provide practical algorithms (exhaustive enumeration for discrete data, ILP for continuous data) to implement worst-case inference, which was previously considered computationally intractable for moderate sample sizes.
4. New Balance Conditions: The explicit incorporation of the balance of the count of always-reporters into the test statistic creates a more powerful and robust test compared to outcome-only tests.

4. Results

• Theoretical: Theorem 4.2 establishes that the worst-case randomization test controls the Type I error rate at level $\alpha$ for the sharp null in finite samples. Theorem 4.4 establishes asymptotic validity for the weak null under mild regularity conditions (bounded moments, non-negligible always-reporter share).
• Simulation:
  • Size Control: In simulations with $N=100$, the empirical rejection rate under the null (ATE=0) was 0.002 (well below the nominal 0.05), demonstrating strict size control.
  • Power: Under the alternative (ATE=1), the rejection rate was 0.92, indicating high power.
  • Runtime: The heuristic pruning step allows the algorithm to terminate quickly in most cases (median runtime ~35 seconds). In rare cases requiring exhaustive verification, runtimes increased, but the method remained feasible.

5. Significance

This paper bridges a critical gap between design-based inference and missing data problems.

• Robustness: By relying on randomization inference rather than asymptotic approximations, the method is robust to heterogeneity in treatment effects and small sample sizes.
• Practicality: It moves worst-case bounds from a theoretical concept to a computable tool for researchers dealing with attrition in field experiments.
• Generalizability: The framework of using worst-case p-values over latent strata configurations can be extended to other principal strata problems (e.g., Complier Average Causal Effect with non-compliance) where the strata are unobserved.

In summary, Chang and Yu provide a rigorous, computationally feasible, and statistically powerful framework for estimating treatment effects in the presence of sample attrition, ensuring valid inference even when the underlying reporting mechanism is only partially observed.