📈 economics

Valid Inference when Testing Violations of Parallel Trends for Difference-in-Differences

This paper proposes simple preliminary tests and corresponding confidence intervals that ensure valid statistical inference for Difference-in-Differences estimates by overcoming the bias and coverage issues of existing methods, relying on a "conditional extrapolation assumption" to link pre-treatment trend violations to post-treatment violations.

Original authors: Jonas M. Mikhaeil, Christopher Harshaw

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas M. Mikhaeil, Christopher Harshaw

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Parallel Trends" Problem

Imagine you are a detective trying to figure out if a new medicine (the treatment) cures a headache. You have two groups of people:

The Treated Group: They take the medicine.
The Control Group: They take a sugar pill.

To know if the medicine worked, you need to know what would have happened to the treated group if they hadn't taken the medicine. Since you can't see the future or the "alternate universe," you make a guess: The Parallel Trends Assumption.

The Assumption: You assume that, without the medicine, the treated group's headaches would have improved (or worsened) at the exact same rate as the control group's headaches. If their lines on a graph were running parallel before the medicine, you assume they would have stayed parallel after.

The Problem: The "Pre-Test" Trap

In the real world, researchers often look at the data before the medicine was given to see if the two groups were actually running parallel.

If the lines look parallel: "Great! The assumption holds. Let's calculate the effect."
If the lines look wobbly: "Uh oh. The assumption might be broken. Maybe we shouldn't trust the results."

The Catch: A recent wave of research has shown that doing this "pre-check" and then deciding whether to report your results creates a statistical mess. It's like a referee who lets a player keep playing only if they look like they aren't cheating, but then uses a broken stopwatch to time the race. The final results become biased, the confidence intervals (the margin of error) become too small, and you might think you found a cure when you didn't.

The Solution: A New Rulebook

The authors of this paper propose a new way to handle this "pre-check" so that the math works out correctly. They introduce a concept called the Conditional Extrapolation Assumption.

The Analogy: The Weather Forecaster

Imagine you are a weather forecaster trying to predict if it will rain tomorrow (the post-treatment period). You look at the weather today and yesterday (the pre-treatment period).

Old Way (Classical DID): You assume if it wasn't raining yesterday, it won't rain tomorrow. You don't check if the wind is blowing weirdly.
The "Bad" Pre-Test Way: You check the wind. If it looks calm, you predict rain. If it looks stormy, you stop. But your prediction tool breaks if you only use it on calm days.
The Authors' New Way: They say, "We have a rule: If the wind today is calm enough (below a specific threshold), we assume the wind tomorrow won't be worse than it is today."

This is the Conditional Extrapolation Assumption. It doesn't promise the wind will be calm tomorrow; it just promises that if today is calm enough to pass a test, then tomorrow won't be a hurricane.

How It Works in Practice

1. The "Severity Test" (The Gatekeeper)
Before doing any math, the researcher runs a simple test to measure how "wobbly" the lines were before the treatment.

They calculate a "severity score" of the wobble.
They compare this score to a pre-set Threshold (let's call it the "Acceptable Wobble Limit").
If the score is below the limit: The test passes. The researcher is allowed to proceed.
If the score is above the limit: The test fails. The researcher must stop and say, "We cannot trust this method because the groups were too different to begin with."

2. The "Wobbly" Confidence Interval
If the test passes, the researcher calculates the effect of the treatment. But here is the clever part: they don't just draw a tiny, precise line around their answer.

They draw a wider confidence interval (a bigger margin of error).
Why? Because they know there was some wobble, even if it was small. They add a "safety buffer" to the math to account for the possibility that the wobble might get slightly worse tomorrow.
If the pre-treatment wobble was tiny, the interval is narrow (like a standard test).
If the pre-treatment wobble was noticeable (but still passed the test), the interval gets wider to be safe.

Why This Is Better

The paper argues that this method fixes the "broken stopwatch" problem.

It admits uncertainty: Instead of pretending the pre-treatment data is perfect, it measures the imperfection and adds it to the final error margin.
It prevents false confidence: If the pre-treatment groups were too different, the test stops you from making a claim at all, rather than letting you make a claim with a falsely narrow margin of error.
It formalizes intuition: It turns the "gut feeling" researchers have when they look at a graph and say, "This looks okay, but not perfect," into a rigorous mathematical rule.

Real-World Examples Used in the Paper

The authors tested this on two real scenarios:

Vietnam Public Services: They looked at whether recentralizing government services improved things like schools and water.
- Result: For schools and water, the "wobble" was small enough to pass the test, so they reported results with wider, safer confidence intervals. For agricultural centers, the "wobble" was too huge, so they correctly refused to make any claim.
Right-to-Carry Laws in Virginia: They looked at whether letting people carry guns changed crime rates.
- Result: When looking at "assault" rates, the groups were moving in very different directions before the law changed. The test failed, and they stopped. When looking at "murder" rates, the groups were closer, so they proceeded with their adjusted, wider intervals.

The Bottom Line

This paper provides a new toolkit for researchers using the "Difference-in-Differences" method. It says: "Don't just check if your groups look parallel and then ignore the check. Instead, measure exactly how much they deviate, set a hard limit on what is acceptable, and if they pass, report your results with a bigger safety margin to account for that deviation."

This ensures that when a study says "We found an effect," we can trust that the math behind it hasn't been broken by the very act of checking the data first.

Technical Summary: Valid Inference when Testing Violations of Parallel Trends for Difference-in-Differences

Problem Statement
The Difference-in-Differences (DID) design is a primary strategy for estimating causal effects in the presence of unmeasured confounding, relying critically on the parallel trends assumption. While this assumption is counterfactual and untestable for the post-treatment period, researchers routinely examine pre-treatment periods to check for trend violations. However, recent literature (e.g., Freyaldenhoven, Hansen, and Shapiro, 2019; Roth, 2022) has demonstrated that using conventional preliminary tests to validate parallel trends induces severe statistical deficiencies in subsequent inference, including bias, low power, and undercoverage of confidence intervals. Applied researchers face a dilemma: preliminary tests are intuitively necessary to assess the plausibility of the research design, yet standard statistical methods fail when conditioned on the results of such tests.

Methodology
The authors propose a framework that formalizes the implicit logic of preliminary testing through a new assumption and derives corresponding inference procedures.

The Conditional Extrapolation Assumption:
The core theoretical contribution is the "Conditional Extrapolation Assumption." Unlike the classical parallel trends assumption (which posits zero violations) or partial identification assumptions (which assume pre-treatment violations unconditionally bound post-treatment violations), this assumption posits a conditional relationship:
- If the severity of pre-treatment parallel trend violations ( $S_{pre}$ ) is below a pre-specified acceptable threshold ( $M$ ), then the severity of post-treatment violations ( $S_{post}$ ) is bounded by $S_{pre}$ (i.e., $S_{post} \leq S_{pre}$ ).
- If $S_{pre} > M$ , no bound on $S_{post}$ is justified, and the DID design is deemed inappropriate.
  Here, severity is measured as an $L_p$ norm of iterative parallel trend violations ( $R(t)$ ).
Preliminary Test:
The authors propose a simple preliminary test to determine if the extrapolation condition holds ( $S_{pre} \leq M$ ). The test statistic is the estimated severity of pre-treatment violations ( $\hat{S}_{pre}$ ). The test rejects the extrapolation condition if $\hat{S}_{pre} \geq M$ .
- Asymptotic Consistency: The test is shown to be asymptotically consistent under a "separation condition." Specifically, if the true $S_{pre}$ is separated from the threshold $M$ by a sequence $s_n$ where $s_n = \omega(n^{-1/2})$ , both Type I and Type II errors vanish as sample size $n \to \infty$ . This differs from conventional hypothesis testing, which seeks to minimize Type II error at a fixed Type I error level; instead, this test prioritizes consistency in well-separated regimes.
Conditionally Valid Confidence Intervals:
If the preliminary test passes (i.e., the extrapolation condition is declared true), the authors construct confidence intervals for the Average Treatment Effect on the Treated (ATT).
- Structure: The interval is centered on the standard DID estimator ( $\hat{\tau}_{DD}$ $\overset{τ}{^}_{D D}$ ) but inflated to account for potential bias. The width includes two components:
  1. A bias term: $\kappa \cdot \hat{S}_{pre}$ , where $\kappa$ is a constant derived from the post-treatment period length and the $p$ -norm choice. This term represents the sharp upper bound on the bias derived from Proposition 3.1.
  2. A statistical fluctuation term: $f(\alpha, \hat{\Sigma}) / \sqrt{n}$ , accounting for the joint uncertainty of the DID estimator and the severity estimate.
- Validity: Theorem 5.1 establishes that these intervals achieve nominal coverage ( $1-\alpha$ ) conditional on the event that the preliminary test passes, provided the data generating process satisfies the separation condition.

Key Results

Consistency: The proposed preliminary test is asymptotically consistent for separating hypotheses where the pre-treatment violation severity is sufficiently far from the threshold $M$ .
Conditional Coverage: The proposed confidence intervals provide valid coverage conditional on passing the test. This resolves the undercoverage issues associated with conventional methods that condition on passing standard pre-tests.
Interval Width Behavior: The width of the confidence intervals scales as $\max(S_{pre}, n^{-1/2})$ $max (S_{p r e}, n^{- 1/2})$ .
- If pre-treatment violations are "present but negligible" (order $n^{-1/2}$ ), the interval width converges at the standard parametric rate ( $n^{-1/2}$ ).
- If violations are non-negligible (order $O(1)$ ), the interval width remains constant, reflecting the persistent bias bound.
Probability of Valid Reporting: Theorem 5.2 shows that the joint probability of passing the test and the interval containing the true effect (valid reporting) also converges to the nominal level under the well-separated null.

Empirical Illustrations
The authors validate their methods using:

Synthetic Simulations: Demonstrating that the conditional coverage of their intervals holds at the nominal level when the extrapolation condition is true, whereas conventional intervals undercover. The simulations also show how interval width scales with the number of pre-treatment periods ( $T_{pre}$ ) and the severity parameter ( $p$ ).
Vietnam Recentralization Data: Re-analyzing Malesky, Nguyen, and Tran (2014). The preliminary test correctly rejects the extrapolation condition for outcomes with severe pre-treatment trend violations (agricultural centers), preventing invalid inference, while providing wider, valid intervals for outcomes with minor violations.
Right-to-Carry Laws (Virginia/Maryland): A semi-synthetic illustration comparing iterative vs. overall violations. The results highlight that the choice of violation metric (iterative $R(t)$ vs. overall $\Delta(t)$ ) and the threshold $M$ significantly impact whether the extrapolation condition is accepted and the resulting interval width.

Significance and Claims
The paper claims to provide the first formal assumption (Conditional Extrapolation) that necessitates preliminary testing, thereby offering a theoretical justification for a common empirical practice that has previously been shown to break standard inference.

Formalization: It formalizes the implicit assumption researchers make when they decide to proceed with DID analysis only after seeing "acceptable" pre-treatment trends.
Remedy: It offers a practical solution to the "pre-test paradox" by providing inference procedures that remain valid after conditioning on the test outcome, provided the separation condition is met.
Modesty: The authors acknowledge that the Conditional Extrapolation Assumption is counterfactual and untestable, similar to the parallel trends assumption itself. They do not claim to verify the assumption but rather to provide a framework where, if the assumption holds and the pre-treatment violations are sufficiently small (separated from the threshold), valid inference is possible. They note that their results rely on the separation condition, meaning inference is not guaranteed for cases where violations are arbitrarily close to the threshold $M$ .

The paper concludes by suggesting future work could extend these procedures to staggered adoption designs and alternative identification strategies, but emphasizes that the current contribution is a formalization of the logic underlying preliminary testing in block-adoption DID designs.