📊 statistics

Bias-Variance Tradeoff of Matching Prior to Difference-in-Differences When Parallel Trends is Violated

This paper extends the analysis of matching prior to Difference-in-Differences by demonstrating that while matching on observed covariates involves a bias-variance tradeoff due to sample size reduction, matching on pre-treatment outcomes is always beneficial for minimizing mean squared error, thereby offering practical guidelines for researchers to improve causal estimates in empirical operations management.

Original authors: Mingxuan Ge, Dae Woong Ham

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingxuan Ge, Dae Woong Ham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Did a specific event (like a new policy or a monetary incentive) actually cause a change in behavior, or was the change just a coincidence?

In the world of business and operations research, the standard tool for solving this is called Difference-in-Differences (DiD). Think of DiD as a "time-traveling comparison." You look at a group that experienced the event (the Treated Group) and a group that didn't (the Control Group). You compare how much they changed before the event versus how much they changed after. If the Treated Group changed differently than the Control Group, you assume the event caused it.

But there's a catch: The Control Group must be a perfect twin of the Treated Group. If they aren't similar to begin with, your comparison is flawed.

To fix this, researchers often use a technique called Matching. Before running the comparison, they go through the Control Group and pick out only the people who look exactly like the Treated Group (same age, same job history, same past performance). This is called Matching-DiD.

For a long time, the debate was simple: "Does matching make our estimate more accurate (less biased)?" The answer was usually "Yes."

This paper adds a new, crucial twist to the story. The authors, Mingxuan Ge and Dae Woong Ham, argue that looking only at accuracy (bias) is like judging a car only by how straight it drives, ignoring how fast it goes. They introduce the concept of Variance (how much your result might bounce around if you ran the experiment again) and Mean Squared Error (MSE), which combines both accuracy and stability.

Here is the breakdown of their findings using simple analogies:

1. The "Sample Size" Trap (Matching on Observed Traits)

Imagine you have a huge bucket of 1,000 potential Control Group members, but only 100 Treated members.

The Old View: "Let's pick the 100 best-matching Control members to match our 100 Treated members. This makes the groups look identical, so our result is accurate."
The New Insight: The authors say, "Wait a minute! By throwing away 900 Control members to find those 100 matches, you are shrinking your data sample."
The Analogy: Imagine trying to guess the average height of a crowd.
- No Matching: You measure 1,000 people. Your guess is very stable (low variance), even if the group isn't perfectly identical to your target.
- Matching: You throw away 900 people to find 100 who look exactly like your target. Your groups are now identical (low bias), but because you only measured 100 people, your guess is "wobbly" and could change a lot if you picked a different set of 100 (high variance).
The Verdict: Sometimes, the "wobble" (variance) caused by throwing away data is so bad that it is actually better NOT to match on observed traits (like age or location) if your sample size is small. The "perfect match" isn't worth the loss of data.

2. The "Magic Mirror" (Matching on Past Outcomes)

Now, imagine you don't just match on who the people are (traits), but on what they did before the event (past performance).

The Finding: The authors discovered that matching on past outcomes is always a win.
The Analogy: Think of the "Past Outcome" as a magic mirror that reflects the hidden, unmeasurable traits of a person (like their natural talent or motivation) that you couldn't see before.
Why it works: When you match on what someone did yesterday, you are automatically matching them on those hidden traits too. You get the benefit of a perfect match (low bias) without the penalty of losing data (low variance).
The Verdict: If you are going to match, always include the person's past performance in the matching process. It reduces the "wobble" and makes the result more stable.

3. The "Goldilocks" Strategy (Balancing Bias and Variance)

The paper concludes that there is no single "best" method. It depends on what you value more:

If you have a huge dataset, you can afford to be picky. You should match on everything (traits and past performance) to get the most accurate answer.
If you have a small dataset, you need to be careful. Matching on traits might hurt you because you lose too much data. In this case, you might be better off not matching on traits at all, or at least being very cautious.

Real-World Example: The Zhihu App

To prove their point, the authors re-analyzed a real study about a Chinese Q&A app called Zhihu. The study wanted to know if paying content creators (the "treatment") made them write more free answers (the "outcome").

The original researchers matched the paid creators with free creators who had similar profiles and past writing habits.
The authors of this paper ran their new "Bias vs. Variance" test on that data.
The Result: They confirmed that the original researchers made the right call. Because the sample size was decent and the "past performance" matching was strong, the benefits of matching (getting a truer answer) outweighed the cost of losing data.

Summary

Old Rule: Always match to make groups look alike.
New Rule: Matching on traits (like age/job) is a gamble; it makes groups alike but shrinks your data, which can make your results shaky.
Golden Rule: Matching on past performance is a free lunch; it makes groups alike and keeps your results stable.
Takeaway: Don't just ask "Is my result accurate?" Ask "Is my result accurate and stable?" Use both measures to decide whether to match and what to match on.

Technical Summary: Bias-Variance Tradeoff of Matching Prior to Difference-in-Differences

Problem Statement
In empirical operations management and related fields, the Difference-in-Differences (DiD) estimator is a standard tool for causal inference, relying on the "parallel trends" (PT) assumption. To strengthen the plausibility of this assumption, researchers frequently employ a "Matching-DiD" (M-DiD) strategy, matching treated and control units on observed covariates prior to estimation. While existing literature (e.g., Chabé-Ferret, 2015; Ham and Miratrix, 2024) has extensively analyzed the bias properties of such estimators, it has largely neglected the variance and Mean Squared Error (MSE) implications. This creates a gap in understanding the overall performance of M-DiD estimators, particularly regarding the trade-off between reducing bias through matching and potentially increasing variance due to sample size reduction. Furthermore, previous studies often rely on population-level limiting expectations rather than finite-sample properties, and some rely on strong normality assumptions.

Methodology
The authors develop a theoretical framework based on a linear structural equation model with unobserved time-varying confounders. The model allows for:

Time-varying effects: The coefficients of observed covariates ( $X$ ) and latent variables ( $\theta$ ) can change over time, leading to potential violations of parallel trends.
Finite-sample analysis: Unlike prior work focusing solely on asymptotic limits, the authors define finite-sample analogues for three estimators:
- Classic unmatched DiD ( $\hat{\tau}_{DiD}$ ).
- DiD after matching on observed covariates ( $\hat{\tau}^X_{DiD}$ ).
- DiD after matching on both observed covariates and pre-treatment outcomes ( $\hat{\tau}^{X,Y_0}_{DiD}$ ).
Relaxed Assumptions: The framework replaces the normality assumptions of previous literature (e.g., Ham and Miratrix, 2024) with Lipschitz continuity conditions on conditional moments, accommodating discrete covariates and broader distributional structures.
Generalization: The analysis is extended from a univariate, two-period setting to a general multivariate setting with multiple pre-treatment periods ( $T > 1$ ), utilizing a two-way fixed effects regression equivalence.

Key Contributions

Finite-Sample Characterization: The paper introduces and proves the consistency of finite-sample DiD and matching DiD estimators, bridging the gap between theoretical limiting values and practical application.
Variance and MSE Analysis: The authors provide exact mathematical characterizations of the variance and MSE for these estimators. They demonstrate that the trade-offs observed in bias do not necessarily hold for variance.
Novel Trade-off Findings:
- Matching on Covariates ( $X$ ): Contrary to bias results which suggest matching on $X$ is always beneficial, the authors show that matching on $X$ involves a sample-size trade-off. When parallel trends are violated, matching on $X$ reduces variance by eliminating variation from $X$ , but this benefit is offset by the reduction in effective sample size (increasing sampling variation). Thus, matching on $X$ is not always recommended when considering variance.
- Matching on Pre-treatment Outcomes ( $Y_0$ ): In contrast, matching additionally on pre-treatment outcomes always reduces variance relative to matching only on $X$ . This is because the effective sample size remains the same (as matching is already performed), and matching on $Y_0$ reduces the irreducible error term and improves the reliability of the latent variable match.
Practical Guidelines: The paper offers step-by-step estimation strategies and guidelines for applied researchers to decide whether to match, and on which variables, based on their specific sample sizes and whether they prioritize bias, variance, or MSE.

Key Results

Bias vs. Variance Divergence: The paper establishes a clear bias-variance trade-off. While matching on $X$ generally reduces bias, it may increase variance if the sample size reduction cost outweighs the benefit of removing covariate imbalance.
The "Reliability" Term: The variance of the estimator matching on pre-treatment outcomes depends on a "reliability" term ( $r_{\theta|x}$ ), which measures how well pre-treatment outcomes predict the latent confounder $\theta$ after controlling for $X$ . As the number of pre-treatment periods increases, this reliability approaches 1, driving the variance of the matched estimator toward the irreducible error floor.
MSE Implications: Because bias and variance move in different directions depending on the matching strategy, the optimal estimator for minimizing MSE is context-dependent. The authors provide estimators for relative reductions in bias, variance, and MSE (under specific sign conditions) to guide decision-making.
Empirical Application: Re-analyzing Wang et al. (2022) regarding monetary incentives on the Zhihu platform, the authors find that the original decision to match on both covariates and pre-treatment outcomes was statistically warranted. The analysis showed significant bias reduction with negligible variance cost, supporting the use of the full M-DiD strategy in that specific context.

Significance and Claims
The paper claims to fill a critical gap in the literature by moving beyond bias-only analysis to a comprehensive bias-variance-MSE framework. Its significance lies in:

Correcting Intuition: It challenges the intuition that matching is always superior, showing that under variance considerations, the classic unmatched DiD may sometimes be preferable due to sample size efficiency.
Managerial Relevance: By providing metrics (MSE) and guidelines relevant to minimizing confidence interval widths and decision-making risks, the work offers practical tools for operations management researchers.
Theoretical Rigor: It generalizes previous results by relaxing normality assumptions and providing finite-sample consistency proofs, making the findings more robust for applied settings with discrete data.

The authors remain modest, acknowledging limitations such as the reliance on a linear structural model and the assumption of time-invariant covariates (though they argue this can be reparameterized to approximate time-varying contexts). They do not claim to eliminate bias entirely but rather to provide a framework for selecting the estimator that minimizes the overall error (MSE) given the specific constraints of the data.