📊 statistics

Discussion of "Matrix Completion When Missing Is Not at Random and Its Applications in Causal Panel Data Models"

This paper applauds Choi and Yuan's (2025) novel matrix completion approach for estimating causal effects in panel data with non-random missingness, situating it within the broader "split-apply-combine" framework while addressing the gap between theory and practice through an empirical application to right-to-carry laws.

Original authors: Eli Ben-Michael, Avi Feller

Published 2026-02-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eli Ben-Michael, Avi Feller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how a new law (like a "Right to Carry" gun law) changes the crime rate in different states. You have data for every state over many years. But here's the catch: you can't see what would have happened if the law hadn't been passed. That missing information is the "counterfactual."

This paper is a discussion about a new mathematical tool called Matrix Completion that tries to fill in those missing pieces of the puzzle. The authors, Eli Ben-Michael and Avi Feller, are reviewing a new method proposed by Choi and Yuan (CY) and asking: "Does this tool work in the real world, or is it just a fancy theory?"

Here is the breakdown in simple terms, using some everyday analogies.

1. The Big Idea: The "Split-Apply-Combine" Strategy

The authors compare the new method to a popular kitchen strategy called "Split-Apply-Combine."

The Old Way (The "Big Pot" Soup): Imagine you have a giant pot of soup (all your data) and you try to taste it all at once to guess the flavor. If the soup has weird chunks (different states, different times), you might get a bad taste. This is like the old "Two-Way Fixed Effects" model that tries to average everything together, often missing the nuances.
The New Way (The "Tasting Spoon" Strategy):
1. Split: Instead of tasting the whole pot, you take a small spoonful of just the people who just got the law, and a spoonful of people who haven't gotten it yet.
2. Apply: You use a fancy algorithm (Matrix Completion) to guess what the "just treated" group would have looked like if they hadn't gotten the law, based on the "untreated" group.
3. Combine: You do this for every group and every year, then average all those guesses together to get the final answer.

The authors say this is smart because it avoids comparing apples to oranges (like comparing a state that got the law 20 years ago to one that got it yesterday).

2. The "Last Mile" Problem: Theory vs. Reality

The authors use a great analogy here: The Last Mile Problem.

The Theory: Imagine a delivery truck that can drive perfectly on a highway (the math works perfectly in a textbook).
The Reality: But getting the package from the local depot to your front door (applying it to real data) is messy. There are potholes, traffic, and weird addresses.

The authors argue that while the new math is brilliant on the highway, we need to make sure it works on the bumpy roads of real life. They suggest three things to fix this:

Time Travel Confusion: Are we measuring the effect 1 year after the law, or 1 year after the law started in that specific state? (Event time vs. Calendar time).
The "Tuning Knob": The math has a dial (hyper-parameter) that needs to be set just right. If it's too tight, you miss the signal; too loose, you get noise. We need better ways to find that perfect setting automatically.
The "Fake Test": Before trusting the results, you should run a "placebo test." Pretend the law passed 10 years ago when it didn't. If the math says crime changed back then, the tool is broken.

3. The Real-World Test: Gun Laws and Crime

To see if this works, the authors applied the method to a famous, messy dataset: Right to Carry (RTC) laws and violent crime.

The Setup: They looked at 50 US states over 40 years. Some states passed the law early, some late, some never.
The Problem: When they used the new Matrix Completion tool without any adjustments, it gave a shocking result: It predicted that these laws caused a massive explosion in violent crime (hundreds of extra crimes per 100,000 people).
The Reality Check: The authors compared this to other methods (like Synthetic Control, which builds a "fake twin" for each state). Those other methods said the effect was tiny or non-existent. The Matrix Completion result was an outlier, likely because the data was too messy and the tool was "over-fitting" (trying too hard to find patterns that weren't there).

The Fix:
They realized that states have different "baseline" crime levels (some are just naturally safer or more dangerous). So, they did a simple pre-step: they subtracted the average differences between states and years first (like leveling the playing field).

The Result: Once they leveled the playing field, the Matrix Completion tool finally agreed with the other methods. It stopped predicting a crime explosion and showed a small, reasonable decrease in crime.

The Takeaway

The paper is essentially saying:

The new tool is promising: The "Split-Apply-Combine" strategy is a smart way to handle messy data where laws are passed at different times.
But be careful: If you just plug the data in without cleaning it first, the tool might give you wild, impossible answers (like predicting a crime apocalypse).
Preparation is key: Just like you need to prep your ingredients before cooking, you need to "clean" your data (remove fixed effects) before using these fancy math tools.

In short: The new math is a powerful engine, but you still need a good driver (the researcher) to steer it safely through the traffic of real-world data.

Based on the discussion paper by Eli Ben-Michael and Avi Feller regarding Choi and Yuan (2025) (CY), here is a detailed technical summary covering the problem, methodology, contributions, results, and significance.

1. Problem Statement

The paper addresses the challenge of estimating causal effects in panel data where missingness is Not At Random (MNAR). Specifically, it focuses on "staggered adoption" scenarios where units (e.g., states) enter treatment at different times, creating a structured pattern of missing counterfactuals (the outcomes that would have occurred had treated units remained untreated).

While standard matrix completion methods often assume Missing At Random (MAR), CY propose that matrix completion can remain effective even under MNAR (selection into treatment) provided the number of treated observations is small relative to the number of control observations. However, Ben-Michael and Feller highlight a "last mile problem": the gap between theoretical guarantees of such estimators and their practical performance in real-world policy evaluation, particularly regarding interpretability, inference robustness, and hyper-parameter selection.

2. Methodology and Framework

The "Split-Apply-Combine" Strategy

The authors frame CY's proposal within a general "split-apply-combine" framework, which is also the backbone of modern Difference-in-Differences (DiD) and Synthetic Control (SC) methods:

Split: The panel data is broken into manageable subsets.
- CY Approach: For a focal time $t_0$ , the data is split into a set of treated units (with the same treatment time) and all not-yet-treated units for pre-treatment and focal periods.
- Contrast with DiD: Traditional DiD splits data into two-period, two-group cases to avoid "forbidden comparisons."
Apply: An estimator is applied to each subset.
- CY Approach: Uses nuclear-norm regularized matrix completion to impute missing counterfactuals ( $Y_{it}(\infty)$ ) for the treated units in the focal period.
Combine: Estimates from subsets are aggregated.
- CY Approach: Averages the difference between observed outcomes and imputed counterfactuals across treated units to estimate the Average Treatment Effect on the Treated (ATT).

Key Technical Distinctions

Calendar Time vs. Event Time: CY target the calendar time ATT ( $\tau^{cal}_{t_0}$ ), averaging effects across units treated at a specific calendar date. Ben-Michael and Feller argue that event time ATT ( $\tau^{event}_k$ ), which averages effects based on time since treatment ( $k$ periods post-treatment), is often more interpretable for policy dynamics. However, estimating event time effects in low-rank matrix models is theoretically complex due to the aggregation of different calendar times and singular values.
Identifiability: The authors note that while CY provide bounds for the structural component of Individual Treatment Effects (ITE), individual counterfactuals are fundamentally unidentifiable due to idiosyncratic errors. Therefore, focusing on the ATT (averaging across units) is a more robust target.
Combine-Apply vs. Split-Apply-Combine: The authors suggest that for ATT estimation, a "combine-apply" strategy (averaging treated units first, then finding a synthetic control) might be more efficient than CY's "split-apply-combine" approach, similar to "fine balance" approaches in weighting.

3. Key Contributions and Critiques

Ben-Michael and Feller do not propose a new estimator but offer a critical discussion and extension of CY's work:

Conceptual Framing: They situate CY's matrix completion approach as a specific instance of the split-apply-combine strategy, clarifying its relationship to DiD and Synthetic Control methods.
Addressing the "Last Mile Problem": They identify practical hurdles in deploying theoretical estimators:
- Interpretability: The difficulty of interpreting calendar-time averages when treatment effects evolve over time.
- Inference: The need for heteroskedastic-robust variance estimators, which CY partially addressed for block missingness but needs extension for staggered adoption.
- Hyper-parameter Selection: The challenge of selecting regularization tuning parameters in structured missing data. They critique standard cross-validation (which may ignore missing data structure) and suggest hold-out pre-treatment periods (similar to SC methods) as a better alternative.
- Diagnostics: They emphasize the need for robust diagnostics beyond standard event study plots. They propose "in-time-placebo" checks (shifting treatment times forward) to detect over-fitting or assumption violations, noting that zero pre-treatment effects can sometimes indicate over-fitting rather than validity.

4. Empirical Results: Right-to-Carry (RTC) Laws

The authors apply CY's methodology (and others) to a classic policy evaluation: the effect of Right-to-Carry (RTC) laws on violent crime rates across US states (1977–2014).

Data Characteristics: $N \approx 50$ states, $T \approx 37$ years. The data is ill-conditioned (high condition number), with staggered adoption ranging from 1980 to 2009.
Estimators Compared:
1. Nuclear-norm regularized Matrix Completion (Full Matrix).
2. CY Estimator (Split-apply-combine).
3. Partially Pooled Synthetic Control (PPSC).
4. Gsynth (Factor model).
5. Standard DiD.

Findings:

Without Pre-processing (Raw Data):
- Matrix completion estimators produced implausibly large positive effects, estimating an increase of ~205 violent crimes per 100,000 (CY estimator: ~136).
- Alternative methods (SC, Gsynth) estimated much smaller effects (~11 and ~4, respectively).
- Pre-treatment placebo estimates were near zero for matrix completion, but the post-treatment magnitudes were unrealistic compared to the variance in the raw data.
With Pre-processing (Residualizing Fixed Effects):
- When unit and time fixed effects were removed before applying matrix completion, the results changed drastically.
- Matrix completion estimates aligned with alternative methods, showing a decrease in violent crime (CY: -10.1; Full Matrix: -8.5).
- This suggests that for ill-conditioned panel data with large unit/time heterogeneity, pre-processing to remove fixed effects is crucial for matrix completion to yield credible results.

5. Significance and Conclusion

Theoretical Validation: The paper validates that matrix completion can be a viable tool for MNAR panel data but highlights that its success is highly sensitive to data conditioning and pre-processing.
Practical Guidance: It provides a roadmap for practitioners using matrix completion in policy evaluation:
1. Prefer event-time aggregation for interpretability.
2. Implement pre-processing (residualizing fixed effects) to handle unit/time heterogeneity and ill-conditioning.
3. Use robust diagnostics (like in-time placebos) to guard against over-fitting.
4. Carefully select hyper-parameters using hold-out strategies that respect the missing data structure.
Broader Impact: The discussion underscores that while new estimators (like CY's) offer theoretical advances, their utility in social science depends on solving the "last mile" problems of inference, diagnostics, and data preprocessing. The application to RTC laws demonstrates that without these practical adjustments, sophisticated matrix completion methods can produce misleadingly large effect estimates.