Degrees of Freedom and Information Criteria for the Synthetic Control Method

This paper provides an analytical characterization of the Synthetic Control Method's degrees of freedom to derive estimable information criteria that improve model selection over cross-validation, particularly in settings with noisy donors and numerous candidates such as the Tianjin car license rationing case.

The Gist

Imagine you are trying to predict how a specific car model (let's call it the "Highlander") will sell in a city called Tianjin after the government suddenly starts rationing license plates. You want to know: How many fewer cars would have been sold if the rationing hadn't happened?

To answer this, you need to build a "Ghost Car." This Ghost Car represents what the Highlander's sales would have looked like in a parallel universe where no rationing existed.

In the past, economists built this Ghost Car by finding one perfect twin city (like Shijiazhuang) that looked exactly like Tianjin in every way. But what if that twin city is noisy? What if its sales data is jittery and unreliable?

This is where the Synthetic Control Method (SCM) comes in. Instead of finding one perfect twin, SCM builds a "Frankenstein" Ghost Car by mixing together parts from many different cities. It takes a little bit of City A, a dash of City B, and a pinch of City C to create a perfect average that mimics Tianjin's pre-rationing sales.

However, the authors of this paper discovered a problem with this method when there are too many cities to choose from.

The Problem: The "Overfitting" Trap

Imagine you are trying to draw a line through a scatter of dots on a piece of paper.

• The Good Way: You draw a smooth line that captures the general trend.
• The Bad Way (Overfitting): You have so many dots that you can draw a squiggly, crazy line that hits every single dot perfectly.

The problem is, that crazy line is just memorizing the noise (the random jitters) rather than learning the real trend. If you use that crazy line to predict the future, you will be wrong.

In the world of Synthetic Controls, if you have 100 cities to choose from but only 10 years of data, the computer can find a weird combination of cities that fits the past data too perfectly. It's like cheating on a test by memorizing the answers to the practice questions but failing the real exam because you didn't learn the concepts.

The Solution: Degrees of Freedom (The "Flexibility Meter")

The authors wanted a way to measure exactly how much the method is "cheating" or flexing its muscles to fit the noise. They invented a new metric called Degrees of Freedom.

Think of Degrees of Freedom as a "Flexibility Score."

• If your model is simple (like a straight line), it has a low score. It's rigid and honest.
• If your model is complex (like that crazy squiggly line), it has a high score. It's flexible and suspicious.

The paper proves a surprising mathematical fact: For the standard Synthetic Control method, the Flexibility Score is roughly equal to the number of cities you actually used minus one.

• If you use 5 cities to build your Ghost Car, your Flexibility Score is 4.
• This gives researchers a clear warning light: "Hey, you are using too many cities for the amount of data you have. You are probably overfitting!"

The Tool: Information Criteria (The "Smart Judge")

Once you have a Flexibility Score, you need a way to pick the best model. Usually, researchers use a method called Cross-Validation.

The Old Way (Cross-Validation):
Imagine you are a teacher testing a student. You give them half the homework to study (training) and the other half to take a test (validation).

• The Flaw: In this specific economic problem, the "homework" period is very short. Splitting a short homework assignment in half leaves the student with almost nothing to study. It's like trying to learn a language by studying for 2 days and then taking a test on the remaining 2 days. The results are unreliable.

The New Way (Information Criteria):
The authors propose a new "Smart Judge" called an Information Criterion.

• Instead of splitting the data, this judge looks at the entire homework assignment.
• It calculates the score using a formula: How well did you fit the past? + (Penalty for being too flexible).
• If your model is too complex (too flexible), the judge adds a heavy penalty. If it's too simple, it gets a penalty for missing the trend.
• The goal is to find the "Goldilocks" model: not too simple, not too complex, just right.

The Real-World Test: Tianjin's Car Market

The authors tested this new "Smart Judge" on the Tianjin car market.

• The Situation: Tianjin introduced a lottery/auction for car licenses. This changed who could buy cars. Wealthier people could afford the auction, so they bought different cars than before.
• The Challenge: They had 76 different car models to analyze, but the sales data for each was noisy.
• The Result:
  • When they used the old "Split the Data" method (Cross-Validation), it picked a model that was too simple and missed the real impact.
  • When they used the new "Smart Judge" (Information Criteria), it found the perfect balance.
  • The Finding: The rationing didn't just lower sales; it changed the mix of cars. Mid-range and luxury cars (like the Toyota Highlander) actually saw their market share increase relative to cheap cars. The "rich" buyers who won the auctions preferred nicer cars.

Summary

This paper is like giving economists a new ruler and a new judge.

1. The Ruler (Degrees of Freedom): Tells you exactly how "flexible" your model is, so you know if it's cheating by memorizing noise.
2. The Judge (Information Criteria): Helps you pick the best model without needing to split your tiny dataset in half, which usually leads to bad decisions.

By using these tools, researchers can finally trust their "Ghost Car" predictions, even when they are working with messy data and a huge number of options. It turns a guessing game into a precise science.

Technical Summary

Here is a detailed technical summary of the paper "Degrees of Freedom and Information Criteria for the Synthetic Control Method" by Pouliot, Xie, and Liu.

1. Problem Statement

The Synthetic Control Method (SCM) has become a standard tool for causal inference in economics and political science, particularly for estimating treatment effects when a single treated unit is compared against a pool of untreated "donor" units. However, as applications move toward "high-dimensional" settings (where the number of donors $p$ is large relative to the number of pre-treatment periods $n$), the method faces significant challenges:

• Overfitting: The implicit model selection (choosing a sparse linear combination of donors) can lead to excellent in-sample fits that do not generalize, resulting in poor counterfactual predictions.
• Limitations of Cross-Validation (CV): Standard model selection techniques like cross-validation are often ill-suited for SCM.
  • Pre-intervention holdout splits short time series, leading to high variance and bias.
  • Leave-one-out on donors relies on strong assumptions about the similarity between donors and the treated unit.
  • CV is "data-hungry," requiring the splitting of already scarce pre-treatment data.
• Lack of Theoretical Metrics: There was no analytical characterization of the "model flexibility" (degrees of freedom) of SCM, making it difficult to construct formal information criteria (like AIC or BIC) for model selection.

2. Methodology

The authors develop a theoretical framework to quantify the flexibility of SCM and its penalized variants using Stein's Lemma and Lagrange Multiplier Theory.

A. Degrees of Freedom (DoF) Derivation

The core theoretical contribution is deriving the degrees of freedom ($df$) for various SCM formulations. The authors define $df$ as the expected number of estimated parameters, specifically measuring the covariance between observed and fitted values:
$$df(\hat{Y}) = \frac{1}{\sigma^2} \sum_{i=1}^n \text{Cov}(Y_i, \hat{Y}_i | X)$$
Using Stein's Lemma, they show that under Gaussian assumptions, this equals the trace of the divergence (Jacobian) of the fitted values with respect to the data.

Key results for DoF include:

1. Standard SCM (No Covariates): The DoF is one less than the expected number of donors with non-zero weights.
   $$df(\hat{Y}_{SCM}) = E[|A|] - 1$$
   where $A$ is the active set of donors. This implies that the implicit model selection in SCM does not incur an additional penalty beyond the cost of the selected coefficients themselves (unlike best subset selection).
2. Penalized SCM (PSCM): For the penalized variant (Abadie & L'Hour, 2021), the DoF is:
   $$df(\hat{Y}_{pen}) = (1 + \lambda)(E[|A|] - 1)$$
   where $\lambda$ is the tuning parameter.
3. SCM with Covariates: The DoF is reduced by the number of covariates ($n_{cov}$) because the covariates constrain the solution space:
   $$df(\hat{Y}) = E[|A|] - n_{cov} - 1$$
4. Ridge and Elastic Net Variants: Closed-form expressions are derived for constrained ridge and elastic net SCM, involving singular value decompositions of the donor matrix.

B. Information Criteria (IC) Construction

Using the derived DoF expressions, the authors construct Information Criteria to estimate out-of-sample risk (Mean Squared Error). The general form is:
$$\widehat{IC} = \|Y - \hat{Y}\|^2_2 + 2\hat{\sigma}^2 \widehat{df}(\hat{Y})$$

• Standard IC: Assumes homoskedasticity.
• Heteroskedasticity-Robust IC (ICHR): To address potential violations of the homoskedasticity assumption, they propose a robust estimator using the sample covariance penalty:
  $$\widehat{IC}_{HR} = n^{-1}\|Y - \hat{Y}\|^2_2 + \frac{2}{n-\hat{p}} \sum_{i=1}^n \hat{\epsilon}_i^2 \frac{\partial \hat{Y}_i}{\partial Y_i}$$
• HAR-IC: They further extend this to handle serial correlation (autocorrelation) using Heteroskedasticity-and-Autocorrelation-Robust (HAR) variance estimation techniques.

3. Key Contributions

1. Analytical Characterization of DoF: The paper provides the first closed-form expressions for the degrees of freedom of SCM, penalized SCM, and SCM with covariates. This resolves the question of whether SCM overfits, showing that in seminal low-dimensional applications, it generally does not, but in high-dimensional settings, it does.
2. Model Selection via Information Criteria: The authors propose using IC (specifically SURE-based criteria) as a superior alternative to cross-validation for selecting tuning parameters ($\lambda$) in penalized SCM and weighting matrices ($V$) in SCM with covariates.
3. Robustness Theory: They demonstrate that their DoF estimates and ICs are robust to departures from Gaussianity (via simulations) and provide heteroskedasticity-robust alternatives.
4. Empirical Application: A novel application of SCM to the Chinese automotive market, specifically analyzing the impact of car license rationing in Tianjin.

4. Results

Theoretical & Simulation Results

• Overfitting Analysis: Simulations confirm that while standard SCM fits well in low-dimensional settings, it overfits when $p \gg n$. The derived DoF accurately captures this flexibility.
• IC vs. Cross-Validation: In simulation studies (both Gaussian and non-Gaussian factor models), the Information Criteria (SURE) consistently outperforms various cross-validation methods (horizontal, vertical, and rolling window).
  • CV methods often select tuning parameters that are far from the "oracle" (optimal) value, leading to higher prediction errors.
  • IC selects parameters that minimize the true risk more accurately, particularly in short time-series settings where data splitting for CV is detrimental.
• Robustness: The heteroskedasticity-robust IC (ICHR) performs well even when error variances are non-constant, whereas the standard IC can be biased in such regimes.

Empirical Application: Tianjin Car Rationing

• Context: The authors study the effect of a lottery-auction hybrid license rationing policy in Tianjin (2013) on car sales.
• Challenge: While a "natural match" (the same car model in a non-rationed city, Shijiazhuang) exists, the data is noisy. The authors use SCM to average over many approximate matches to reduce variance.
• Findings:
  • Model Selection: Using IC to select the penalty parameter $\lambda$ yielded a different (and more conservative) model than cross-validation. The IC-selected model penalized "far-away" donors more heavily, avoiding coincidental in-sample fits.
  • Treatment Effects: The analysis revealed that mid-to-high-priced car models (e.g., Toyota Highlander, Magotan) saw a relative increase in market share post-rationing, while budget models saw larger declines. This aligns with the theory that rationing (via auctions) shifted demand toward wealthier buyers.
  • Comparison: The unpenalized SCM (or one selected by CV) underestimated the treatment effect magnitude compared to the IC-selected penalized model.

5. Significance

• Methodological Advancement: This paper fills a critical gap in the SCM literature by providing the necessary statistical machinery (DoF and IC) to treat SCM as a rigorous regression tool with standard model selection properties.
• Practical Utility: It offers applied researchers a reliable, non-data-hungry alternative to cross-validation for tuning penalized SCM models, which is crucial for studies with short pre-treatment periods and many potential donors.
• Policy Insight: The application demonstrates that SCM can be effectively used not just when a perfect match is missing, but also to "filter" noisy data by averaging over imperfect matches, provided the model selection is handled correctly to prevent overfitting.
• Generalizability: The theoretical framework (Stein's Lemma application to constrained optimization) is applicable to other constrained regression problems beyond SCM.

In summary, the paper establishes that Information Criteria based on analytically derived Degrees of Freedom provide a theoretically sound and empirically superior method for model selection in Synthetic Control Methods, particularly in high-dimensional and data-scarce environments.