Simulation-Based Comparison of ControlledInterrupted Time Series (CITS) and Multivariable Regression

Through a simulation-based comparison, this study demonstrates that Controlled Interrupted Time Series (CITS) designs outperform multivariable regression in estimating population-level policy effects by providing more accurate confidence interval coverage and robustness against serial correlation, particularly in settings with moderate to high autocorrelation.

The Gist

Imagine you are a detective trying to figure out if a new traffic law (like a speed limit change) actually reduced car accidents in your city. You can't run a perfect experiment where you flip a coin to decide which neighborhoods get the law and which don't—that's unethical and impossible. So, you have to look at the data you already have.

This paper is like a simulation lab where the researchers built thousands of "fake worlds" to test two different detective tools to see which one solves the mystery better.

Here is the breakdown of the two tools they compared:

The Two Detective Tools

1. The "Control Group" Tool (Controlled Interrupted Time Series - CITS):

   • How it works: You look at the city where the law changed (the "treatment" city) AND you look at a neighboring city that didn't change its laws (the "control" city).
   • The Analogy: Imagine you are testing a new fertilizer on a tomato plant. You don't just watch that one plant; you watch a second, identical plant next to it that doesn't get the fertilizer. If the first plant grows big and the second one stays small, you know the fertilizer worked. If both grow big, maybe it was just a really sunny week, not the fertilizer.
   • Why it's smart: It accounts for things that happen to everyone at the same time, like a sudden heatwave or a new national holiday that affects driving habits everywhere.

2. The "Math-Only" Tool (Multivariable Regression):

   • How it works: You look only at the city where the law changed. You try to use complex math to guess what would have happened if the law hadn't changed, based on past trends.
   • The Analogy: This is like looking at your single tomato plant and trying to guess how it would have grown without fertilizer, just by staring at its history. You have to assume that nothing else changed (no extra sun, no extra rain) to make your guess accurate.
   • The Risk: It's very easy to get fooled. If a sunny week happened right after you added the fertilizer, this tool might think the fertilizer did all the work, when it was actually the sun.

The "Serial Correlation" Monster

The paper mentions a tricky problem called serial correlation. In everyday language, think of this as "The Echo Effect."

If you drop a pebble in a pond, the ripples don't stop instantly; they keep going for a while. In data, if car accidents are high today, they are likely to be high tomorrow, not because of the law, but just because of the "ripple" from yesterday.

• The Math-Only Tool often gets scared by these ripples. It thinks the "echo" is a new signal, leading it to make confident but wrong guesses (like saying a law worked when it didn't).
• The Control Group Tool is better at ignoring the echoes because it sees that the echoes are happening in both cities, so it cancels them out.

What Did the Simulation Reveal?

The researchers ran their "fake worlds" with different lengths of data, different sizes of effects, and different levels of "echoes" (autocorrelation). Here is what they found:

• When the effect is huge: Both tools can usually find the answer.
• When the effect is small or the data is short: Both tools struggle a bit, but the Control Group tool is still more reliable.
• The Big Winner: The Control Group Tool (CITS) was the clear champion.
  • It was more accurate (less "noise" in the results).
  • It knew exactly how confident it should be.
  • It didn't get tricked by the "Echo Effect" (serial correlation).

The Math-Only Tool was dangerous. Even when the researchers tried to fix its math with special "patches" (Newey-West adjustments), it still got tricked by the echoes. It often told you, "We are 95% sure this worked!" when it was actually just guessing based on the ripples of yesterday.

The Bottom Line

If you want to know if a big public policy (like a new health law or traffic rule) actually worked, don't just look at the place where the rule changed.

You need a control group—a similar place where the rule didn't change. This acts as your "control plant" in the garden. Without it, you might blame the fertilizer for growth that was actually caused by the weather. The paper proves that using a control group gives you a much truer picture of reality, especially when data tends to "echo" from one day to the next.

Technical Summary

1. Problem Statement

While Randomized Controlled Trials (RCTs) are the gold standard for causal inference, they are often impractical for evaluating population-level policies. Consequently, Interrupted Time Series (ITS) designs are frequently employed as the most robust quasi-experimental alternative. However, standard ITS designs face two critical methodological challenges:

• Serial Correlation: Time-series data often exhibit autocorrelation, which, if unaccounted for, violates regression assumptions.
• Time-Varying Confounding: Unobserved factors that change over time and influence both the intervention and the outcome can introduce significant bias.

These issues can lead to biased inference and incorrect conclusions regarding policy effectiveness. While Controlled Interrupted Time Series (CITS) and Multivariable Regression (specifically multivariable negative binomial regression) are widely used to address these issues, their comparative performance in typical population health settings—particularly regarding statistical power and error calibration—has not been rigorously examined.

2. Methodology

The authors conducted a simulation-based study to directly compare the performance of CITS against multivariable regression for estimating policy effects in count time series data.

• Experimental Design: The study simulated data under various "data generating situations" to test robustness across different scenarios.
• Key Variables Manipulated:
  • Series length (short vs. long).
  • Intervention effect size (small, moderate, large).
  • Magnitude of lag-1 autocorrelation (weak to high).
• Performance Metrics: The study evaluated both methods based on:
  • Bias: Accuracy of point estimates.
  • Standard Error Calibration: Consistency between model-based and empirical standard errors.
  • Confidence Interval (CI) Coverage: Proportion of intervals containing the true parameter (targeting 95% nominal levels).
  • Mean Squared Error (MSE): A composite measure of variance and bias.
  • Statistical Power: Ability to detect true effects.
• Adjustments: The multivariable regression approach included Newey-West adjustments to attempt to correct for serial dependence.

3. Key Results

The simulation yielded distinct findings regarding point estimation versus inferential properties:

• Point Estimates (Bias):
  • Both methods produced unbiased estimates for moderate and large intervention effects.
  • Small effects in short series resulted in more pronounced bias for both methods, though the bias was generally manageable in longer series.
• Inferential Properties (Variance and Coverage):
  • CITS Performance:
    • Consistently demonstrated smaller Mean Squared Error (MSE).
    • Showed excellent consistency between model-based and empirical standard errors.
    • Maintained confidence interval coverage near the 95% nominal level across weak to moderate autocorrelation scenarios.
  • Multivariable Regression Performance:
    • Was highly sensitive to serial dependence.
    • Even with Newey-West adjustments, it frequently underestimated standard errors.
    • This led to undercoverage of confidence intervals (i.e., the true effect fell outside the interval more often than the 5% error rate would suggest), particularly under moderate to high autocorrelation.

4. Key Contributions

• Direct Comparative Evidence: The study provides rare, direct empirical evidence comparing CITS and multivariable regression specifically for count time series data in policy evaluation contexts.
• Validation of Control Series: It quantitatively demonstrates the superiority of including a concurrent control series (CITS) over relying solely on multivariable adjustments for time-varying confounders.
• Limitations of Standard Adjustments: The findings highlight that standard corrections for serial correlation (like Newey-West) in multivariable regression may be insufficient to prevent undercoverage in the presence of moderate-to-high autocorrelation.

5. Significance and Implications

The paper offers critical guidance for researchers and policymakers evaluating population health interventions:

• Structural Superiority: The study argues that structurally accounting for serial correlation (via the design of a control series in CITS) is more effective than attempting to statistically adjust for it post-hoc in multivariable models.
• Reliability of Inference: For robust policy evaluation, CITS is recommended as the preferred method because it ensures valid statistical inference (correct standard errors and confidence intervals), whereas multivariable regression risks false positives due to underestimated uncertainty.
• Best Practice: When RCTs are not feasible, researchers should prioritize the CITS design to minimize bias and ensure that confidence intervals accurately reflect the uncertainty of the estimated policy effects.