Causal Mediation Analysis for an Interrupted Time Series: Stabilized Mediator Weighting with an Application to a Vehicle Emissions Policy

Policymakers often introduce broad changes, like new laws or environmental regulations, at a specific date, hoping to see a clear shift in how things work. To measure the success of these changes, scientists often look at a single, long line of data points collected over time, such as daily air quality readings. This approach, known as an interrupted time series, is excellent at spotting the total change that happens after a policy is implemented. However, it often leaves a crucial question unanswered: did the policy work by directly changing the outcome, or did it work by first changing an intermediate factor that then influenced the outcome? For example, if a city bans a certain type of car, does the air get cleaner because there are fewer cars, or because the remaining cars are driving less? Untangling these two paths—the direct route and the indirect route through a middle step—is difficult when the data comes from just one place and one continuous stream of time, rather than from many separate groups of people.

A team of statisticians at the University of Manitoba has developed a new way to solve this puzzle. They created a method that can separate the total effect of a policy into its direct and indirect parts, even when working with a single, unbroken record of daily data. Their approach was tested first on computer simulations designed to mimic real-world daily patterns, where they found that older, simpler methods often produced misleading results when hidden factors were influencing the data. They then applied their new technique to a real-world event: the end of Ontario's "Drive Clean" vehicle emissions testing program in 2019. By analyzing daily air quality data from four regions in Toronto, they discovered that ending the testing program led to a measurable drop in ground-level ozone, but the path to that drop was more complex than a simple cause-and-effect chain.

The researchers focused on a specific policy change: the termination of mandatory vehicle emissions testing for light-duty vehicles in Ontario on April 1, 2019. Before this date, cars had to pass regular checks to ensure they were not polluting too much. After the program ended, the researchers wanted to know how this decision affected ground-level ozone, a harmful component of smog. They suspected the effect might travel through nitrogen dioxide, a gas emitted by vehicles that is closely linked to ozone formation. In many urban areas, the relationship between these two pollutants is counterintuitive; sometimes, a drop in nitrogen dioxide can actually lead to a rise in ozone, while in other cases, it leads to a drop. This complexity makes it hard to tell if a change in ozone is a direct result of the policy or a side effect of changes in nitrogen dioxide.

To figure this out, the team had to overcome a major statistical hurdle. Standard methods for analyzing such data often assume that the data points are independent, like flipping a coin many times. But daily air quality data is not independent; the air quality on Tuesday is heavily influenced by the weather and pollution levels from Monday. Furthermore, the policy change happened on a fixed date, which is a certainty, not a random event. The researchers realized that existing tools for separating direct and indirect effects were built for studies with many different people or groups, not for a single, continuous timeline. They adapted a technique called "stabilized mediator weighting." In plain terms, this method creates a virtual population where the link between the intermediate factor (nitrogen dioxide) and the final outcome (ozone) is not distorted by other changing weather conditions. By adjusting the data to account for these hidden influences, they could isolate the true effect of the policy.

They also had to account for a second major event that happened later: the onset of the pandemic in March 2020, which drastically changed traffic patterns and emissions. To ensure they were measuring the effect of the emissions testing program and not the pandemic, they treated the pandemic start as a second interruption in their timeline. They tested their new method extensively using computer simulations that generated thousands of days of fake data with known patterns. In these tests, they found that the old, unweighted methods failed badly when hidden factors were present, often missing the indirect effect entirely or getting the direction of the effect wrong. Their new method, which used the weighted approach, successfully recovered the true effects in the simulations, reducing errors significantly and providing a much more reliable picture of what was happening.

When they applied this refined method to the real data from Toronto, the results revealed a nuanced story. The analysis covered four distinct regions of the city: Downtown, East, West, and North. The researchers found that ending the emissions testing program was associated with a direct reduction in ground-level ozone. Across all four regions, the direct effect was a drop of about 2.1 parts per billion. This means that the policy change itself, or the conditions surrounding it, led to cleaner air directly, regardless of what happened to nitrogen dioxide. This finding held true even when they looked at the data before the pandemic started, suggesting the result was robust and not an artifact of the lockdowns.

However, the indirect path told a different story. The effect of the policy on ozone that traveled through nitrogen dioxide was not consistent across the city. In three of the four regions, the indirect effect was positive, meaning that changes in nitrogen dioxide actually pushed ozone levels up. But in the dense Downtown core, the indirect effect was negative, pulling ozone levels down. When the researchers combined these regional results, the positive and negative indirect effects largely canceled each other out, resulting in a total indirect effect that was close to zero and not statistically significant. This highlights a critical point: if the researchers had only looked at the total change without separating the direct and indirect paths, they might have missed the strong, consistent direct reduction in ozone and the complex, opposing forces at play in the different neighborhoods.

The study concludes that the new method provides a powerful tool for understanding how policies work in the real world. By separating the direct impact of an intervention from the indirect impact that flows through intermediate steps, scientists can get a clearer picture of cause and effect. In the case of Ontario's Drive Clean program, the evidence suggests that the termination of testing led to a direct improvement in ozone levels, a finding that aligns with what atmospheric scientists have observed about how ozone behaves in busy cities. The research also demonstrates that relying on simple, unadjusted methods can lead to incorrect conclusions, especially when dealing with complex, interconnected environmental data. The ability to distinguish between a direct policy effect and a chain of indirect reactions offers a more precise way to evaluate the true impact of public health and environmental regulations.