Evaluating the effects of policy interventions subject to early adoption: A case study of prescription drug monitoring programs and opioid dispensing
This paper proposes a two-stage synthetic control method that accounts for early adoption in staggered policy implementations to correct bias in effect estimates, and applies it to find that prescription drug monitoring programs likely reduced opioid dispensing, though the results were not statistically significant.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the United States, the opioid crisis has evolved through distinct waves, with the first driven by a rapid surge in prescriptions for powerful painkillers like hydrocodone and oxycodone. To combat rising misuse and overdose deaths, many states introduced Prescription Drug Monitoring Programs, or PDMPs. These are state-run databases that track who is prescribing and receiving controlled substances, allowing doctors and pharmacists to spot dangerous patterns before they lead to tragedy. The goal is simple: if a doctor can see a patient's full history of opioid use, they can make safer choices, perhaps avoiding a new prescription or tapering an existing one. However, measuring whether these programs actually work is notoriously difficult. Policies are rarely turned on like a light switch; they often roll out in phases, starting with voluntary access before becoming a legal requirement. This staggered timeline creates a fog for researchers, making it hard to tell if a drop in prescriptions happened because doctors were already using the new tools on their own, or because the law finally forced them to.
A team of researchers at Harvard T.H. Chan School of Public Health set out to untangle this specific problem. They focused on a phenomenon they call "early adoption," which occurs when a policy's infrastructure becomes available and people start using it voluntarily before a mandate takes effect. In the context of PDMPs, this means doctors began checking the database years before they were legally required to do so. Standard statistical tools used to evaluate such policies often fail in these situations because they assume that nothing changes until the law officially kicks in. When that assumption is broken, the tools produce misleading results, often hiding the true impact of the policy or inventing effects that aren't there. The researchers developed a new method to separate the influence of this voluntary early use from the later, mandatory phase, allowing them to see the distinct impact of each stage.
To test their new approach, the team first ran extensive computer simulations. They created thousands of fake scenarios where they knew the exact truth: how much the policy changed outcomes due to early adoption versus the mandate. They then compared their new method against the standard tools used by scientists today. The results were clear. When early adoption was present but ignored, the standard tools produced heavily biased estimates, often missing the mark by a wide margin. In contrast, the new two-stage method successfully isolated the early adoption effect from the mandate effect, recovering the true numbers with high accuracy. This simulation work proved that their technique could handle the messy reality of staggered policy rollouts, where different groups adopt new systems at different times and for different reasons.
With the method validated, the researchers applied it to real-world data from the United States, looking at opioid dispensing patterns across 49 states and the District of Columbia between 2000 and 2016. They tracked the quantity of hydrocodone and oxycodone distributed, measured in morphine milligram equivalents per person. The data showed a familiar story: dispensing rose steadily through the early 2000s, peaked around 2010 to 2012, and then began a slow decline. By using their new method, the team could look closely at what happened during the long periods when PDMPs were available but not yet mandatory. They found that during this "early adoption" phase, there was a modest, though statistically uncertain, reduction in opioid dispensing. This suggests that simply making the database available and letting doctors use it voluntarily did begin to change behavior, even before the law required it.
Once the mandate took effect, forcing doctors to check the database, the researchers observed a further decline in dispensing. The total effect of the policy, combining both the voluntary early use and the later legal requirement, pointed toward a reduction of about 23.6 morphine milligram equivalents per person per quarter compared to what would have happened without the policy. However, the researchers were careful to note the limits of their findings. While the numbers suggested a downward trend, the statistical uncertainty was large. The confidence intervals for both the early adoption effect and the mandate effect included zero, meaning the data did not provide strong enough evidence to say with certainty that the reductions were not just due to random chance. The study suggests that PDMPs likely helped reduce opioid dispensing, but it cannot definitively prove the size of that effect or separate the two phases with absolute precision.
The core contribution of this work is not a definitive answer to whether PDMPs stopped the opioid crisis, but rather a better way to ask the question. By acknowledging that policies often have a "pre-game" period where people start using them early, the researchers provided a clearer lens for viewing public health interventions. Their method allows policymakers to understand that the impact of a program might begin long before the first legal penalty is enforced. While the specific numbers for the opioid study remain imprecise, the framework offers a more honest way to evaluate complex, real-world policies where the line between "voluntary" and "mandatory" is often blurred. It reminds us that in the messy reality of public health, the moment a tool becomes available is often just as important as the moment it becomes required.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.