📊 statistics

State policy heterogeneity analyses: considerations and proposals

This paper critiques the limitations of standard heterogeneity analyses in state-level policy studies, arguing that Conditional Average Treatment Effects (CATE) often fail to capture policy-revariant state-specific impacts, and proposes a bounding approach within a difference-in-differences framework to more reliably estimate state-specific treatment effects, illustrated through an analysis of the Affordable Care Act's Medicaid expansion.

Original authors: Max Rubinstein, Megan S. Schuler, Elizabeth A. Stuart, Bradley D. Stein, Max Griswold, Elizabeth M. Stone, Beth Ann Griffin

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Rubinstein, Megan S. Schuler, Elizabeth A. Stuart, Bradley D. Stein, Max Griswold, Elizabeth M. Stone, Beth Ann Griffin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a policy analyst trying to figure out if a new law (like expanding health insurance) actually helps people. You look at data from 50 different states. Some states adopted the law, some didn't. You want to know: Did the law work? And did it work differently in different places?

This paper argues that the way researchers usually answer these questions is often misleading, like trying to guess the flavor of a soup by tasting a spoonful that has been mixed with five other soups. Instead, the authors propose a new, more honest way to look at the data: giving a "range of possibilities" for each state rather than a single, precise number.

Here is a breakdown of their argument using simple analogies:

1. The Problem: The "Smoothie" Effect

Usually, researchers try to find the "average effect" of a policy. They might say, "On average, this law increased prescriptions by 10%." To do this, they often group different versions of the law together.

The Analogy: Imagine you are testing a new recipe for a smoothie. In State A, they use a blender with a high-speed setting. In State B, they use a blender with a low-speed setting. In State C, they use a different brand of blender entirely.
The Mistake: If you just call them all "Smoothie Blenders" and mix the results together, you get a confusing average. Maybe the high-speed blender made a great smoothie, but the low-speed one made a mess. By lumping them together, you lose the specific details.
The Paper's Point: In real life, states don't just "adopt a policy." They adopt different versions of it, enforce it differently, and have different populations. When researchers group these distinct realities into one single "treatment" variable, they create a "smoothie" of data that hides the truth. The standard statistical methods they use (called CATEs) often tell you about associations (e.g., "States with X characteristics saw bigger numbers") rather than causes (e.g., "The law caused the change in this specific state").

2. The Goal: The "State-Specific" Truth

The authors argue that policymakers don't really care about a national average. They care about their specific state.

The Question: "Will this law work in my state?"
The Challenge: To answer this perfectly, you need to know what would have happened in your state if you hadn't passed the law (a "counterfactual"). But you can't go back in time to see that. You only see one reality.

3. The Solution: The "Safety Net" (Bounding)

Since we can't know the exact truth, the authors suggest we stop trying to guess a single number and instead calculate a range (a "bound") that is likely to contain the truth.

The Analogy: Imagine you are trying to guess the temperature outside, but your thermometer is broken.
- Old Way: You guess, "It's exactly 72 degrees," and give a tiny margin of error. If you're wrong, your answer is useless.
- New Way (Bounding): You look at the wind, the clouds, and the season. You say, "Based on what I know, it is definitely between 60 and 80 degrees."
- The "Sensitivity" Knob: The authors introduce a "knob" (a sensitivity parameter) that represents how much you trust your assumptions.
  - If you turn the knob to "very strict," your range is narrow (e.g., 70–74), but you have to be very sure your assumptions are perfect.
  - If you turn the knob to "very cautious," your range is wide (e.g., 50–90), but you can be almost 100% sure the real temperature is inside that box.
- The Benefit: This method admits uncertainty. Instead of pretending to know the exact answer, it says, "Here is the range of plausible outcomes given what we know."

4. How They Tested It

The authors ran computer simulations (like a video game of policy making) to see if their "bounding" method was better than the old "average" method.

The Result: The old method often gave the wrong sign (saying a policy helped when it actually hurt, or vice versa) because it was too sensitive to the "smoothie" mixing problem.
The Winner: The new "bounding" method was much better at correctly identifying the direction of the effect (positive or negative), even if it couldn't pinpoint the exact number. It was more reliable at saying, "We are confident this is a good thing," or "We are confident this is a bad thing," without getting the math wrong.

5. Real-World Example: Opioid Medication

To show how this works in real life, they looked at the Affordable Care Act's Medicaid expansion in 2014 and its effect on doctors prescribing buprenorphine (an opioid addiction medication).

The Old Way: When they looked at the average across all states, the result was "nothing significant happened." It was a wash.
The New Way: They applied their bounding method to each state individually.
- Illinois: The bounds suggested the policy likely decreased high-volume prescribing (strictly negative).
- New Mexico: The bounds suggested the policy likely increased prescribing (strictly positive).
- Other States: For many states, the range was too wide to be sure (it included zero).
The Takeaway: Instead of a boring "no effect" average, they found that the policy had very different, specific impacts on different states. This is the kind of information a governor actually needs to make decisions.

Summary

This paper is a call to stop pretending we can calculate a perfect, single number for how a policy affects every state. Because states are so different and policies are implemented so differently, that single number is often a lie.

Instead, the authors propose a method that says: "We can't tell you the exact number, but we can tell you the range of possibilities that makes sense." This approach is more honest, handles the messiness of real-world data better, and gives policymakers a clearer picture of what might happen in their specific backyard.

Technical Summary: State Policy Heterogeneity Analyses: Considerations and Proposals

Problem Statement
State-level policy studies frequently employ heterogeneity analyses to quantify how treatment effects vary across state characteristics, aiming to inform context-specific policy decisions. However, the authors argue that in settings characterized by varied contexts, multiple policy versions, and differential implementation, the causal quantities implicitly targeted by standard analyses often misalign with inferential goals.

Three primary issues complicate these analyses:

Misalignment of Estimands: Standard heterogeneity analyses typically estimate Conditional Average Treatment Effects (CATEs) or their linear projections. These quantities reveal associations between causal effects and state characteristics but do not necessarily answer policy-relevant questions such as "What is the causal effect of adopting this policy in this specific state?" or "Would this policy work better if combined with another?"
Treatment Coarsening: Analysts often collapse distinct policy versions or implementations (e.g., mandatory vs. voluntary Prescription Drug Monitoring Programs) into a single binary treatment indicator. This "coarsening" obscures the link between heterogeneity analyses and well-defined causal quantities, potentially manufacturing apparent heterogeneity even when underlying effects are constant.
Data Limitations: With a maximum of 50 states, statistical power is often insufficient to estimate complex, identified estimands (like CATEs or Controlled Direct Effects) with precision. This forces analysts toward likely misspecified parametric models and less interpretable target estimands.

Methodology
The paper proposes shifting the inferential focus from point identification of average effects to partial identification of State-Specific Treatment Effects (ITEs) using a bounding approach within a Difference-in-Differences (DiD) framework.

Causal Framework: The authors distinguish between the ITE (the effect for a specific state), the CATE (average effect conditional on covariates), and the Controlled Direct Effect (CDE). They demonstrate that treatment coarsening complicates the definition of these quantities, often rendering the CATE a weighted average of distinct causal effects with unclear policy implications.
Bounding Approach: Instead of assuming parallel trends hold exactly, the authors assume that the error between the counterfactual estimate and the true unobserved outcome is bounded.
- They define a sensitivity parameter, $\hat{\tau}^m_i$ , representing the maximum deviation from the parallel trends assumption.
- This parameter is informed by pre-treatment data (e.g., the magnitude of pre-treatment prediction errors or violations of parallel trends).
- The resulting bounds for the ITE are calculated as: $\hat{\psi}^{DiD}_{i,m} \pm \hat{\tau}^m_i$ .
Handling Coarsening and Covariates: The method extends to handle treatment coarsening (where the specific policy version adopted is unknown for untreated states) by either using conservative sensitivity parameters, assuming knowledge of the likely policy version, or taking the union of bounds across possible policy versions. Covariates are incorporated by adjusting the counterfactual trend estimates, though the sensitivity parameter is ultimately used to account for uncertainty in these adjustments.
Simulation and Application: The authors validate the method through simulations comparing ITE bounds against conventional CATE-based confidence intervals. They also apply the method to a real-world case study: the effect of the Affordable Care Act (ACA) Medicaid expansion on high-volume buprenorphine prescribing in 2014, examining heterogeneity based on mandatory PDMP laws and rural provider composition.

Key Contributions

Conceptual Clarification: The paper systematically organizes sources of effect heterogeneity (population composition, contextual factors, policy climate, policy grouping, and implementation) and clarifies why standard estimands (CATE, CDE, ITE) differ, particularly under treatment coarsening.
Alternative Inferential Goal: It proposes bounding state-specific effects as a viable alternative when point identification is infeasible or when the CATE is not policy-relevant. This approach yields ranges for each state's policy effect under explicit assumptions.
Practical Implementation: The authors develop a specific implementation within a DiD framework that utilizes pre-treatment data to inform sensitivity parameters, allowing researchers to assess the robustness of findings to deviations from ideal identifying conditions.
Empirical Illustration: The application to Medicaid expansion demonstrates that while conventional heterogeneity analyses often lack power or yield inconclusive results due to small sample sizes and model misspecification, the bounding approach can identify sign-conclusive state-specific effects (e.g., negative effects in Illinois, positive effects in New Mexico) even when average effects are null.

Results

Simulations: In a simulation setting with $N \in \{15, 25, 50\}$ states, the proposed ITE bounds achieved higher coverage of the true ITE and more reliably recovered the correct sign of the effect compared to conventional CATE-based confidence intervals. The CATE intervals often failed to capture the true state-specific effect because they target a projected average rather than the specific unit effect.
Application: In the analysis of Medicaid expansion and buprenorphine prescribing, conventional two-way fixed-effects models found no statistically significant average effects. In contrast, the bounding approach identified that the effect was strictly negative in Illinois and strictly positive in New Mexico across multiple sensitivity specifications and robustness checks. Other states yielded bounds containing zero, transparently communicating uncertainty.

Significance and Claims
The authors claim that in many state-policy settings, the CATE is a poor target of inference because it cannot reveal the true causes of heterogeneity and offers primarily associational interpretations. They argue that the more policy-relevant state-specific effects are rarely point-identified without implausible assumptions.

Consequently, the paper advocates for partial identification of state-specific effects as a principled and transparent alternative. The significance of this approach lies in its ability to:

Provide policy-relevant information (what happened or would happen in a specific state) even when data is insufficient for precise point estimation.
Explicitly quantify uncertainty through bounds that are directly interpretable as "how big the effect could plausibly be."
Avoid the pitfalls of misspecified parametric models and the misleading nature of coarsened estimands.

The authors conclude that researchers should not abandon heterogeneity analyses but should carefully choose their inferential goals. When point identification is not feasible, bounding state-specific effects offers a method that aligns causal inference more closely with the questions policymakers seek to answer, while maintaining transparency about the assumptions required to guarantee valid bounds.