📊 statistics

An average-case sensitivity analysis for unmeasured confounding

This paper proposes a new average-case sensitivity model for unmeasured confounding, parameterized by the second moment of the propensity score ratio, which yields sharp closed-form bounds and efficient estimators for average potential outcomes that offer tighter results and better calibration than traditional worst-case approaches.

Original authors: Yao Zhang, Qingyuan Zhao

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yao Zhang, Qingyuan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a specific clue (like eating fish) caused a specific result (like high mercury levels in the blood). You can't run a perfect experiment where you force some people to eat fish and others not to, so you have to look at past records. This is called an observational study.

The big problem with these studies is unmeasured confounding. This is like a hidden variable you didn't record. Maybe the people who ate fish were also more likely to live near the ocean, and the ocean air affects mercury levels. If you don't measure "living near the ocean," you might wrongly blame the fish.

To be honest about this risk, statisticians do something called sensitivity analysis. They ask: "How strong would this hidden variable have to be to completely change our conclusion?"

The Old Way: The "Worst-Case" Scenario

For a long time, researchers used a method called the Marginal Sensitivity Model. Think of this as a detective who assumes the absolute worst possible scenario.

Imagine you are checking a building for fire safety. The old method says: "We must assume that if there is a fire, it will happen in the single most dangerous room possible, and it will be the biggest fire imaginable."

In the paper, the authors call this the Worst-Case Model. It sets a limit (called $\Gamma$ ) on how much the hidden variable can skew the results. The problem is that this limit is based on the single most extreme, rare outlier in the data.

The Flaw: If one person in a city of a million has a very strange habit, the old model assumes everyone has that habit. It's overly pessimistic. It makes the study look very fragile, even if the hidden variable is only strong for a few people.

The New Way: The "Average-Case" Scenario

The authors (Yao Zhang and Qingyuan Zhao) propose a new method called the Average-Case Sensitivity Model.

Instead of asking, "What is the worst single person?", they ask, "What is the average strength of the hidden variable across the whole group?"

The Analogy:
Imagine you are testing the strength of a bridge.

The Old Way: You assume a giant, mythical monster (a "worst-case" scenario) jumps on the bridge. If the bridge breaks under the monster, you say the bridge is unsafe, even if the monster never actually exists.
The New Way: You measure the average weight of all the cars, trucks, and pedestrians that actually use the bridge. You ask, "If the average weight of traffic increases by 20%, does the bridge hold?"

This new method uses a parameter called $\Sigma$ (Sigma). Instead of bounding the maximum difference between groups, it bounds the average squared difference. It acknowledges that while some people might be heavily influenced by the hidden variable, most might not be.

Why This Matters: A Real-World Example

The authors tested this with a real study about fish consumption and blood mercury levels.

Using the Old (Worst-Case) Method: They found that if an unmeasured factor (like a hidden lifestyle trait) was as strong as "education level" or "income," the study's conclusion (that fish raises mercury) would flip. The "safety margin" was very small.
Using the New (Average-Case) Method: They asked, "How strong would the hidden factor need to be, on average, to flip the result?"
- They found that even if the hidden factor was as strong as education or income, the average effect wasn't strong enough to break the conclusion.
- Result: The new method gave a tighter, more optimistic bound. It showed that the conclusion "fish raises mercury" is actually quite robust, because it's unlikely that a hidden factor is strong enough on average to ruin the study, even if it's strong for a few people.

How They Did It (The Magic Math)

To make this work, the authors had to solve a complex math puzzle.

They treated the problem like a portfolio optimization problem (like managing a stock portfolio to balance risk and return).
They derived a "one-step estimator." Think of this as a super-efficient calculator that doesn't just guess the answer but uses a specific formula (called an Efficient Influence Function) to get the most accurate answer possible with the data they have.
They also used a technique called Multiplier Bootstrap to draw a "confidence band." Imagine drawing a safety net around their results. This net covers all possible scenarios at once, ensuring they aren't just lucky with one specific number.

The Bottom Line

The paper argues that the old way of doing sensitivity analysis is too scared. It assumes the worst single outlier defines the whole group. The new Average-Case method is more realistic. It looks at the group as a whole, leading to:

Tighter bounds: We can be more confident in our results.
Better calibration: We can compare the hidden risk to things we can measure (like income or education) and see that the hidden risk is usually not as scary as the old method suggests.

In short, this new method helps researchers say, "We know there are hidden factors, but on average, they aren't strong enough to make us change our minds."

Technical Summary: An Average-Case Sensitivity Analysis for Unmeasured Confounding

Problem Statement
In observational studies, causal inference relies on the unconfoundedness (ignorability) assumption, which posits that treatment assignment is independent of potential outcomes conditional on observed covariates. Since this assumption is untestable, sensitivity analysis is required to assess how conclusions might change under violations of unconfoundedness. Existing approaches, particularly the marginal sensitivity model (Tan, 2006; Zhao et al., 2019), typically employ a "worst-case" parameter ( $\Gamma$ ) that bounds the $L_\infty$ -norm of the logit difference between the observed and full-data propensity scores. While mathematically tractable, this worst-case approach can be overly pessimistic, as the bound is determined by the most extreme (and potentially rare) values of the covariates, failing to capture the average strength of unmeasured confounding.

Methodology
The authors propose a new average-case sensitivity model parameterized by the second moment of the propensity score ratio, $\Sigma$ . Instead of bounding the maximum deviation, this model constrains the expected squared ratio of the propensity scores:
$E \{ \nu_{1,h}(X) \} \le \Sigma,$
where $\nu_{1,h}(X) = E\{h^2(X, Y) \mid X, Z=1\}$ and $h(X, Y) = e(X)/e(X, Y)$ is the propensity score ratio. This formulation requires only that the average strength of unmeasured confounding is bounded.

The paper addresses the associated sensitivity analysis as an optimization problem to derive sharp bounds on the average potential outcomes, $E\{Y(1)\}$ and $E\{Y(0)\}$ .

Population Solutions: The authors analytically solve the optimization problem using Lagrangian formulations.
- They reformulate the problem to minimize a trade-off between the "risk" (second moment) and "return" (first moment) of the propensity score ratio, drawing an analogy to the Markowitz portfolio problem.
- They derive closed-form solutions for the optimal propensity score ratio function $h^*(X, Y)$ . Unlike the worst-case model, where the optimal solution is a piecewise constant function (step function), the average-case solution is a piecewise linear function of the outcome $Y$ .
- They establish that these solutions yield sharp bounds, meaning any value within the derived interval is attainable by a valid data-generating distribution satisfying the constraints.
Estimation and Inference:
- The authors derive the Efficient Influence Functions (EIFs) for the bounds and sensitivity values under both the average-case and worst-case models.
- They propose one-step estimators based on these EIFs, utilizing cross-fitting to handle nuisance parameters (such as the propensity score and conditional outcome density). This approach allows for $\sqrt{n}$ -consistency and asymptotic normality even when nuisance parameters are estimated at slower rates (e.g., $o_P(n^{-1/4})$ ).
- To facilitate inference over a range of sensitivity parameters, they apply the multiplier bootstrap to construct simultaneous confidence bands for the sensitivity curves.

Key Results

Tighter Bounds: Through a numerical example involving a "gated confounding" structure, the authors demonstrate that the average-case model yields significantly tighter bounds on the Average Treatment Effect (ATE) compared to the worst-case model. The worst-case parameter $\Gamma$ is shown to be driven by extreme outliers, whereas the average-case parameter $\Sigma$ better reflects the confounding strength for the majority of the population.
Efficiency: Simulation studies confirm that the proposed one-step estimators based on EIFs converge faster and have lower root-mean-squared error (RMSE) compared to direct plug-in estimators, particularly when nuisance parameters are estimated via nonparametric methods.
Real-Data Application: In a study estimating the effect of fish consumption on blood mercury levels (NHANES 2013-2014), the authors calibrate the sensitivity parameters against observed covariates (education, income, age). The average-case model suggests the conclusion (that fish consumption increases mercury) is robust to unmeasured confounding of a magnitude comparable to these observed covariates, whereas the worst-case model suggests the conclusion is fragile. The average-case bounds are notably more optimistic and tighter.

Significance and Claims
The paper claims to provide a more realistic and less pessimistic framework for sensitivity analysis by shifting from worst-case to average-case constraints. Key contributions include:

Analytical Tractability: Unlike previous generalizations of marginal sensitivity models that relax constraints to make optimization tractable (resulting in non-sharp bounds), this paper provides closed-form, sharp solutions for the average-case problem.
Unified View: The derived solutions offer a unifying perspective on worst-case and average-case models, highlighting the structural difference between piecewise constant (worst-case) and piecewise linear (average-case) optimal weights.
Practical Utility: By enabling the construction of simultaneous confidence bands and providing a method to calibrate sensitivity parameters using observed covariates, the approach facilitates more nuanced and optimistic interpretations of observational study results without sacrificing mathematical rigor.

The authors maintain that while the average-case model is more optimistic, it remains a valid sensitivity analysis tool that captures the "average" extent of unmeasured confounding, which is often more relevant for policy and scientific inference than the theoretical worst-case scenario.

The Old Way: The "Worst-Case" Scenario

The New Way: The "Average-Case" Scenario

Why This Matters: A Real-World Example

How They Did It (The Magic Math)

The Bottom Line

More like this