📊 statistics

Sensitivity of weighted least squares estimators to omitted variables

This paper introduces a distribution-free sensitivity analysis framework for weighted least squares estimators that quantifies the impact of unobserved confounding on causal effect estimates using intuitive weighted partial $R^2$ statistics and provides formal bounds and adjusted inference procedures applicable to any non-negative weighting scheme.

Original authors: Leonard Wainstein, Chad Hazlett

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonard Wainstein, Chad Hazlett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of social science, researchers often try to answer a simple but stubborn question: does a specific event cause a specific change in people's lives? Imagine trying to understand if experiencing violence changes a person's desire for peace. In a perfect experiment, scientists would randomly assign some people to face violence and others to remain safe, then compare their attitudes. This random assignment ensures that the two groups are identical in every other way, so any difference in their views can be blamed solely on the violence. However, in the real world, we cannot force people into such situations. We can only observe what has already happened. This creates a problem: the people who experienced violence might have been different from those who did not before the event even began. Perhaps they lived in more dangerous villages, or perhaps their gender made them more likely to be targeted. These pre-existing differences are called confounders. If researchers do not account for them, they might mistakenly think the violence caused a change in attitude, when in reality, the change was just a reflection of who those people were to begin with.

To fix this, scientists use a technique called weighting. They look at the people they observed and assign them numerical importance, or weights, to make the group that experienced violence look statistically identical to the group that did not, regarding all the known factors like age, gender, and village size. Once the groups are balanced, they can compare their outcomes with much greater confidence. But a nagging doubt remains: what about the things we did not measure? What if there is a hidden factor, something the researchers never asked about, that influenced both who got hurt and how they feel about peace? If such a hidden factor exists, the entire conclusion could be wrong. For years, checking for this hidden influence has been difficult, especially when using these complex weighting methods. Researchers needed a way to ask, "How strong would a hidden factor have to be to completely change our conclusion?"

Leonard Wainstein and Chad Hazlett have developed a new set of tools to answer exactly that question. They created a method to test the sensitivity of weighted studies to unobserved confounding. Their approach is distinct because it does not try to guess how the hidden factor would change the weights themselves. Instead, it asks a simpler, more direct question: if we had included this hidden factor in our final calculation, how much would our result change? They found that the answer depends on just two intuitive numbers. The first is how much of the difference between the groups the hidden factor explains regarding who got treated. The second is how much of the difference in the outcome the hidden factor explains after accounting for everything else. These two numbers act as a dial. Researchers can turn the dial to see how strong a hidden factor would need to be to make their result disappear or flip signs.

The authors tested their tools on a real-world study concerning the Darfur conflict. In this study, researchers examined whether direct exposure to violence by government forces and militias changed the attitudes of refugees toward peace. The original analysis suggested that those who suffered direct harm were actually more likely to support peace. The researchers had already used weighting to balance the groups based on known factors like gender and village location. Wainstein and Hazlett applied their new sensitivity tools to this data. They asked: how strong would an unmeasured factor need to be to undo this finding? They used "gender" as a benchmark, a known factor that strongly influences both who gets hurt and how people feel about peace. They found that for the conclusion to change, an unmeasured factor would need to be as strong as gender in predicting who gets hurt, and just as strong in predicting the outcome. When they simulated a hidden factor with that level of influence, the result remained positive and statistically significant. The conclusion held up.

This robustness was not limited to one specific way of calculating the weights. The authors tested their method across three different approaches: one that used statistical models to estimate the probability of being treated, another that matched individuals one-to-one based on their similarities, and a third that mathematically forced the groups to have identical averages for certain traits. In every case, the finding that violence exposure increased support for peace proved resilient. Even when they imagined a hidden factor that was twice as strong as gender in predicting who got hurt, the result still pointed in the same direction, though the certainty of the finding became slightly weaker. The tools also revealed that in some matching scenarios, the hidden factor would need to be quite powerful to overturn the results, but in others, the margin for error was tighter.

The value of this work lies in its transparency and its flexibility. Previous methods for checking hidden bias often required researchers to make heavy assumptions about the shape of the data or the nature of the hidden factor. These new tools require no such assumptions. They work with any non-negative weights, whether they come from matching, statistical balancing, or other methods. The researchers also provided a software package that allows other scientists to easily apply these tests to their own work. By turning the abstract fear of "hidden bias" into a concrete calculation, they allow researchers to say with clarity: "Our conclusion is robust to any hidden factor that is not stronger than X." In the study of Darfur, this meant that the link between violence and a desire for peace was not a statistical fluke, but a finding that could withstand the scrutiny of a powerful, unseen influence. The work does not prove that no hidden factors exist, but it provides a rigorous way to measure how much those factors would need to matter to change the story.

Technical Summary: Sensitivity of Weighted Least Squares Estimators to Omitted Variables

Problem Statement
In observational studies, researchers frequently employ weighting schemes (e.g., inverse propensity score weighting, matching, covariate balancing) to reduce dependence on estimated linear models and balance observed covariates ( $X$ ) between treatment ( $D$ ) and control groups. Following weighting, estimates of causal effects are typically derived via a weighted linear regression of the outcome ( $Y$ ) on $D$ and $X$ . However, a primary concern remains the potential for unobserved confounding ( $Z$ ) to induce bias, regardless of the weighting technology used. Existing sensitivity analysis methods for weighted estimators often require researchers to model how the presence of $Z$ would alter the weights themselves (e.g., how the propensity score model would change). This approach is complex, often requiring specific distributional assumptions and limiting applicability to specific weighting methods.

Methodology
The authors propose a sensitivity analysis framework that decouples the assessment of unobserved confounding from the generation of weights. Instead of asking how $Z$ would change the weights, the method asks how the inclusion of $Z$ as a regressor in the weighted outcome model would alter the estimated treatment effect ( $\hat{\tau}_{wls}$ ), holding the original weights fixed.

The core of the methodology relies on the Omitted Variable Bias (OVB) framework adapted for weighted least squares. The bias of the weighted estimator is shown to be a function of two intuitive, bounded sensitivity parameters:

$\hat{R}^2_w(D \sim Z | X)$ : The proportion of weighted variance in the treatment explained by $Z$ given observed covariates.
$\hat{R}^2_w(Y \sim Z | D, X)$ : The proportion of weighted variance in the outcome explained by $Z$ given the treatment and observed covariates.

These parameters represent the strength of the relationship between the unobserved confounder and the treatment/outcome within the weighted empirical distribution.

To facilitate routine reporting and benchmarking, the authors extend the "robustness value" (RV) and "extreme scenario" statistics proposed by Cinelli and Hazlett (2020) to the weighted setting. They introduce a novel benchmarking procedure that compares the strength of a hypothetical $Z$ to that of an observed covariate $X^{(j)}$ . This involves:

Constructing "semi-weights" ( $w^{(-j)}$ ) by applying the weighting procedure while omitting $X^{(j)}$ .
Defining relative strength parameters ( $\kappa$ ) that quantify how much stronger $Z$ is than $X^{(j)}$ in predicting $D$ and $Y$ within the weighted and semi-weighted distributions.
Addressing the challenge that weighting often renders $D$ and $X$ uncorrelated (making standard benchmarking infeasible) by utilizing the semi-weighted distribution where $D$ and $X^{(j)}$ retain correlation.

For inference, the paper proposes three options for calculating adjusted standard errors and confidence intervals:

Pairs Bootstrap with Weight Re-estimation: Suitable for smooth weights (e.g., propensity scores) under a super-population framework.
Fixed Weights Bootstrap: A hybrid approach for fixed designs, resampling units while keeping weights fixed.
Closed-Form Implied Coefficient: A closed-form estimator akin to unweighted OVB analysis, requiring a "Uniform Shrinkage" assumption regarding how residuals change upon including $Z$ .

Key Contributions

General Applicability: The tools apply to any non-negative weighting scheme (propensity scores, balancing weights, matching, stratification) without requiring assumptions about the origin of the weights or how they would change if $Z$ were observed.
Distribution-Free Assumptions: The approach imposes no distributional assumptions on $Z$ or the data, nor does it require assumptions about the number of unobserved confounders.
Misspecification Handling: The framework can address both identification bias (unobserved confounding) and specification bias (misspecified functional forms) by treating $Z$ as a proxy for any omitted variable or function that corrects the model.
Software Implementation: The authors provide the sensewls R package to implement these methods.

Results and Application
The authors demonstrate the utility of their tools using data from Hazlett (2020a) regarding the effect of exposure to violence in Darfur on attitudes toward peace. They apply the sensitivity analysis to three distinct weighting strategies:

Inverse Propensity Score Weighting (ATE): The results show the estimate is robust to unobserved confounding as strong as the observed covariate "gender."
Propensity Score Matching (ATT): The authors show how the benchmarking procedure adapts when weights and semi-weights differ significantly (e.g., due to matching on different subsets of covariates). They find the estimate is sensitive to confounding twice as strong as gender/age regarding the treatment.
Exact Matching and Mean Balancing (ATT): The tools are applied to exact matching on village and gender, and maximum entropy balancing. In all cases, the authors successfully bound the sensitivity parameters and determine the robustness of the statistical significance of the treatment effect.

Significance and Claims
The paper claims that by focusing solely on the sensitivity of the weighted regression step rather than the weights themselves, researchers gain a powerful, generalizable tool for assessing robustness. The authors argue that their approach offers a "meaningful contribution" in interpretability, particularly through the use of $\hat{R}^2_w$ parameters which are bounded between 0 and 1, contrasting with unbounded odds-ratio-based measures like Rosenbaum bounds.

The authors are modest regarding limitations, noting that:

The benchmarking procedure for $\hat{R}^2_w(D \sim Z | X)$ can be challenging when weighted and semi-weighted distributions differ starkly, requiring investigators to entertain larger relative strengths for $Z$ .
Adjusted inference options for fixed designs (Options 2 and 3) have trade-offs regarding conservativeness and reliance on assumptions (e.g., uniform shrinkage of residuals).
The tools are currently designed for weighted least squares and weighted difference-in-means, though they note potential extensions to augmented estimators and synthetic control methods as future directions.

Ultimately, the paper positions these tools as a routine reporting standard that allows investigators to formally bound the strength of unobserved confounding relative to observed covariates across a wide variety of weighting-based causal inference strategies.

More like this