📊 statistics

Selection on the Outcome: The Fraction of Nonresponse Bias That Survey Weighting Removes, with an Application to Subjective Well-Being

This paper demonstrates that survey weighting on socio-demographic variables only removes a fraction of nonresponse bias equal to the proportion of outcome variance those variables explain, revealing that such weighting is largely ineffective for subjective well-being research where demographic predictors account for minimal variance in happiness.

Original authors: Shay Tsaban

Published 2026-09-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shay Tsaban

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every time a researcher wants to understand how people feel about their lives, they usually ask a sample of people to fill out a survey. The goal is to take the answers from those few thousand respondents and use them to describe the feelings of millions. But there is a catch: not everyone answers the phone or fills out the form. Some people are too busy, some are too sick, and some simply do not want to talk to strangers. This is called nonresponse, and it creates a gap between the people who answered and the people who did not. If the people who stay home are different from those who answer in the very way the researcher is trying to measure, the final number will be wrong. For decades, the standard fix for this problem has been weighting. Researchers take the answers they have and mathematically adjust them to match the known demographics of the whole country, such as age, sex, education, and where people live. The idea is that if the sample looks like the population on paper, the answers must be representative too.

This approach works well when the reason people skip the survey is related to those demographic facts. If older people are less likely to answer, and the researcher knows how many older people are in the country, they can simply give the answers from the older respondents more weight to balance the scale. However, a new analysis suggests that this standard fix fails completely when the reason people skip the survey is the very thing being measured: their own happiness. If unhappy people are less likely to participate because they feel too down to talk, or if very happy people are too busy enjoying their lives to answer, no amount of demographic adjustment can fix the resulting error. The paper argues that in the field of subjective well-being, the standard correction removes almost none of the bias, leaving the final numbers skewed in a way that researchers have largely failed to notice.

The study, led by Shay Tsaban, investigates exactly how much of this hidden error remains after researchers apply their usual weighting techniques. The core discovery is that the effectiveness of the fix depends entirely on how much a person's happiness can be predicted by their background. If happiness were strongly tied to age or income, the standard weighting would work well. But decades of research in this field have established the opposite: a person's life circumstances explain very little of why they feel happy or sad. The paper calculates that the standard variables used for weighting—age, sex, race, education, and region—explain only about 1.7 percent of the differences in reported happiness among nearly 70,000 people in the General Social Survey. Because these variables explain so little, the weighting process can only remove about 1.7 percent of the bias caused by unhappy people dropping out. It leaves more than 98 percent of the error untouched.

This finding turns a common statistic on its head. For years, researchers have cited the low predictive power of demographics as proof that happiness is an internal state, not just a product of material circumstances. Tsaban's work shows that this same low number is also the repair rate of the field's standard correction. The very fact that circumstances do not determine feelings is what makes the standard survey fix so ineffective. The paper demonstrates this with a clear contrast. In a different study involving health data from Denmark, researchers had access to national registers that included medical records for both participants and non-participants. Because health records are strong predictors of who drops out of a study, the weighting variables in that case explained a large portion of the outcome. There, the weighting successfully removed most of the bias. The difference between the two cases was not the quality of the math, but the information available to do the math. In well-being research, the necessary information simply does not exist in the standard demographic data.

The consequences of this oversight are significant for how we understand the world. The paper examines six common types of conclusions drawn from well-being surveys, such as comparing happiness between different countries, tracking changes over time, or looking at differences between age groups. In almost every case, the bias runs in a predictable direction. Because unhappy people are more likely to drop out, the remaining group is slightly happier than the true population. This means that national averages are likely too high, and that the gap between happy and unhappy groups is smaller than it really is. For example, when researchers see that happiness rises in old age, the paper suggests this might be an illusion created by the fact that the least happy elderly people have already stopped participating. Similarly, when researchers look at inequality in happiness, the data likely understates how unequal things really are, because the most distressed individuals are missing from the bottom of the scale.

Despite the severity of the problem, the paper notes that the research community has paid almost no attention to it. An audit of over 1,600 recent articles in top well-being journals found that while 72 percent of them relied on survey data, only about 1.4 percent mentioned nonresponse, attrition, or the limitations of their sample. Researchers have spent decades refining the questions they ask and checking if different cultures interpret the words the same way, yet they have largely ignored whether the people answering the questions represent the whole population. The paper argues that this is a critical blind spot. The solution does not require a new theory of happiness, but rather a change in how researchers treat their data. Instead of assuming that weighting has fixed the problem, researchers should report how much of the bias remains and use methods from health research to bound the possible error. This involves acknowledging that the final number is an estimate with a known, significant margin of error, rather than a precise measurement of the population's mood.

The study concludes that the field must stop treating the weighted average as a clean solution. The standard demographic adjustments are not a magic wand that makes the missing people appear; they are a partial fix that works only when the missing people are missing for demographic reasons. When they are missing because of their feelings, the fix fails. The path forward involves accepting that the current data is incomplete and using new tools to measure the size of the gap. By borrowing techniques from medical research, where missing data is treated as a serious threat to validity, well-being researchers can finally begin to see the full picture of human happiness, including the parts that are currently hidden in the silence of those who did not answer.

Technical Summary: Selection on the Outcome and the Limits of Survey Weighting

Problem Statement
Unit nonresponse is a primary threat to inference in social surveys, with response rates declining for decades. The standard remedy is weighting on socio-demographic auxiliaries (e.g., age, sex, education) to restore representativeness. This procedure relies on the assumption that missingness depends on observed covariates. However, in many fields, participation depends directly on the survey outcome itself (selection on the outcome), a mechanism known as "missing not at random" (MNAR). While it is established that weighting cannot fully resolve MNAR, the literature lacks a quantitative magnitude for how much bias remains. This paper addresses the specific case where the probability of participation depends on the outcome variable ( $Y$ ) directly, in addition to depending on auxiliary variables ( $X$ ).

Methodology and Derivation
The paper derives a closed-form expression for the fraction of nonresponse bias that survives standard weighting under a specific response mechanism.

Setup: Let $Y$ be the outcome of interest with mean $\mu$ and variance $\sigma_Y^2$ . Let $R$ be a response indicator with propensity $\rho$ . The nonresponse bias in the respondent mean is defined as $B = E[Y|R=1] - \mu = \text{Cov}(\rho, Y) / \bar{\rho}$ .
Response Mechanism: The paper posits a linear probability model for response propensity: $\rho = \alpha + \beta Y + \gamma'X$ , where $\beta > 0$ represents the direct dependence of participation on well-being, and $\gamma'X$ represents dependence on auxiliaries.
Weighting Effect: Weighting on $X$ (via post-stratification or propensity adjustment) removes the component of the covariance between propensity and outcome that operates between cells of $X$ , but retains the component operating within cells.
The Result: Conditioning on $X$ eliminates the $\gamma'X$ term. The residual bias is driven solely by the direct dependence on $Y$ . The paper derives that the share of the outcome-dependent bias remaining after weighting is:
$\frac{B_W}{B_{direct}} = 1 - R^2_{Y \cdot X}$
where $R^2_{Y \cdot X}$ is the coefficient of determination from a regression of the outcome $Y$ on the weighting auxiliaries $X$ .

Key Contributions

Quantification of Residual Bias: The paper provides a specific, estimable magnitude for the failure of standard weighting under outcome-dependent selection. It demonstrates that the repair rate is governed by a single quantity: the proportion of variance in the outcome explained by the weighting variables.
Generalization: While derived in a linear benchmark, the result is generalized (Proposition 2) to non-linear response mechanisms, showing that the residual share is governed by the ratio of within-cell covariance to unconditional covariance.
Identification of the Governing Parameter: The paper identifies that the field of subjective well-being already estimates this governing parameter ( $R^2_{Y \cdot X}$ ) for substantive reasons (to determine how much life circumstances predict feelings), but has not connected it to the efficacy of survey corrections.
Empirical Estimation: Using 69,719 respondents from the General Social Survey (1972–2024), the paper estimates that standard weighting variables explain only 1.7% of the variance in reported happiness. Consequently, weighting removes less than 2% of any outcome-dependent bias, leaving over 98% unaddressed.
Contrasting Case: The paper validates the mechanism by examining a Danish health survey where national registers provided auxiliaries (including healthcare utilization) that strongly predicted the outcome (mental disorder). In that case, where $R^2$ was high, weighting successfully recovered most of the selection bias, confirming that the explanatory power of the auxiliaries for the outcome is the critical factor, not just their power to predict response.
Audit of Research Practice: A systematic audit of 1,618 articles in five well-being journals (2023–2026) found that while 72% rely on survey samples, only 1.1% mention nonresponse, attrition, or coverage in their abstracts. This highlights a gap between the severity of the problem and its acknowledgment in the literature.

Results and Implications for Inference
The paper argues that the severity of the problem varies by field, depending on how strongly the outcome is demographically patterned.

Subjective Well-Being: Sits at the "unfavourable extreme" because socio-demographics explain very little variance in happiness. Standard weighting is largely ineffective against selection on well-being.
Consequences for Standard Inferences: The paper outlines how this residual bias affects specific inferences:
- Cross-national comparisons: Differing response rates across countries lead to differing residual biases, meaning scalar invariance alone does not guarantee comparable means.
- National estimates: Anomalously low national means may be partly artefactual due to outcome-dependent nonresponse.
- Trends and Group Differences: Changes in respondent composition (e.g., differential participation by well-being) can mimic changes in well-being.
- Age Profiles: Attrition and mortality select for survivors with higher well-being, potentially overestimating late-life happiness.
- Adaptation: Non-recoverers from adverse events are more likely to drop out, leading to overestimates of adaptation.
- Inequality: Selection thinning the lower tail of the distribution leads to underestimates of well-being inequality.

Significance and Claims
The paper does not claim that well-being estimates should be discarded, nor does it argue that weighting is useless for selection on observables. Instead, it claims that:

Weighting does not solve outcome-dependent selection: A weighted mean should not be interpreted as having corrected for selection on the outcome.
The repair rate is low and predictable: The fraction of bias removed is exactly the $R^2$ of the outcome on the auxiliaries. For well-being, this is negligible (~1.7%).
The remedy lies in auxiliary measurement, not better weighting models: Improving the propensity model for participation does not help. The only way to increase the repair rate is to use auxiliaries that predict the outcome (e.g., administrative health records or brief well-being screens at recruitment), as demonstrated in the Danish case.
Reporting standards must change: Researchers should report the $R^2$ of their weighting variables on the outcome, provide bounds for potential bias (using sensitivity analysis or worst-case bounding), and report attrition patterns by baseline well-being.

The paper concludes that the field has rigorously audited its measurement scales but has largely ignored its sampling frames. The same statistic used to argue that circumstances weakly determine feelings also quantifies the inability of standard corrections to fix selection bias in those same feelings.

Technical Summary: Selection on the Outcome and the Limits of Survey Weighting

More like this