📊 statistics

Correcting for Nonignorable Nonresponse Bias in Ordinal Observational Survey Data

This paper introduces a maximum likelihood estimator that extends the variable-response-propensity framework to ordinal outcomes, using response-propensity proxies to correct for nonignorable nonresponse bias in political surveys while integrating observable covariates and population weights.

Original authors: Lukáš Lafférs, Jozef Michal Mintal, Ivan Sutóris

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lukáš Lafférs, Jozef Michal Mintal, Ivan Sutóris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Survey Mystery: When the People Who Don't Show Up Are the Most Important

Imagine you are trying to guess the average height of everyone in your city. You stand in the town square and ask people walking by, "How tall are you?" Most of the people who stop to answer are average height. But the very tall people are too busy stretching to reach the top shelf, and the very short people are hiding under umbrellas. If you only count the people who stopped to talk, your guess will be wrong. This is the problem of nonresponse bias in surveys. In the world of political science and social research, scientists rely on surveys to understand what people think about everything from the economy to their happiness. But as more people ignore survey calls or skip online forms, researchers worry that the people who do answer are different from the people who don't.

Usually, scientists try to fix this by "weighting" the answers. If there are fewer young people in the survey than in the real world, they just count the young people's answers more heavily to balance the scales. This works great if the only difference between the talkers and the silent crowd is their age or gender. But what if the silence itself is the clue? What if the people who refuse to answer are silent specifically because they are unhappy, angry, or dissatisfied? This is called nonignorable nonresponse. It's like the silent people in the town square aren't just busy; they are hiding because they are embarrassed about their height. If you don't account for why they are hiding, your math will be off, no matter how many young people you try to count. This is the tricky puzzle that the paper by Lafférs, Mintal, and Sutóris sets out to solve.

The Paper: Catching the Silent Crowd with a "Cooperation Meter"

The authors of this paper, published in 2026, have built a new mathematical tool to fix surveys where the people who don't answer are different from those who do, specifically when the answers are on a scale (like "Very Happy" to "Very Unhappy") rather than just "Yes" or "No."

Think of a standard survey as a game of "Guess the Score." Usually, if someone doesn't show up to the game, the referee just ignores them. But this paper suggests that the referee can look at a "Cooperation Meter" to guess what the missing players would have scored. In their method, they use a question asked after the interview, like "How much did you enjoy this chat?" or "How cooperative were you?" This is the response-propensity proxy. It's a clue left behind by the people who did talk.

The authors realized that if people who hated the interview (low cooperation) also tended to give low scores on life satisfaction, then the people who didn't answer the survey at all (zero cooperation) probably gave even lower scores. They created a formula that connects these dots. It's like looking at a line of people waiting for a ride. If the people at the front of the line (who loved the ride) are smiling, and the people at the back (who barely tolerated it) are frowning, you can guess that the people who never showed up to the line at all were probably frowning even harder.

What They Found: Not All Surveys Need a Rescue

The team tested their new tool using data from the 2024 American National Election Studies (ANES), a massive survey of about 3,000 people. They simulated a scenario where about 65% of the people didn't answer the survey (a very realistic number for modern surveys). They then used their "Cooperation Meter" method to guess what the silent 65% would have said.

The results were a mix of "Wow, that changed everything" and "Hmm, not so much."

The Big Shift (Life Satisfaction): When they looked at how satisfied people were with their lives, the correction made a huge difference. The unadjusted survey suggested most people were "Very" or "Extremely" satisfied. But after the authors accounted for the silent, unhappy crowd, the picture changed. The corrected data showed that a much larger chunk of the population was actually "Slightly satisfied" or "Not satisfied at all." The math showed a strong link (a correlation of about 0.47) between hating the interview and being unhappy with life. It turns out, the people who were too grumpy to talk were the ones dragging down the average happiness score.
The Small Shift (The Economy): When they looked at whether people thought the national economy was getting better or worse, the correction barely moved the needle. The link between hating the interview and thinking the economy was bad was very weak (a correlation of only 0.14). Whether they included the silent crowd or not, the survey results stayed roughly the same.
The "No Change" Zone: For questions about the death penalty, abortion, or trusting the media, the correction did almost nothing. The silent crowd didn't seem to have a different opinion than the chatty crowd on these topics.

The Takeaway: One Size Does Not Fit All

The main lesson from this paper is that fixing survey bias isn't a magic wand that works for every question. The authors show that you have to check each topic individually. For some things, like how happy people are with their lives, the people who refuse to talk are hiding a very different reality, and ignoring them gives a false picture. For other things, like opinions on the economy or the death penalty, the people who talk and the people who don't seem to think pretty much the same thing.

The authors didn't just guess this; they ran computer simulations to prove their math works better than old methods, and then they applied it to real data. They found that while their new method is powerful, it's most useful when the "Cooperation Meter" (like how much someone liked the interview) actually varies a lot and clearly signals who is willing to talk. If the meter doesn't move, the method can't guess what the silent people are thinking.

In short, this paper gives researchers a new, practical way to peek behind the curtain of survey silence. It tells us that sometimes, the people who say "No comment" are the most important voices of all, but only if we know how to listen to the clues they left behind.

Technical Summary: Correcting for Nonignorable Nonresponse Bias in Ordinal Observational Survey Data

Problem Formulation
Survey research frequently relies on weighting adjustments (e.g., post-stratification, raking) to align respondents with population demographics. However, these methods fail when nonresponse is "nonignorable"—that is, when the likelihood of responding depends systematically on the unobserved outcome itself. In such cases, standard adjustments introduce bias even into basic descriptive statistics. While the Variable-Response-Propensity (VRP) framework proposed by Peress (2010) addresses this for binary outcomes by leveraging response-propensity proxies (e.g., interviewer-coded cooperativeness) to extrapolate toward nonrespondents, it does not accommodate ordinal outcomes. Many key political science measures, such as trust, satisfaction, and economic evaluations, utilize ordinal scales. Collapsing these to binary categories discards information and risks altering substantive conclusions, particularly if selection mechanisms vary across the response scale.

Methodology
The authors generalize the VRP framework to handle ordinal outcomes ( $y_n \in \{1, \dots, Y\}$ ) using a correlated latent-variable ordered probit model. The method integrates three core components:

Outcome Model: An ordered probit model where the latent outcome $y^*_n$ is a function of covariates $x_n$ and an error term $\epsilon_n$ .
Response Model: An ordered probit model for a response-propensity proxy $r_n$ (observed only for respondents), where the latent response $r^*_n$ is a function of covariates $z_n$ and an error term $\eta_n$ . The proxy is ordered such that lower values indicate higher propensity to respond. Unit nonresponse is modeled as an additional category ( $R+1$ ) in this equation.
Selection Mechanism: The model assumes a joint normal distribution for the error terms $(\epsilon_n, \eta_n)$ with unit variances and a correlation parameter $\rho$ . This parameter $\rho$ captures the nonignorable selection on the unobserved outcome after conditioning on covariates.

The log-likelihood function is constructed to account for observed respondents (where both $y_n$ and $r_n$ are known) and the mass of nonrespondents (determined by the known nonresponse rate $N_{miss}$ and strata population shares). The parameters $(\alpha, \beta, \lambda, \theta, \rho)$ are estimated via maximum likelihood. Standard errors are derived using the delta method with a numerical Jacobian.

Identification
Identification of the selection correlation $\rho$ relies on systematic variation in the distribution of the outcome $y_n$ across categories of the proxy $r_n$ among respondents. Unlike traditional Heckman selection models, this approach does not strictly require an exclusion restriction (a variable in $z_n$ excluded from the outcome equation) because the ordered-probit threshold structure, combined with unit-variance normalization, pins down $\rho$ . However, the authors note that including a predictor in the response equation that affects response propensity but is plausibly excluded from the outcome can sharpen identification, particularly at extreme nonresponse rates.

Empirical Results
The method is applied to the 2024 American National Election Study (ANES) Preliminary Release, utilizing a 65% nonresponse rate assumption. The response-propensity proxy used is the respondent's self-rating of the interview (7 levels). Covariates include marital status, spouse's gender, race, and education, defining 60 strata.

The analysis reveals that the magnitude of correction is highly outcome-specific:

Life Satisfaction: A strong nonignorable nonresponse bias is detected ( $\hat{\rho} \approx 0.47$ ). Correcting for this shifts the estimated population distribution significantly, assigning a larger proportion to "Slightly satisfied" or "Not satisfied at all" categories compared to unadjusted estimates. This aligns with prior findings that well-being data are sensitive to respondent difficulty.
Retrospective Economic Evaluations: The bias is modest ( $\hat{\rho} \approx 0.14$ ), resulting in only small shifts in the distribution toward "gotten worse."
Other Outcomes: For measures such as media trust, abortion importance, and the death penalty, the estimated $\rho$ is statistically indistinguishable from zero, and the corrected distributions nearly coincide with survey-weighted unadjusted distributions.

The authors provide an R routine implementing this estimator, which handles both ordinal and binary outcomes and runs efficiently on standard hardware (approx. 10 minutes for the full ANES estimation).

Significance and Claims
The paper claims to provide a practical, computationally light tool for sensitivity-aware descriptive inference in observational surveys. Its primary contributions are:

Generalization: Extending the VRP framework from binary to ordinal outcomes, preserving information inherent in multi-category scales.
Practicality: Offering a maximum likelihood estimator that integrates response-propensity proxies and post-stratification without requiring randomized instruments or multi-stage callback designs.
Empirical Nuance: Demonstrating that the value of correcting for nonignorable nonresponse is not uniform; it is substantively meaningful for specific attitudinal measures (like life satisfaction) but negligible for others.

The authors maintain a modest stance, acknowledging that conclusions depend on the assumed latent structure and the credibility of the response-propensity proxy. They position the method not as a definitive correction but as a transparent mechanism to make the extrapolation from respondents to nonrespondents explicit, allowing researchers to assess the sensitivity of their survey estimates to nonignorable selection.

The Great Survey Mystery: When the People Who Don't Show Up Are the Most Important

The Paper: Catching the Silent Crowd with a "Cooperation Meter"

What They Found: Not All Surveys Need a Rescue

The Takeaway: One Size Does Not Fit All

More like this