📈 economics

Coarsening Latent-Class Probabilities: Directional Distortion and Coverage Loss

This paper demonstrates that coarsening calibrated probability vectors into hard labels introduces anisotropic directional distortion and severe coverage loss in regression estimates, but provides a method to quantify and approximate this bias using observable data before inference is reported.

Original authors: Marcell T. Kurbucz

Published 2026-08-13

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcell T. Kurbucz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about why some people get a promotion while others don't. You suspect it has something to do with their "group," like their background or where they grew up. But here's the catch: the official records don't say which group anyone belongs to. Instead, you have a super-smart computer program that looks at a person's resume and says, "I'm 80% sure this person is from Group A, 15% Group B, and 5% Group C." This is called a probability vector. It's a nuanced, detailed guess.

In the world of data science, this is a common tool. Sometimes, however, people opt for a simpler approach. They look at that fancy 80-15-5 guess and say, "Eh, let's just pick the biggest number. This person is definitely Group A." They throw away the 15% and the 5% and turn the soft, wobbly guess into a hard label. It's like taking a high-definition photo of a blurry face and then tracing over it with a thick black marker, deciding the person is only one thing. The big question is: does this shortcut hurt our investigation? Does turning a detailed probability into a simple "yes/no" or "Group A" label ruin our ability to measure the truth?

This paper, written by Marcell T. Kurbucz, dives deep into that exact problem. The author asks: if we take a calibrated probability vector (a smart, nuanced guess) and smash it into a hard label (a simple, rigid category), what happens to our results? The paper finds that this "coarsening" isn't just a small mistake; it's a directional distortion. Imagine you are trying to measure the wind. If you turn your sensitive wind vane into a simple "North or South" sign, you might get the direction right sometimes, but you'll lose all the information about how strong the wind is blowing from the East or West. The paper shows that this shortcut doesn't just shrink your results; it warps them in specific directions, making some differences look smaller than they are while leaving others mostly alone.

The most surprising and useful finding is that we can actually predict exactly how bad the distortion will be before we even run the final analysis. The author developed a mathematical "distortion map" (called a coarsening operator) that uses only the data we already have—the probabilities and the other information—to tell us how much our results will be twisted. The paper runs simulations and checks real-world data, like voter records and job applications, to prove this. In one test, using a hard label instead of the probability vector made a confidence interval (a range where the true answer is likely to hide) 39% narrower, but it only captured the true answer 1% of the time. It was a very precise-looking lie.

The paper also argues against the idea that we can just "fix" this later by assuming the computer made random mistakes. The author shows that the way the computer makes mistakes isn't random; it's tied to the specific details of the data. If you try to correct for it using old methods that assume random errors, you might make things worse. Instead, the paper suggests that if you must use hard labels, you should at least know exactly how much you are losing. But the best advice? Don't throw away the nuance. Keep the probability vector. It's the difference between seeing a blurry but accurate map and tracing over it with a marker that leads you straight off a cliff.

The Core Discovery: The "Distortion Map"

The paper's main discovery is that when you replace a detailed probability guess with a simple hard label, you aren't just adding noise; you are applying a specific, uneven filter to your data. Think of it like looking at a 3D object through a funhouse mirror. Some parts of the object get stretched, some get squashed, and some get twisted sideways.

The author calls this the coarsening operator. It's a mathematical machine that takes your true effect (the real difference between groups) and spits out a distorted version. The paper proves that this distortion depends entirely on what information you threw away. If the information you discarded (the 15% and 5% guesses) was totally unrelated to what you kept, you'd be fine. But in the real world, that discarded info is usually related to what you kept. So, the distortion happens.

Crucially, the paper shows that this distortion is anisotropic. That's a fancy word meaning "it depends on the direction." If you are measuring the difference between Group A and Group B, the hard label might shrink that difference by half. But if you are measuring Group A versus Group C, it might only shrink it by 10%. You can't just say "the results are 20% smaller." You have to say "the results are smaller in this specific direction."

The "Coverage" Trap

One of the most dramatic findings concerns confidence intervals. In statistics, when we say "we are 95% confident the answer is between X and Y," we expect that if we did the experiment 100 times, the true answer would fall inside that range 95 times.

The paper simulates what happens when researchers use hard labels. They found that the intervals look better than they actually are. The hard label makes the numbers look more precise, so the range (the interval) gets narrower. It looks like a tighter, more confident guess. But because the center of that interval has been shifted by the distortion, it misses the true answer almost every time.

In one simulation, the paper showed that after using a hard label (specifically, picking the most likely group), the resulting interval was 39% narrower than it should have been. But instead of covering the truth 95% of the time, it only covered it 1% of the time. It was a very narrow, very confident, and completely wrong guess. The paper provides a formula to calculate exactly how bad this will be before you even publish your results, using only the data you already have.

Real-World Audits: When the Shortcut Fails

The author didn't just stop at math; they tested this on real data to see if the theory holds up.

The Voter Turnout Audit: The paper looked at voter records in North Carolina to see if there were racial disparities in voting. Since the voter files didn't always list race, they used a surname-based probability (a guess based on last names). When they used the full probability guess, they found clear gaps in voting rates between different groups. When they switched to hard labels (just picking the most likely race), those gaps shrank significantly. The hard label made the disparity look smaller than it really was, effectively hiding the inequality.
The Job Application Audit: They looked at a dataset of job applications to check for racial bias in income. Here, the paper found a twist. The "hard label" approach actually worked better in some cases, but not because the math was right. It worked better because the computer's guess was based on things (like job titles) that directly influenced income, which violated the rules of the experiment. The hard label accidentally "filtered out" some of that bad information. This taught the author that you can't just blindly trust the soft or the hard method; you have to check why the computer is making its guesses.

What This Means for You

The paper isn't telling us to stop using computer guesses. It's telling us to stop opting for simpler approaches with them.

Don't throw away the nuance: If you have a probability vector (a 60/30/10 split), use it. Don't just pick the 60.
Know your distortion: If you must use a hard label, the paper gives you a tool to calculate exactly how much you are distorting the truth. You can see which directions are being squashed and which are being stretched.
Beware the "Confident" Lie: A narrow confidence interval isn't always a good thing. If you've coarsened your data, that narrow interval might be a trap, giving you false confidence in a wrong answer.

The paper concludes that while hard labels are tempting because they are simple, they come with a hidden tax: directional distortion. We can measure this tax, and we can predict it, but the only way to avoid paying it is to keep the detailed probabilities alive in our analysis. The "funhouse mirror" effect is real, and the only way to see the true shape of the world is to stop tracing over the reflection.

Technical Summary: Coarsening Latent-Class Probabilities: Directional Distortion and Coverage Loss

Problem Statement
In empirical research involving latent group membership (e.g., protected attributes in fairness audits, disease subtypes, or poverty status), researchers often observe a calibrated probability vector $\mathbf{p}$ rather than the true discrete membership indicator $\mathbf{g}$ . A pervasive practice is to "coarsen" this vector into a hard label (e.g., an $\text{argmax}$ class assignment or a thresholded label) before regressing an outcome $Y$ on it. This paper investigates the statistical costs of this coarsening. Specifically, it asks: what is the bias introduced to the estimated group effects, how does this bias vary across different directions of the effect vector, and how does this distortion impact the validity of standard confidence intervals? The paper addresses this without requiring ground-truth labels or a fitted latent-class model, relying instead on the observed probability vector and covariates.

Methodology and Framework
The analysis rests on two core assumptions:

Structural Conditional Mean: $E[Y | \mathbf{g}, \mathbf{p}, \mathbf{X}] = \mu(\mathbf{X}) + \boldsymbol{\tau}^\top \mathbf{g}$ , where $\boldsymbol{\tau}$ represents the constant structural effects of latent classes relative to a reference, and $\mu(\mathbf{X})$ is a nonparametric control function.
Conditional Calibration: $E[\mathbf{g} | \mathbf{p}, \mathbf{X}] = \mathbf{p}$ almost surely.

Under these assumptions, the problem reduces to a partially linear regression of the outcome residual $R = Y - \mu(\mathbf{X})$ on the residualized score $\mathbf{a} = \mathbf{p} - E[\mathbf{p}|\mathbf{X}]$ . The identification of $\boldsymbol{\tau}$ relies on the residual-score covariance matrix $\mathbf{M} = E[\mathbf{a}\mathbf{a}^\top]$ .

The paper introduces a coarsening operator to characterize the distortion caused by replacing $\mathbf{p}$ with a coarser map $\mathbf{h}(\mathbf{p}, \mathbf{X})$ (e.g., hard labels). The plug-in estimator using $\mathbf{h}$ converges to $A_h \boldsymbol{\tau}$ , where the distortion matrix is:
$A_h = \mathbf{I} + \mathbf{D}_h^{-1} E[\mathbf{a}_h \mathbf{u}^\top]$
Here, $\mathbf{a}_h$ is the residual of the coarsened signal, and $\mathbf{u} = \mathbf{a} - \mathbf{a}_h$ is the discarded signal. The paper demonstrates that coarsening is unbiased if and only if the discarded signal is uncorrelated with the retained signal. Otherwise, the distortion is anisotropic: it attenuates different contrasts (directions) by different amounts, determined by the spectral properties of $\mathbf{M}$ and the specific coarsening rule.

Key Contributions and Results

Directional Distortion: Unlike the binary case where coarsening acts as a scalar dilution factor, in the multiclass setting, the distortion is a matrix operation. The paper proves that hard-labeling (e.g., $\text{argmax}$ ) does not shrink all effects uniformly; it disproportionately damages contrasts aligned with weak eigen-directions of $\mathbf{M}$ . The retention coefficient $s_j$ for the $j$ -th eigen-direction is generally distinct and often significantly less than 1.
Coverage Loss and Diagnostic: The paper derives the limiting coverage of the standard Wald interval constructed from coarsened labels. The coverage is given by $\Phi(z - \lambda_v) - \Phi(-z - \lambda_v)$ , where $\lambda_v$ is the ratio of the directional bias to the reported standard error.
- Paradox of Precision: Because coarsening often reduces the variance of the estimator (making the interval narrower) while introducing bias, the reported intervals can be deceptively precise yet have near-zero coverage.
- Pre-reporting Diagnostic: Crucially, the paper shows that the coverage loss can be estimated before reporting the interval. Since the distortion matrix $A_h$ and the standard error depend only on observables $(\mathbf{Y}, \mathbf{p}, \mathbf{X})$ , researchers can compute an estimated non-centrality index $\hat{\lambda}_v$ to approximate the true coverage of their interval without needing ground-truth labels.
Spectral Identification and Failure: The paper characterizes identification failure spectrally. If an eigenvalue of $\mathbf{M}$ is zero, the corresponding linear combination of effects is set-identified (partially identified). If eigenvalues are small but non-zero, the effects are point-identified but with high variance. The paper establishes that miscalibration (violation of Assumption 2) is most damaging in these weak directions, with bias bounded by $\delta \|\boldsymbol{\tau}\|_1 \lambda_j^{-1/2}$ .
Empirical Audits and Recalibration Protocol:
- UCI Adult & Forest CoverType: In these real-data audits, the paper finds that exclusion violations (where proxy features directly predict the outcome) and miscalibration lead to severe directional distortion. A proposed sequential protocol involves recalibrating the scores on a labeled subsample. If recalibration fixes the audit, the failure was calibration; if the audit worsens or remains broken, the failure is likely an exclusion violation (direct channel), rendering soft estimators unreliable.
- North Carolina Voter Turnout: In an audit using surname-based race proxies, the framework demonstrates that the "soft" estimator recovers the magnitudes of large disparity gaps that are systematically attenuated by hard-labeling. However, the paper explicitly notes that the soft estimator does not match the full-information benchmark exactly; it exhibits statistically resolvable deviations and residual bias on the Black gap, reflecting that the national surname list is not perfectly conditionally calibrated to a single county. Despite these deviations, the soft estimator remains twice as close to the benchmark as hard-labeling in terms of RMSE.

Significance and Scope
The paper claims to provide a diagnostic framework rather than a correction method. Its primary contribution is quantifying the cost of the "classify-then-analyze" strategy in terms of directional bias and interval validity. It argues that for multi-class problems, the standard practice of converting probabilities to hard labels induces complex, direction-specific distortions that standard error estimates fail to capture.

The framework is most applicable when:

Membership is unobserved but a calibrated probability vector is available.
The classifier's features are strictly richer than the downstream controls (ensuring residual variation).
The classifier is genuinely uncertain (not deterministic).
There is no trusted generative model for the measurement process.

The paper concludes that while the "soft" (probability-based) estimator is generally superior to hard-labeling, its validity is contingent on conditional calibration and the exclusion restriction. The proposed diagnostics allow researchers to assess the reliability of their intervals and identify which specific assumptions (calibration vs. exclusion) may be violated, particularly in settings where ground-truth labels are scarce or unavailable for the full sample.

The Core Discovery: The "Distortion Map"

The "Coverage" Trap

Real-World Audits: When the Shortcut Fails

What This Means for You

More like this