🤖 machine learning

Hierarchical Bayesian Crowdsourcing with Item Difficulty

This paper introduces a hierarchical Bayesian crowdsourcing model that extends the Dawid-Skene framework by incorporating item-level effects for difficulty, discriminativeness, and guessability to better handle noisy and biased rating data, while also providing methods to constrain adversarial raters and validate performance through posterior predictive checks and cross-validation.

Original authors: Seong Woo Han, Ozan Adıgüzel, Bob Carpenter

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seong Woo Han, Ozan Adıgüzel, Bob Carpenter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out the true answer to a difficult question, like "Is there a cavity in this tooth?" or "Does this sentence logically follow that one?" You don't know the answer yourself, so you ask a crowd of people to vote.

In the past, the standard way to handle this was Majority Voting: if 6 out of 10 people say "Yes," the answer is "Yes." But this paper argues that this is like asking a room full of people to guess the weight of a pumpkin, where some people are experts, some are guessing randomly, and some are actively trying to trick you. If you just take the average, you get a messy, biased result.

This paper introduces a smarter, more mathematical way to listen to the crowd, called Hierarchical Bayesian Crowdsourcing. Here is how it works, broken down into simple concepts:

1. The Problem: Not All Voters Are Created Equal

The authors point out that in real life, voters (or "raters") are messy.

The Expert: Always gets it right.
The Confused: Gets it right half the time.
The Gambler: Guesses randomly.
The Troll: Knows the answer but deliberately votes the opposite way (an "adversarial rater").

Old models (like the famous Dawid-Skene model) tried to fix this by asking, "How good is this person at saying 'Yes' when it's 'Yes'?" and "How good are they at saying 'No' when it's 'No'?" But these models treated every item (like every tooth or every sentence) as if it were equally easy to judge. They missed the fact that some items are just hard.

2. The Solution: A Three-Layered Filter

The authors built a new model that acts like a sophisticated filter with three specific knobs for every single item being rated:

Difficulty (The Mountain): Some items are just hard to judge (a steep mountain). Even experts might struggle. The model learns which items are "steep mountains" and which are "flat hills."
Discrimination (The Magnifying Glass): Some items clearly separate the experts from the novices. If an item has high "discrimination," the experts will all agree, and the novices will all disagree. If it has low discrimination, everyone is confused.
Guessability (The Lucky Coin): Sometimes, even if an item is impossible, people might guess the right answer by luck. The model accounts for this "lucky coin" factor so it doesn't mistake a lucky guess for skill.

3. Catching the "Trolls"

A major innovation in this paper is how it handles adversarial raters (the trolls who vote the opposite of the truth).

The Old Way: The math often got confused, thinking a troll was actually an expert who just saw things differently. This created a "bimodal" problem (two possible answers that both looked mathematically correct).
The New Way: The authors added a rule: "We assume no one is trying to trick us on purpose." They mathematically constrained the model so that a rater cannot be worse than random guessing. If someone is voting randomly, the model treats them as a "spammy" voter, not a "negative" expert. This makes the final answer much more stable and reliable.

4. The "Probabilistic" Secret Sauce

Usually, when we use crowd data to train a computer (like a neural network), we force the computer to pick a single "Gold Standard" answer (e.g., "Yes, it's a cavity").

The Paper's Insight: This throws away valuable information.
The Better Way: Instead of forcing a "Yes" or "No," the model tells the computer: "There is a 70% chance it's a cavity, but we aren't 100% sure."
The Analogy: Imagine training a student.
- Old Way: You tell the student, "The answer is A." If the student was unsure, they just memorized "A."
- New Way: You tell the student, "Based on the crowd, there's a 70% chance it's A, but be careful, the evidence is shaky."
- Result: The paper shows that training with these "shaky probabilities" makes the computer much smarter than training with forced, hard labels.

5. Did It Work? (The Test Drive)

The authors tested their new model on two real-world datasets:

Dental X-rays: 5 dentists rating thousands of images for cavities.
Language Tasks: Nearly 200 internet workers rating sentence pairs for logical meaning.

The Results:

Their new model (which includes difficulty, discrimination, and guessability) was the best at predicting the true answers.
The old "Majority Vote" and the older "Dawid-Skene" models failed to capture the reality of the data. They thought there were more "middle-of-the-road" votes than actually existed.
The new model correctly identified that some items were just too hard to judge and that some raters were unreliable, leading to a much clearer picture of the truth.

Summary

Think of this paper as upgrading from a simple vote counter to a detective. Instead of just counting heads, the detective looks at who is voting, how hard the question is, and how likely it is that someone is guessing or trolling. By doing this, they can reconstruct the true "gold standard" answer much more accurately, which helps train better AI systems.

Technical Summary: Hierarchical Bayesian Crowdsourcing with Item Difficulty

Problem Statement
In applied statistics and machine learning, training data often relies on crowdsourced ratings that are noisy, biased, and inconsistent across raters and items. While the widely used Dawid and Skene (1979) model adjusts for rater sensitivity and specificity, it fails to capture the distributional properties of rating data, such as continuous variation in item difficulty, discriminativeness, and guessability. Furthermore, existing models often struggle with identifiability issues arising from adversarial raters (those systematically more likely to be incorrect than correct) and do not adequately model the heterogeneity of both raters and items. Heuristic approaches like majority voting ignore uncertainty, leading to biased estimates and degraded downstream performance.

Methodology
The authors propose a general-purpose hierarchical Bayesian measurement-error model for binary crowdsourced ratings. The framework extends Item Response Theory (IRT) by integrating rater-specific sensitivity and specificity with item-level parameters for difficulty, discrimination, and guessability.

Generative Model: The model assumes each item $i$ has a latent category $z_i \in \{0, 1\}$ with prevalence $\pi$ . Ratings are generated conditionally on $z_i$ . Each rater $j$ is characterized by sensitivity ( $\alpha^{sens}_j$ ) and specificity ( $\alpha^{spec}_j$ ) on the log-odds scale. Each item $i$ is parameterized by difficulty ( $\beta_i$ ), discrimination ( $\delta_i > 0$ ), and guessability ( $\lambda_i \in (0, 1)$ ). The probability of a correct rating follows a three-parameter logistic model (IRT-3PL) generalized with sensitivity and specificity:
$P(y_n = 1 | z_i=1) = \lambda_i + (1-\lambda_i)\text{logit}^{-1}(\delta_i(\alpha^{sens}_j - \beta_i))$
$P(y_n = 1 | z_i=0) = 1 - [\lambda_i + (1-\lambda_i)\text{logit}^{-1}(\delta_i(\alpha^{spec}_j - \beta_i))]$
Identifiability and Adversarial Raters: To address the bimodal posterior often found in rating models (where a rater's accuracy can be indistinguishable from their inverse), the authors introduce a constraint to exclude adversarial raters. An adversarial rater is defined as one where sensitivity is less than one minus specificity ( $\alpha^{sens}_j < -\alpha^{spec}_j$ ). The model reparameterizes sensitivity as $\alpha^{sens}_j = -\alpha^{spec}_j + \log(1 + \exp(\eta_j))$ , ensuring $\alpha^{sens}_j > -\alpha^{spec}_j$ while retaining flexibility near the "spammy" boundary.
Hierarchical Priors: To model heterogeneity across raters, a hierarchical bivariate normal prior is placed on $(\eta_j, \alpha^{spec}_j)$ . The model employs a non-centered reparameterization to decouple latent variables from their scales, mitigating Neal's funnel and ensuring stable convergence.
Probabilistic Training: The paper advocates for using the posterior probabilities of the latent categories directly for training downstream classifiers (e.g., via log-odds regression) rather than collapsing ratings into hard labels. This preserves uncertainty and avoids bias.
Model Reductions: The framework allows for orthogonal reductions (e.g., fixing $\lambda_i=0$ , $\delta_i=1$ , or tying sensitivity and specificity) to recover standard IRT variants (1PL, 2PL, 3PL) and the Dawid-Skene model as special cases.

Key Contributions

Proposed Model: A hierarchical Bayesian measurement-error model for crowdsourced binary ratings that incorporates item-level difficulty, discrimination, and guessability alongside rater sensitivity and specificity.
Identifiability Constraint: A reparameterization strategy that resolves bimodality by excluding adversarial raters, improving identifiability without sacrificing flexibility near the spammy boundary.
Open-Source Implementation: The model is implemented in Stan, enabling use in R, Python, and Julia, with code made publicly available.

Results
The model was validated using two datasets: dental X-ray caries ratings (5 dentists, ~4000 items) and natural language inference ratings (Mechanical Turkers, ~3000 sentence pairs). Evaluation utilized Posterior Predictive Checks (PPC) and Leave-One-Out Cross-Validation (LOO).

Predictive Accuracy: For both datasets, the IRT-2PL + sens/spec model achieved the best predictive performance (highest $elpd_{loo}$ ). Models incorporating sensitivity and specificity consistently outperformed standard IRT models and the Dawid-Skene model.
Goodness-of-Fit: Posterior predictive checks showed that models without sensitivity and specificity (standard IRT) and the Dawid-Skene model failed to capture the distribution of ratings adequately. Specifically, the Dawid-Skene model inflated the number of "middling" positive votes compared to observed data.
Dispersion: The IRT-2PL + sens/spec model closely matched the dispersion of positive votes per item in the data, whereas the Dawid-Skene model underestimated dispersion and the full IRT-3PL + sens/spec model showed increased variability not well-supported by the data.
Training Impact: Synthetic experiments demonstrated that training classifiers using the posterior probabilities (specifically via log-odds regression) significantly reduced parameter estimation error compared to using hard labels derived from majority voting or maximum probability categories.

Significance and Claims
The paper claims that explicit modeling of rater sensitivity and specificity within a hierarchical Bayesian framework is critical for improving fit and predictive performance over classical IRT models and the Dawid-Skene approach. By capturing item-level variation (difficulty, discrimination) and structured rater behavior, the proposed models produce more accurate and well-calibrated predictions. The authors conclude that most machine learning training efforts utilizing human feedback are not maximizing their potential and can be improved by applying these structured probabilistic crowdsourcing models, particularly by utilizing probabilistic training data rather than collapsed gold standards. The work also highlights the importance of identifying and excluding adversarial raters to ensure model identifiability in typical crowdsourcing settings.