💬 NLP

Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

This paper argues that the standard practice of collapsing annotator disagreement into majority vote labels creates a structural flaw in hate speech detection by falsely certifying contested boundary cases as ground truth, leading models to exhibit high confidence in errors that downstream interventions cannot fix.

Original authors: Joshua Muhumuza, Joab Ezra Agaba, Mercy Amiyo

Published 2026-06-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Joshua Muhumuza, Joab Ezra Agaba, Mercy Amiyo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: When "The Majority" Silences the Minority

Imagine you are trying to decide if a joke is funny or mean. You ask a group of 100 people to vote.

51 people say, "It's just a joke, it's funny."
49 people say, "It's actually mean and hurtful."

In the world of computer science and AI, the standard rule is: "The majority wins." The computer is told, "Okay, 51 people said it's funny, so it is funny." The computer then learns that this is the absolute truth.

This paper argues that this simple rule is dangerous. It treats a messy, complicated human disagreement as if it were a clear fact. The authors show that when you force a computer to learn from this "majority vote," it fails badly on the exact posts where people are most confused.

The Setting: The "HateXplain" Dataset

The researchers looked at a dataset called HateXplain, which contains social media posts labeled by human workers. The workers had to choose between three categories:

Normal (harmless)
Offensive (rude or hurtful, but not hate speech)
Hate Speech (dangerous, targeting a group)

The researchers found that humans often struggle to tell the difference between Offensive and Hate Speech. It's a blurry line, like deciding if a sunset is "orange" or "red."

The Key Findings

1. The Disagreement isn't Random; It's a "Border War"

The authors discovered that nearly 43% of all the arguments between human workers happened right on the border between "Offensive" and "Hate Speech."

Analogy: Imagine a border between two countries. Most people agree on what's inside Country A and what's inside Country B. But right on the border line, people argue constantly.
The Paper's Claim: The data shows that when humans disagree, they aren't just confused randomly. They are specifically fighting over where the line is drawn between "rude" and "hateful."

2. The "Silence" Effect

When the dataset creators took the votes and picked the winner (the majority), they effectively silenced the minority.

If 2 people said "Hate" and 1 person said "Offensive," the computer learns "Hate."
If 2 people said "Offensive" and 1 person said "Hate," the computer learns "Offensive."
The Result: The computer never learns that there is a legitimate debate happening. It thinks the answer is clear-cut, even when it's not.

3. The AI Gets Confidently Wrong

The researchers trained three different types of AI models to see if they could fix this:

Model A (The Standard): Learned from the "Majority Vote" labels.
Model B (The Soft Learner): Learned from the percentages (e.g., "60% said Hate, 40% said Offensive").
Model C (The Individualist): Tried to learn what each specific worker thought, rather than just the group average.

The Shocking Result:

All three models were great at labeling posts where humans agreed (about 80% accuracy).
But on the posts where humans disagreed (the border cases), all three models crashed, dropping to about 55% accuracy.
The "Overconfidence" Trap: The standard model (Model A) didn't just get these wrong; it got them wrong with high confidence. It would say, "I am 71% sure this is Hate Speech," when it was actually a messy, debatable case.
The "Honest" Model: Model C (the individualist) was less confident (49%) on these errors. It was more honest about its uncertainty, but it was still just as inaccurate.

Analogy: Imagine a weather forecaster.

Model A looks at a cloudy day and says, "It will definitely rain!" (High confidence, but wrong).
Model C looks at the same day and says, "I'm not sure, maybe rain, maybe not." (Low confidence, but still doesn't know the answer).
The Point: Being "honest" about not knowing doesn't help if the system still makes the wrong decision.

Why Can't We Just Fix the AI?

The researchers tried three different "downstream" fixes (changing how the AI learns or how it votes). None of them worked.

The Analogy: Imagine you are trying to teach a student to draw a perfect circle. But the teacher keeps showing the student a picture of a square and saying, "This is a circle."
- You can try to teach the student with a softer voice (Soft Labels).
- You can try to show them every teacher's perspective (Per-Annotator Heads).
- But it doesn't matter. If the "Ground Truth" (the answer key) is wrong, the student will never learn to draw a circle.

The paper concludes that the problem isn't the AI model; the problem is the annotation design. You cannot fix a broken foundation by painting the walls.

The Proposed Solution: Change the Rules Before You Start

Since fixing the AI didn't work, the authors say we must fix the human process before the AI ever sees the data. They suggest three changes:

Stop Using "Yes/No" Labels: Instead of forcing workers to choose between "Offensive" or "Hate," give them a scale (like 1 to 5). This acknowledges that some things are "kind of offensive" and "kind of hate," rather than forcing a binary choice that creates arguments.
Flag the "Border Cases": When the AI sees a post where it's unsure (low confidence), it shouldn't make a final decision. It should flag it for a human to review.
Include the People Affected: The paper argues that the people whose content is being judged (e.g., specific cultural or minority groups) should be the ones doing the labeling. Their definition of "hate" might be different from the majority. If you only ask the majority, you will always miss the minority's perspective.

A Warning on the Solution

The authors are careful to note a major ethical issue with point #3. Asking members of marginalized communities to look at hate speech directed at them can be psychologically harmful. They argue that if we do this, we must pay them well, give them mental health support, and let them opt-in voluntarily. We shouldn't just dump the burden of moderation onto the people getting hurt.

Summary

The paper argues that majority voting in AI training is a trap. It turns complex human disagreements into fake "facts." This makes AI models confidently wrong on the most sensitive topics. The only way to fix this isn't to build smarter AI, but to change how humans label the data in the first place.

Technical Summary: Majority Vote Silences Minority Values

Problem Statement
Current hate speech detection pipelines treat the distinction between "hate speech" and "offensive speech" as an objective fact, aggregating crowdworker labels via majority vote to create a single ground truth. This paper argues that this aggregation is not neutral; it systematically silences minority values by collapsing genuine annotator disagreement into a single label. The authors posit that this "false certainty" is inherited by machine learning models, leading to dangerous failure modes where systems are both incorrect and highly confident on culturally contested inputs. The core problem is structural: standard evaluation metrics and downstream interventions fail to detect or recover the accuracy lost when majority vote discards minority perspectives at the hate/offensive boundary.

Methodology
The study utilizes the HateXplain dataset, comprising 20,148 posts annotated by three crowdworkers each with labels of hatespeech, offensive, or normal. The authors define "disagreement" as any post where annotators did not unanimously agree and "boundary cases" as disagreements occurring exclusively between hatespeech and offensive (excluding normal).

To test the impact of aggregation, the authors trained three BERT-base-uncased classifiers of increasing sophistication on the test set ( $n=1,892$ ):

Model A (Hard): Trained on standard majority vote labels using cross-entropy loss.
Model B (Soft): Trained on soft labels derived from the raw annotator distribution (e.g., a 2-1 split yields probabilities of [0.67, 0.0, 0.33]) using KL divergence loss.
Model C (Per-Annotator): A shared encoder with separate classification heads for each of the 106 qualifying annotators. During inference, heads are ensembled by averaging softmax probabilities.

The authors employed statistical analysis (Chi-square tests, Mann-Whitney U tests) to measure disagreement concentration, accuracy gaps between agreed and disagreed posts, and model confidence levels. They also analyzed token-level rationale annotations to distinguish between genuine threshold disagreement and annotation noise (misreading).

Key Contributions

Concentration of Disagreement: The study quantifies that 42.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary ( $\chi^2 = 135.199, p < 0.0001$ ). This is not random noise but a structural pattern where annotators apply different thresholds for harm. Offensive posts exhibit a significantly higher disagreement rate (67.9%) compared to normal posts (35.5%).
The Agreement Gap: A persistent accuracy gap exists between agreed posts (~~80% accuracy) and disagreement posts (~~55–58% accuracy) across all models ( $p < 0.0001$ ). This gap is obscured by standard overall accuracy metrics.
Failure of Downstream Interventions: Three increasingly sophisticated downstream interventions (hard labels, soft labels, and per-annotator heads) all fail to recover boundary accuracy. In fact, the strongest baseline (Model C) widens the agreement gap from 22.2 points (Model A) to 28.1 points, with offensive disagreement accuracy collapsing to 0.245.
Overconfidence in Hard Labels: Standard hard-label models (Model A) express significantly higher confidence on boundary case errors (0.710) compared to per-annotator models (Model C, 0.495). This indicates that standard metrics will not detect the failure, as the model is "wrong and certain."

Results

Accuracy: While Model C achieved the highest accuracy on agreed posts (0.826), it performed the worst on disagreement posts (0.545). No model recovered the accuracy lost at the boundary.
Calibration: The only consistent improvement across models was calibration. Model C became appropriately uncertain on boundary inputs (confidence dropped to 0.495), whereas Model A remained overconfident (0.710). However, this increased uncertainty did not translate to correct predictions.
Annotation Error vs. Threshold Disagreement: Analysis of token-level rationales showed that 73.1% of different-label pairs highlighted substantially overlapping tokens (Jaccard $\ge 0.3$ ), and a distinct mode existed at Jaccard = 1.0 (identical tokens, different labels). This evidence suggests the disagreement stems from differing thresholds on shared evidence rather than annotator inattention or error.
Minority Alignment: All models remained strongly majority-aligned (~70% of predictions on disagreement posts matched the majority label), confirming that downstream interventions do not recover minority perspectives.

Significance and Claims
The paper claims that the failure to handle hate/offensive boundaries is structural, not architectural. The problem lies in the annotation design where majority vote presents a contested judgment as ground truth. Consequently, models inherit this "false certainty."

The authors argue that downstream interventions are insufficient. Because the label itself is unreliable on contested inputs, no amount of model sophistication or soft-labeling can recover the lost accuracy. The necessary intervention must be upstream in the annotation process. The paper suggests three specific upstream changes:

Replacing discrete hate/offensive distinctions with graduated harm scales.
Flagging high-disagreement posts at inference for human review.
Ensuring annotation pools represent the target communities whose content is being moderated, though the authors note the ethical complexities and risks of exposing marginalized annotators to harmful content.

The study concludes that pluralistic alignment requires representation at the annotation stage, not just the model stage, to prevent the systematic misclassification of linguistic norms specific to minority communities.