Quantifying and Predicting Disagreement in Graded Human Ratings

Technical Summary: Quantifying and Predicting Disagreement in Graded Human Ratings

Problem Statement
Human annotators frequently exhibit disagreement in Natural Language Processing (NLP) tasks, particularly in socially sensitive domains such as hate speech, offensive language, and toxic content detection. While it is acknowledged that annotation variation is inherent, the degree of this divergence is not uniform across all instances. Some items elicit strong consensus, while others trigger significant opinion divergence or opposing stances. Current literature often treats disagreement as a categorical prediction problem (e.g., binary classification of whether disagreement exists) or focuses on specific task labels (e.g., NLI entailment). There is a lack of research examining whether the degree of annotation uncertainty and the full structure of fine-grained annotation distributions (specifically Likert-scale ratings) can be inferred solely from item-level textual features. Furthermore, standard metrics for distributional shape (like bimodality coefficients) often fail when applied to sparse annotation data (e.g., 3–5 annotators per item).

Methodology
The authors investigate the predictability of annotation variation using three datasets annotated with Likert-scale ratings: Offensive Language (3-point scale), Hate Speech (3-point scale), and Toxicity (5-point scale). The study focuses on two primary objectives:

1. Inferring Rating Variance: Estimating the unbiased variance of annotator responses for a given item based on its textual features.
2. Identifying Opposing Opinions: Detecting instances where annotators hold genuinely conflicting stances (polarization).

To address these objectives, the authors propose two prediction approaches:

• Direct Variance Prediction: Training a regression model to predict the variance value ($\sigma^2$) directly from item features.
• Distribution-Based Prediction: Training models to predict the full Likert rating distribution ($p_i$) for an item, from which variance and other statistics are derived.

The study employs several loss functions tailored to the ordinal nature of Likert data, moving beyond standard categorical cross-entropy:

• Earth Mover's Distance (EMD): Accounts for the ordinal distance between rating categories.
• EMD with Mean Regularization: Combines EMD with a mean squared error term on the expected rating to constrain central tendency.
• Ordinal Cumulative Cross Entropy: Transforms the multi-class problem into $K-1$ binary decisions.
• Kullback–Leibler (KL) Divergence: Used as a baseline for soft-label representation without explicit ordinal distance consideration.

To quantify opposing stances, the authors introduce the Opposition Index. This metric divides the rating distribution into low, mid, and high segments ($P_{low}, P_{mid}, P_{high}$). The index is defined as $2 \cdot \min(P_{low}, P_{high}) \cdot (1 - P_{mid})$. It ranges from 0 (consensus/unimodal) to 1 (maximal polarization, where half the annotators choose the lowest rating and half the highest).

Key Contributions

• Opposition Index: A novel metric designed to quantify perspective opposition in sparse annotation settings where traditional bimodality tests are unstable.
• Distributional Modeling: A demonstration that predicting the full Likert distribution (using ordinal-aware loss functions) is a viable alternative to direct variance regression, often recovering the structure of annotation variation more robustly.
• Empirical Analysis of Polarization: A detailed investigation into the predictability of high-opposition items, revealing specific challenges in modeling extreme polarization.

Results

• Variance Prediction: There is a moderate positive correlation (Spearman's $r \approx 0.3$ to $0.45$) between estimated and observed annotation variance across all datasets.
  • Performance: Direct variance regression and distribution-based prediction (specifically using EMD+MSE) achieve comparable performance. EMD+MSE consistently outperforms standard EMD and KL-soft models.
  • Limitations: Models tend to underestimate high-variance items and rarely predict variance values of exactly 0, often concentrating predictions in a middle range.
• Opposing Stance Prediction: Models successfully distinguish between items with no opposition and those with moderate opposition. However, they struggle with extreme polarization.
  • Underestimation: Items with high true Opposition Index values (approaching 1) are frequently underestimated by models, with predicted scores often plateauing in a mid-range (approx. 0.4).
  • Causes: This underestimation is attributed to the scarcity of highly polarized examples in training data, the sensitivity of the index to annotation noise, and the fact that extreme opposition may stem from factors (ideology, personal experience) not inferable from text alone.
• Disagreement Detection: Models achieve strong performance (F1 > 0.6) in detecting the presence of any annotation variation, though performance varies slightly by dataset and loss function.

Significance and Claims
The paper claims that item-level annotation patterns, specifically variance and stance opposition, can be partially inferred from textual features, though with moderate correlation. The authors position their work as a step toward characterizing the structures of human annotation variation.

The primary significance lies in practical annotation workflow optimization. By identifying items likely to elicit substantial opinion divergence, researchers can allocate resources more efficiently: assigning more annotators to controversial or perspective-divergent cases to capture latent viewpoints, while reducing effort on items with strong consensus. The study also highlights the limitations of current models in capturing extreme polarization, suggesting that purely text-based features may be insufficient for fully modeling human judgment in socially complex domains. The authors remain modest, noting that embedding models may not capture all linguistic cues and that the discrete nature of sparse Likert data limits the granularity of distributional estimates.