From Ground Truth to Measurement: A Statistical Framework for Human Labeling

1. Problem Statement

Supervised machine learning relies on the assumption that labeled data provides accurate measurements of the target concepts. However, human labeling introduces systematic variation due to ambiguous items, divergent interpretations, and cognitive errors. Current research often treats all label disagreement as random "noise" to be filtered out or corrected post-hoc. This approach obscures the underlying structure of errors, limits the understanding of what models actually learn, and fails to distinguish between:

• Measurement Error: Mistakes arising from ambiguity or lack of skill.
• Human Label Variation (HLV): Systematic, valid differences in interpretation due to diverse perspectives, values, or expertise (common in subjective tasks like toxicity detection or sentiment analysis).

The paper argues that treating labeling as a simple error-correction problem ignores the structured nature of human judgment, which is better understood through the lens of measurement theory.

2. Methodology

The authors propose a statistical framework that reframes annotation as a measurement process, extending classical measurement error models to accommodate both shared and individualized notions of truth.

A. Conceptual Decomposition of Variance

The framework decomposes the observed labeling outcome ($Y_{ijt}$) into four interpretable sources of variation:

1. Instance Difficulty ($\beta_j$): Systematic error arising from the inherent ambiguity or complexity of the data instance (e.g., occlusion in images, semantic ambiguity in text).
2. Annotator Bias ($\rho_i$): Stable, between-person differences in tendencies, expertise, or ideology (e.g., a strict vs. lenient annotator).
3. Situational Noise ($\sigma_{ijt}$): Within-person, transient fluctuations due to fatigue, distraction, or environmental factors.
4. Relational Alignment / Individual Truth ($\delta_{ij}$): In tasks with Human Label Variation, the "true" label is not a single point but an annotator-specific latent construct. This component captures stable, systematic differences in how different annotators interpret the same instance.

B. Statistical Modeling

The authors operationalize these concepts using Generalized Linear Mixed Models (GLMMs):

• Latent Propensity: They posit a latent continuous variable $Y^*_{ijt}$ representing the propensity to assign a correct label:
  $$Y^*_{ijt} = \mu_j + \beta_j + \rho_i + \sigma_{ijt}$$
  (Where $\mu_j$ is the latent truth, $\beta_j$ is instance difficulty, $\rho_i$ is annotator bias, and $\sigma_{ijt}$ is noise).
• Observation Model: The observed categorical label is generated via a link function (e.g., logit/probit) applied to the latent propensity.
• Two Regimes:
  1. Global Ground Truth: Assumes a single $\mu_j$ for all annotators. Error is defined as deviation from a consensus.
  2. Individual Ground Truth: Assumes annotator-specific truths ($\mu_{ij} = \mu_j + \delta_{ij}$). Error is defined relative to an annotator's own self-consistent interpretation.

C. Diagnostic Approach

To determine which regime applies to a specific dataset, the authors propose a Pairwise Validation Model. This model analyzes the probability that Judge $k$ validates Labeler $i$'s annotation for Item $j$. It decomposes variance into:

• Item-level variance ($\sigma^2_t$): Indicates shared difficulty (Global Truth regime).
• Labeler/Judge variance ($\sigma^2_l, \sigma^2_j$): Indicates stable individual tendencies.
• Interaction variance ($\sigma^2_{lj}$): Indicates structured relational alignment (e.g., specific pairs of annotators consistently agree or disagree), supporting the HLV regime.

3. Key Contributions

1. General Measurement Error Framework: Formalizes annotation as a measurement process, decomposing variance into instance, annotator, and residual components to distinguish between ambiguity and bias.
2. Extension to Individual Ground Truths: Adapts measurement theory to tasks where multiple valid interpretations exist, linking HLV to concepts of reliability and validity in psychometrics.
3. Empirical Demonstration: Applies the framework to the VariErr NLI dataset (a Natural Language Inference dataset with multi-annotator labels and written explanations).
4. Diagnostic Tool: Develops a method to empirically assess whether a task is best modeled as a search for a single global truth or as an aggregation of structured individual perspectives.

4. Results (Case Study: VariErr NLI)

The authors applied their models to 500 NLI items labeled by four annotators, who also provided explanations and re-evaluated their own labels later.

• Global Ground Truth Model:
  • Found that ambiguity was the strongest predictor of error (ambiguous items were ~9x more likely to be mislabeled).
  • Significant variance existed across items, annotators, and trials, confirming that errors are structured, not random.
• Individual Ground Truth Model:
  • Annotator-level variance dominated over item-level variance.
  • Interestingly, ambiguity predicted fewer self-perceived errors in this model. Annotators viewed their own decisions on ambiguous items as internally coherent, even when others disagreed.
  • Textual features (lexical overlap, sentence length) explained little variance; stable annotator tendencies were the primary driver.
• Pairwise Validation Model:
  • Revealed substantial labeler–judge interaction variance. Certain pairs of annotators consistently aligned or diverged in their reasoning.
  • This indicates that disagreement follows predictable relational patterns (interpretive subcommunities) rather than random noise.

Conclusion of Results: The VariErr dataset is better characterized by the Human Label Variation (HLV) regime. Disagreement is driven by stable, systematic interpretive differences among annotators rather than just shared item difficulty.

5. Significance and Implications

• Paradigm Shift: Moves the field from "denoising" data to diagnosing the nature of disagreement. It suggests that in many subjective tasks, collapsing labels into a single consensus may discard valid information.
• Data-Centric AI: Provides a principled basis for dataset curation.
  • If Global Truth dominates: Focus on disambiguating items and improving instructions.
  • If HLV dominates: Focus on preserving disagreement, collecting rater metadata, and modeling label distributions.
• Modeling & Evaluation:
  • Training: In HLV regimes, models should be trained on label distributions or conditioned on rater clusters rather than single consensus labels.
  • Evaluation: Metrics should move beyond simple accuracy against a single label to distributional metrics or rater-conditioned agreement.
• Scientific Lens: Treats annotation as a window into human reasoning, norms, and interpretive communities, transforming labeling from a preprocessing bottleneck into a source of scientific insight.

In summary, the paper provides a rigorous statistical toolkit to determine why annotators disagree, enabling more responsible and effective machine learning systems that respect the complexity of human judgment.