Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

1. Problem Statement

In many machine learning classification tasks (e.g., content moderation, radiology, sentiment analysis), there is no definitive "ground truth." Instead, evaluation relies on human judgments, which often disagree. The paper addresses two critical challenges in these settings:

1. Evaluation: How should a classifier be scored against human labels? The common practice is to combine multiple raters' labels (e.g., via majority vote) to create a "gold standard" and score the classifier against this aggregate. The authors argue this is often theoretically unjustified.
2. Benchmarking: How can we compare a classifier's performance to a human alternative? Specifically, how many human raters would be required to match the performance of a given classifier?

The core issue is that treating human labels as a single "truth" ignores the nature of the task (subjective vs. objective) and the varying costs of errors (equanimity).

2. Theoretical Framework

The authors propose a decision framework based on two properties of the evaluation setting:

A. Two Utility Models

1. Subjective Utility Model:
   • Context: No objective ground truth exists (e.g., "Is this comment dehumanizing?"). Disagreement among raters reflects valid differences in perspective, not noise.
   • Assumption: Utility is defined as the expected score against a randomly selected rater from the population.
   • Implication: The "truth" is the distribution of human judgments, not a single label.
2. Objective Utility Model:
   • Context: An objective ground truth exists (e.g., "Is there a tumor?"), but it is not immediately observable. Rater disagreement is viewed as noise in perceiving the truth.
   • Assumption: Utility depends solely on matching the hidden ground truth ($g_i$).
   • Implication: Human labels are imperfect proxies. If ground truth is unavailable, no finite panel size guarantees a correct evaluation.

B. Equanimity

• Definition: Whether the harm of a misclassification is constant regardless of how obvious the item is.
• Relevance: If equanimity does not hold (e.g., missing a tumor that 90% of doctors would catch is worse than missing a borderline case), the subjective utility model is often the only pragmatic choice even in objective settings because it is directly measurable from individual labels.

3. Methodology and Key Findings

A. Evaluation Strategy: "Score Against One, Average the Scores"

The paper proves that under the Subjective Utility Model:

• Majority Vote is Biased: Scoring a classifier against the majority vote of a panel ($k > 1$) introduces bias. It over-represents the most frequent label, distorting the expected utility.
• Single Rater is Unbiased: Scoring against a single, randomly selected rater provides an unbiased estimate of the expected utility.
• Optimal Practice: To reduce variance without introducing bias, one should score the classifier against each available rater individually and then average the resulting scores.
  • Formula: $Utility \approx \frac{1}{n} \sum_{i} Score(c(x_i), e_{i1})$
  • Contrast: Do not compute $Score(c(x_i), maj(e_{i1}, ..., e_{ik}))$.

B. Benchmarking: Rater Equivalence

To compare a classifier to a human process, the authors introduce Rater Equivalence ($REQ$):

• Definition: The smallest number of human raters ($k$) in a benchmark panel whose combined judgment (via a specific combiner) yields a higher expected score than the classifier.
• Power Curve: A plot showing the expected score of a human benchmark panel as the panel size ($k$) increases.
• Interpretation: If a classifier has an $REQ$ of 3.5, it performs better than a 3-person human panel but worse than a 4-person panel.

C. The Anonymous Bayesian Combiner (ABC)

To ensure fair and optimal benchmarking, the choice of how to combine human labels (the "combiner") is crucial.

• Problem: Simple combiners like Majority Vote or Frequency are not "calibrated" and can artificially inflate the benchmark's performance, leading to an overestimation of the classifier's $REQ$.
• Solution: The Anonymous Bayesian Combiner (ABC).
  • It learns from the dataset to predict the Bayesian posterior probability of the next rater's label given the observed labels of previous raters.
  • It is provably optimal (minimizes $REQ$) when using proper scoring rules like Cross-Entropy.
  • It treats raters as anonymous draws from a distribution, avoiding overfitting to specific rater identities.

4. Results and Case Studies

The authors validated their framework using three case studies from prior literature:

1. Jigsaw Personal Attacks (Wikipedia Comments):
   • Finding: The Jigsaw classifier had an $REQ$ of 7.63 (using ABC + Cross-Entropy).
   • Insight: The original study claimed the model outperformed 5 raters. The new framework shows it actually requires nearly 8 raters to match the classifier, indicating the original evaluation (using majority vote) was overly optimistic about the classifier's relative performance.
2. News Credibility (CredBank):
   • Finding: The classifier had an $REQ$ of 0.48.
   • Insight: The classifier performed worse than a single human rater. The human process (collecting a label for 48% of items and guessing the rest) was superior to the automated model.
3. Reddit Upvotes (Social Rating):
   • Finding: The Reddit net-upvote system had an $REQ$ of 2.39.
   • Insight: Despite having thousands of votes, the aggregate Reddit signal conveyed less information about user preference than a simple poll of 3 people.

5. Significance and Contributions

1. Paradigm Shift in Evaluation: The paper challenges the standard "majority vote as ground truth" approach. It demonstrates that for subjective tasks, averaging scores against individual raters is theoretically superior to scoring against an aggregated label.
2. Quantifiable Human Benchmarking: The concept of Rater Equivalence provides a concrete, interpretable metric for model performance. Instead of saying "the model is 85% accurate," stakeholders can say "the model is equivalent to a 4-person human team."
3. Optimal Combiner Algorithm: The introduction of the Anonymous Bayesian Combiner (ABC) offers a computationally tractable, theoretically optimal method for simulating human benchmarks, ensuring that comparisons are not skewed by poor aggregation methods.
4. Negative Results for Objective Settings: The paper rigorously proves that in settings with objective ground truth but no available labels, no finite evaluation panel size guarantees a correct ranking of classifiers compared to the true ground truth. This highlights the critical need for obtaining ground truth labels (even for a subset) in objective tasks.
5. Practical Tooling: The authors provide a software library (surveyequivalence) to implement these methods, making the framework accessible for practitioners.

Conclusion

The paper argues that evaluating AI systems against human labels requires a nuanced understanding of the nature of the task. By distinguishing between subjective and objective utility and introducing Rater Equivalence, the authors provide a principled, mathematically grounded framework for evaluating and comparing machine learning models in human-centric domains. The key takeaway is: Average the scores, don't score against the average label.