🤖 machine learning

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

This paper reveals that significant disagreement between expert and crowd annotators at the item level is masked by small leaderboard sizes, creating a false sense of stability in model rankings that fails to generalize as the number of models increases.

Original authors: Anik Jha

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anik Jha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the race to build smarter artificial intelligence, researchers rely on a simple but powerful idea: to know if a machine is getting better, you must ask a human to judge its work. This process, known as a preference benchmark, involves showing a human two different answers from an AI and asking which one is better. The collection of these human votes becomes the "gold standard," a trusted reference point against which all new models are measured. For years, the field has operated on a quiet assumption: that it does not matter which humans are hired to do the judging. The prevailing belief is that while individual people might make random mistakes, the group as a whole is a reliable instrument, measuring a single, stable truth about what makes an answer good. If this assumption holds, then the specific identity of the judges is just a minor detail, like the brand of pen used to write the score.

But a new investigation suggests that this detail is actually the most important part of the equation. Researchers set out to test whether the choice of human judges truly matters by running the same set of AI comparisons twice, using two completely different groups of people. One group consisted of ordinary crowdworkers, the kind of people hired by the thousands for online tasks. The other group was made up of screened domain experts, individuals with specialized knowledge in the field. The researchers took a massive dataset of thousands of AI comparisons and asked both groups to vote on the winners. They found that the two groups of humans did not agree with each other nearly as often as the field assumed. On nearly one in four items, the two groups chose different majority winners. In almost one out of every ten cases, the crowdworkers and the experts picked opposite winners, declaring the loser of one group to be the champion of the other.

Despite this massive disagreement at the level of individual items, the final result looked deceptively stable. When the researchers used these conflicting votes to rank the six AI models in the study, the final leaderboard was identical regardless of which group of humans did the judging. The order of the models did not change at all. This created a confusing picture: the judges were shouting different answers, yet the scoreboard remained silent. The researchers realized that this stability was not a sign of a robust system, but rather a lucky accident caused by the small number of models being compared. Because the models were spaced far apart in their performance, the small shifts in voting caused by changing the judges were not enough to push one model past another.

To test how fragile this stability really was, the researchers ran simulations to see what would happen if the leaderboard were larger, containing the dozens of models that are common in real-world competitions. The results were stark. While a six-model leaderboard remained unchanged, a leaderboard with just ten models would see a model displaced in eighty-six percent of cases. With twenty models, the chance of a reshuffle was nearly one hundred percent. The study concluded that the field's current practice of reporting small leaderboards is safe only because the models are few and far between. If the field were to publish a larger, more realistic arena of models, the choice of human judges would immediately scramble the rankings. The "gold standard" is not a fixed truth; it is a moving target that shifts depending on who is hired to hold the measuring tape.

The investigation did not stop at human judges. The researchers also examined how artificial intelligence judges, which are increasingly used to automate these rankings, handle the disagreement between human groups. They tested three different AI judges, including models from different companies, to see which group of humans they agreed with more. Every single AI judge aligned significantly more with the crowdworkers than with the experts. This suggests that the AI judges have learned to mimic the preferences of the crowd, likely because the data used to train them came from similar crowdsourced workers. The researchers found that an AI judge's agreement with humans is not just a measure of its intelligence, but also a reflection of the hiring decisions made years ago when the training data was collected.

The study also uncovered that the way these AI judges are prompted can change their minds more than the choice of human judges does. When the researchers changed the order in which the answers were presented, the AI judges reversed their decisions on nearly half of the items (specifically 44.6% for one model). Furthermore, when the researchers changed only the output format (asking for a JSON object versus a bare token) while keeping everything else fixed, the same judge agreed with itself on only 47.2% of items, showing a systematic shift in one direction. This sensitivity to presentation order and formatting is a much larger source of error than the difference between expert and crowd opinions. The researchers argue that the entire field needs to change how it reports these results. Simply stating that a model agrees with humans a certain percentage of the time is incomplete without specifying which humans were used. The safety of current rankings is an illusion created by small sample sizes, and the reliability of future rankings depends entirely on acknowledging that the choice of annotators is not a minor detail, but a fundamental variable that shapes the outcome.

Technical Summary: "Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards"

Problem Statement
Preference benchmarks in Large Language Model (LLM) evaluation rely on human annotations to establish a "gold standard." However, the identity of these annotators (e.g., crowdworkers vs. domain experts) is typically treated as an implementation detail, under the implicit assumption that annotators are merely noisy instruments measuring a single underlying truth. This paper challenges that assumption, investigating whether the choice of annotator pool significantly alters item-level labels and, consequently, model rankings. The author notes that existing datasets like MULTIPREF explicitly assume no intra-group variability, an assumption this work seeks to test.

Methodology
The study employs a comparative analysis across two distinct corpora where multiple annotator pools rated the same items:

MULTIPREF: A dataset of 10,461 pairwise comparisons annotated by both crowdworkers and screened domain experts. The analysis focuses on a subset of 2,885 items where both pools were internally unanimous (no tie-breaking conventions required), ensuring a clean comparison of "expert-gold" vs. "crowd-gold" labels.
MT-BENCH: A subset of 246 items rated by both the benchmark's original authors and a recruited pool of experts.

The author measured three levels of divergence:

Item-Level Divergence: The frequency with which pools assign different majority labels or reverse the winner.
Aggregate Invariance: The stability of model rankings (measured via Kendall $\tau$ ) when switching between expert and crowd labels.
Perturbation Analysis: A statistical examination of how switching pools affects model win rates and the probability of model displacement in leaderboards of varying sizes ( $K=6$ to $K=50$ ).
LLM Judge Validation: Three LLM judges (two from one vendor, one from another) were evaluated against both pools to determine if they align more closely with crowd or expert preferences.

Crucially, the author pre-registered decision rules to avoid post-hoc selection bias, including a rule to stop analysis if rankings remained identical ( $\tau > 0.9$ ) to prevent "hunting" for flips.

Key Results

Significant Item-Level Disagreement: Despite the assumption of uniformity, the pools disagree substantially. On the unanimous subset of MULTIPREF, expert and crowd pools assigned different majority labels to 23.6% of items and reversed the winner in 9.2% of cases. Similar results were found in MT-BENCH (30.5% label difference, 8.5% winner reversal).
Aggregate Invariance in Small Leaderboards: Despite the high item-level noise, the resulting model leaderboards for both corpora were bit-identical ( $\tau = 1.00$ ) with zero models displaced when switching pools. This invariance held across multiple tie-breaking conventions.
Fragility of Invariance: The author demonstrates that this stability is an artifact of the small number of models (6) and the specific spacing of their win rates.
- Switching pools shifts model win rates by an average of 1.9 percentage points (pp) (SD).
- While the 6-model leaderboard remained stable, a parametric estimate shows that a 10-model leaderboard would be displaced with 86% probability, and a 20-model leaderboard with 99.97% probability under the same perturbation.
- The observed stability is the "common outcome" for small $K$ , not a property of the aggregation method.
LLM Judge Bias: All three tested LLM judges aligned measurably more with the crowd majority than the expert majority (differences of -6.9pp, -5.2pp, and -3.7pp). This suggests that an LLM judge's reported agreement with "humans" is partially a statement about which human pool was hired.
Judge Sensitivity: The study also found that LLM judges are highly sensitive to output format (JSON vs. bare tokens) and presentation order, with format changes causing self-agreement to drop to ~47%.

Key Contributions

Measurement of Divergence: Quantifies that item-level divergence between expert and crowd pools is large (approx. 9–30% depending on the metric) and not limited to difficult edge cases.
Distinction Between Levels: Establishes a critical distinction between item-level reliability and leaderboard stability. The paper argues that a stable leaderboard does not imply reliable per-item labels; rather, it often implies that the models are spaced far enough apart to absorb the noise.
LLM Judge Alignment: Demonstrates that LLM judges systematically track crowd preferences over expert preferences, challenging the validity of using judges as neutral validators without specifying the reference pool.
Dataset Limitations: Highlights that many existing corpora (e.g., PRISM, HELPSTEER2) discard per-annotator identity, making this type of analysis impossible and potentially masking significant evaluation biases.

Significance and Claims
The paper claims that the field's reliance on small leaderboards (e.g., 6 models) creates a false sense of security regarding evaluation robustness. The observed invariance ( $\tau=1.00$ ) is not evidence that annotator choice is irrelevant; it is evidence that the specific leaderboard configuration happened to be "safe" for that perturbation size.

The author argues that:

Reporting a leaderboard without specifying the annotator pool is under-specified.
Inference of per-item reliability from leaderboard stability is invalid.
Any evaluation mechanism that consumes per-item labels (such as reward model training, active annotation routing, or LLM judge validation) is not safe at any size, as the underlying label noise is substantial.
The "safety" of current small leaderboards does not generalize to the larger arenas the field actually publishes.

The paper concludes that the choice of annotator pool is a fundamental design decision that directly impacts measured model competence, framing this as a question of human-AI coevolution rather than mere dataset hygiene.

Technical Summary: "Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards"

More like this