🤖 machine learning

Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

This paper addresses the reproducibility crisis in generative AI evaluation by introducing a multi-level bootstrapping approach that leverages large-scale datasets with persistent rater identifiers to model annotator variance and determine the optimal balance between the number of items and responses per item required for statistical significance.

Original authors: Deepak Pandita, Flip Korn, Chris Welty, Christopher M. Homan

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Deepak Pandita, Flip Korn, Chris Welty, Christopher M. Homan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge which of two new AI chatbots is "safer" and more helpful. You ask a group of people to rate them. But here's the problem: if you only ask three people to rate ten different conversations, you might get a lucky (or unlucky) result that doesn't reflect reality. This is the "reproducibility crisis" the paper talks about—science is struggling to get consistent results because we aren't measuring things carefully enough.

The authors of this paper argue that we are making a big mistake by treating every human opinion as if it came from a completely different person who has no connection to the others. In reality, the same person often rates multiple items, and they carry their own personal "flavor" or bias with them.

Here is a breakdown of their findings using simple analogies:

1. The "Taste Test" Problem

Imagine you are hosting a massive taste test for two new flavors of ice cream (Model A and Model B).

The Old Way (What most people do): You ask 100 different people to taste one scoop each. You assume everyone's opinion is independent.
The Reality: In many studies, you actually ask just 5 people to taste 20 scoops each.
The Issue: If "Bob" happens to hate chocolate, he will rate every chocolate scoop poorly, regardless of the brand. If you treat Bob's 20 opinions as 20 independent data points, you trick yourself into thinking you have a huge amount of evidence. You are actually just listening to Bob's voice 20 times.

The paper shows that when you ignore the fact that the same people are rating multiple items, you get false confidence. You think you have found a clear winner between the two AI models when, in reality, you just haven't asked enough different people to get a true average.

2. The "Multi-Level" Solution

The authors propose a new way of looking at the data, which they call Multi-Level Bootstrapping.

The Analogy: Instead of just counting votes, imagine you are a detective who realizes that "Bob" is a grumpy man who always gives low scores. You need to account for Bob's grumpiness when you calculate the final score.
How they did it: They took real-world datasets where they knew exactly who rated what (like a dataset of 350 conversations rated by 123 people). They used a computer simulation to "resample" the data, effectively asking: "What if we had asked different people? What if we asked the same people again?"

3. The Big Discovery: More People, Not Just More Items

The paper answers a critical question: Should we ask more people to rate fewer items, or fewer people to rate more items?

The Finding: It is much better to have more people (raters) rating fewer items than to have a few people rating everything.
The Metaphor: Imagine trying to guess the average height of a city.
- Strategy A: Ask 1 person to measure 1,000 different buildings. (This is bad; if that person is short-sighted or uses a broken tape measure, your whole city's data is wrong).
- Strategy B: Ask 1,000 different people to measure 1 building each. (This is better; even if one person makes a mistake, the average of 1,000 people will be accurate).

The paper found that to get a statistically reliable result (a "significant" finding), you often need to increase the number of people (K) significantly, sometimes up to 100 people per item, rather than just adding more items to the list.

4. The "Batch" Effect

The researchers also looked at how data is collected in "batches" (groups).

The Analogy: Imagine a classroom where the teacher hands out a test to the whole class at once. The students might chat and influence each other, or they might all be tired at the same time.
The Finding: When people rate items in groups (batches), their opinions become even more correlated. The paper shows that if you ignore these "batches," you underestimate how much data you actually need. You might think you need 500 ratings to be sure, but because of the "batch" effect, you might actually need 2,500.

5. The Bottom Line

The paper concludes that the current standard of using only 3 to 5 raters per item is often not enough to trust the results, especially when trying to prove that one AI model is safer than another.

The Recommendation: If you want to know if an AI is truly safe or better, you need to stop treating human raters as interchangeable coins. You need to acknowledge that every human has a unique "voice." To get a true answer, you need to hire more humans to give their opinions, even if it costs more money.

In short: Don't just ask the same few people to judge a thousand things. Ask a thousand different people to judge a few things. It's the only way to stop fooling yourself with fake statistics.

Technical Summary: Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

Problem Statement
The rapid proliferation of generative AI (GenAI) has highlighted a critical reproducibility crisis in machine learning evaluation. Current evaluation protocols often rely on benchmarks where each input is paired with a single "gold-standard" output, a paradigm that fails to account for the inherent stochasticity of inference and the subjective variance introduced by human annotators. Standard practices typically aggregate 3 to 5 annotations per item into a single label, effectively treating rater disagreement as noise rather than a signal of subjective complexity. Furthermore, existing statistical approaches often assume that annotations for different items are independent, ignoring the reality that the same annotators frequently rate multiple items. This failure to model the hierarchical dependency between raters and items leads to an overestimation of statistical power, resulting in underestimated p-values and potentially unreliable conclusions regarding model performance.

Methodology
To address these limitations, the authors propose a multi-level bootstrapping framework designed to realistically model annotator behavior and variance. The study utilizes three real-world datasets containing persistent rater identifiers and multiple annotations per item: DICES 350 (fully crossed matrix), the Stanford Toxicity dataset (batched annotations), and D3code (cross-cultural annotations).

The core methodological innovation involves three distinct non-parametric bootstrap sampling strategies (S1, S2, S3) implemented within a simulation framework based on the Variance Estimation Toolkit (VET):

S1 (Sample Items and Raters): Assumes a fully crossed design where items and raters are sampled independently with replacement. This approach ignores the dependency of raters across items.
S2 (Sample Items and Raters Within an Item): Samples items with replacement, then samples raters specifically from the pool of raters who annotated that specific item. This models the reality that raters annotate subsets of items but does not enforce batch structures.
S3 (Sample Stratified Batches): Samples batches of items annotated by the same set of raters, then resamples items and raters within those batches. This captures batch-specific rater behavior.

The simulation framework generates data for a gold standard (G) and two models (A and B), where Model B is perturbed by a small amount ( $\epsilon$ ) to simulate a performance difference. Null Hypothesis Significance Tests (NHSTs) are then applied to compare Model A and Model B under these different sampling regimes to observe how p-values and required sample sizes (N items, K raters) shift.

Key Contributions

Multi-Level Modeling: The paper introduces a framework that explicitly accounts for the hierarchical structure of rater-item interactions, moving beyond the assumption of conditional independence between items.
Empirical Trade-off Analysis: The study systematically analyzes the trade-offs between the number of items ( $N$ ) and the number of responses per item ( $K$ ) required to achieve statistical significance under different sampling assumptions.
Metric Sensitivity: The authors evaluate eight distinct metrics (including Accuracy, MAE, Wins, Precision, Recall, F1-Score, KL-Divergence, and Jensen-Shannon Distance) to determine how different evaluation measures respond to rater variance modeling.

Results
The experimental results demonstrate that failing to account for rater behavior across items leads to substantial underestimates of p-values, thereby inflating the perceived statistical power of a dataset.

Impact of Modeling: Comparing S1 (independent sampling) to S2 (item-specific rater sampling), S2 consistently yields lower p-values and requires higher budgets to achieve significance. For instance, on the DICES dataset with the Accuracy metric, the p-value dropped from 0.045 (S1) to 0.019 (S2) at a fixed budget of $N \times K = 5000$ .
Budget Requirements: Accounting for rater variance increases the total annotation budget ( $N \times K$ ) required to obtain reliable null hypothesis significance tests. For example, on the Toxicity dataset, achieving significance for Accuracy required a budget of 500 under S2, but this increased fivefold to 2500 under the stratified batch approach (S3).
Optimal K vs. N: The study finds that increasing the number of responses per item ( $K$ ) is often more critical for achieving significance than increasing the number of items ( $N$ ), particularly for distribution-sensitive metrics like MAE and JSD. In several cases, optimal $K$ values shifted from 40 to 100 when moving from S1 to S2 modeling.
Downsampling Effects: Experiments downsampling the DICES dataset to 5 raters showed that small rater pools lead to separation in p-value distributions across budgets, suggesting that limited rater diversity significantly impacts estimation stability.

Significance and Claims
The paper argues that the current reliance on small annotation counts (3–5 per item) and the aggregation of labels into single "gold standards" obscures the true variance in human judgment. By demonstrating that accounting for rater behavior results in higher p-value estimates, the authors claim that many previous evaluations may have been statistically underpowered or falsely significant.

The authors conclude that to ensure the safety and trustworthiness of GenAI systems, evaluation datasets must move beyond "plurality-vote architectures." They recommend that dataset creators increase the number of annotators responding to each input (higher $K$ ) and provide disaggregated, annotator-level data. While acknowledging that this increases evaluation costs, the paper posits that this investment is necessary to capture the full distribution of human responses and to avoid marginalizing minority viewpoints. The work provides a strategic roadmap for optimizing annotation budgets ( $N \times K$ ) to achieve rigorous, repeatable, and statistically sound AI evaluations.

1. The "Taste Test" Problem

2. The "Multi-Level" Solution

3. The Big Discovery: More People, Not Just More Items

4. The "Batch" Effect

5. The Bottom Line

More like this