🤖 AI

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

The paper introduces STABLEVAL, a disagreement-aware evaluation framework that models latent item correctness and annotator-specific confusion patterns to produce stable, uncertainty-aware system rankings, addressing the fragility and bias inherent in traditional majority vote aggregation methods.

Original authors: Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge which of six different chefs makes the best soup. You ask 65 food critics to taste the soups and rate them.

The Old Way (Majority Vote):
In the past, researchers would just count the votes. If 33 critics say "Chef A's soup is good" and 32 say "Chef A's soup is bad," Chef A gets a "Good" rating. The 32 "bad" votes are ignored.

The Problem: This is fragile. If just one critic changes their mind or if you happen to pick a different group of 65 critics for the next round, the winner might flip-flop. It treats all critics as if they are equally perfect, even if some are known to be harsh, some are too nice, and some just guess randomly. It throws away the nuance of why people disagreed.

The New Way (STABLEVAL):
The authors of this paper, STABLEVAL, say: "Don't just count the votes; understand the voters."

They propose a system that acts like a smart detective rather than a simple vote-counter. Here is how it works, using a few analogies:

1. The "Confusion Matrix" (The Detective's Profile)

Instead of assuming every critic is perfect, STABLEVAL builds a profile for each one.

The Strict Critic: Maybe Critic #5 always thinks the soup is too salty, even when it's fine. STABLEVAL learns, "Ah, Critic #5 is a harsh judge; I'll weigh their 'bad' vote less heavily."
The Lazy Critic: Maybe Critic #10 just guesses "Good" for everything. STABLEVAL learns, "Critic #10 isn't paying attention; I'll ignore their opinion."
The Confused Item: Sometimes a soup is just weirdly ambiguous. STABLEVAL realizes, "This specific bowl of soup is hard to judge," and doesn't force a single "Good" or "Bad" label. Instead, it says, "There's a 60% chance this is good and a 40% chance it's bad."

2. The "Soft Score" vs. The "Hard Label"

Old Way (Hard Label): The soup is either "Good" (1) or "Bad" (0). It's a binary switch.
STABLEVAL (Soft Score): The soup gets a "credit score" of 0.65. This acknowledges that while the soup might be leaning toward "good," there is still some uncertainty. It keeps the uncertainty alive instead of smashing it into a single number.

3. The "Stability Test" (The Shuffle)

The paper introduces a new way to measure success called Ranking Stability.
Imagine you have a deck of cards representing your critics.

Majority Vote: If you shuffle the deck and remove just a few cards (critics), the order of the chefs might completely change. The ranking is unstable, like a house of cards.
STABLEVAL: Because it understands the reliability of each critic, if you shuffle the deck and remove a few cards, the order of the chefs stays the same. The ranking is stable, like a solid stone statue.

What They Found

The researchers tested this on real-world data (like judging AI chatbots and medical summaries) and made-up scenarios with "troublemaker" critics.

The "Denoising" Trap: They compared STABLEVAL to an old method called "Dawid-Skene." That old method is great at guessing the true answer (like a detective solving a crime). However, the paper found that just knowing the "true answer" doesn't always mean you get a stable ranking of the chefs. You can know the truth but still have a ranking that flips around if you change the group of judges slightly.
The Winner: STABLEVAL didn't always guess the "true" label perfectly, but it produced much more consistent rankings. When the critics disagreed a lot (which happens often with complex tasks like AI safety or medical summaries), STABLEVAL kept the leaderboard steady, while the old methods made the rankings jump around wildly.

The Bottom Line

The paper argues that in AI evaluation, disagreement isn't just noise to be deleted; it's a signal to be understood.

If you want to know which AI is truly better, you shouldn't just take a majority vote. You should build a system that knows which judges are reliable, which items are tricky, and how much uncertainty exists. This ensures that when you say "AI Model A is better than AI Model B," you aren't just saying it because you happened to pick a specific group of judges that day. You are saying it because the result is stable, no matter who you ask.

Technical Summary: STABLEVAL

1. Problem Statement

Human evaluation remains the primary standard for assessing modern AI systems, yet it is plagued by annotator disagreement, bias, and variability. Current evaluation pipelines predominantly rely on majority vote (MV) aggregation, which collapses multi-annotator judgments into a single hard label. The authors argue that this approach treats disagreement as noise to be eliminated, thereby discarding critical information regarding annotator reliability and item-level ambiguity.

Consequently, system rankings derived from majority vote are often fragile: small changes in the composition of the annotator pool can lead to significant shifts in agent scores and rankings. This instability undermines the robustness and reproducibility of AI evaluation, particularly in settings with heterogeneous annotator behavior or adversarial noise. The central challenge identified is not merely recovering a denoised ground truth label, but producing stable and uncertainty-aware system comparisons that remain consistent under realistic annotator variability.

2. Methodology: The STABLEVAL Framework

The paper introduces STABLEVAL, a disagreement-aware evaluation framework designed to model latent item correctness and annotator-specific confusion patterns. Unlike classical label-denoising approaches (e.g., Dawid–Skene) that aim to infer a single latent label per item, STABLEVAL is explicitly optimized for stable system evaluation.

2.1 Probabilistic Modeling

The framework models the evaluation problem using a latent variable model:

Latent Correctness ( $z_i$ ): Each item $i$ has an unobserved correctness level $z_i \in \{0, \dots, K-1\}$ representing graded quality.
Annotator Behavior: Annotators are not assumed to be identical. Each annotator $r$ is modeled via a confusion matrix $\pi_r[c, o] = P(y_{ir} = o | z_i = c)$ , which captures the probability of reporting label $o$ given true correctness $c$ . This allows the model to distinguish between item difficulty and annotator-specific behaviors (e.g., strictness, leniency, or systematic noise).
Priors: Dirichlet priors are placed over class priors ( $\mu$ ) and confusion matrices to enable Bayesian estimation and prevent degenerate solutions in low-annotation regimes.

2.2 Posterior Inference and Scoring

STABLEVAL utilizes an Expectation–Maximization (EM) algorithm to estimate latent correctness and annotator reliability parameters.

E-Step: Computes the posterior probability $\gamma_{ic}$ that item $i$ belongs to correctness level $c$ , conditioned on all observed annotations and estimated reliabilities.
Posterior Expected Item Credit: Instead of collapsing $\gamma_{ic}$ into a hard label (e.g., via $\arg\max$ ), STABLEVAL retains the full posterior distribution. The credit for an item is defined as the expected correctness value:
$\text{credit}(i) = \sum_{c=0}^{K-1} \gamma_{ic} v(c)$
where $v(c)$ is a task-specific scoring function. This formulation assigns partial credit proportional to posterior belief, preserving graded correctness and item-level ambiguity.
Agent Scoring: An agent's score is the mean posterior item credit over its evaluated items. Uncertainty is quantified via bootstrap resampling to generate confidence intervals.

2.3 Formalizing Ranking Stability

The paper formalizes ranking stability as a first-class objective. A method is stable if agent orderings remain consistent under random subsampling of annotators.

Instability of Majority Vote: The authors prove (Proposition 1) that majority vote is inherently unstable when an item's label depends on a single-vote margin; removing the decisive annotator can flip the label, causing ranking volatility.
Asymptotic Stability of STABLEVAL: The authors demonstrate (Proposition 2) that under correct model specification, STABLEVAL scores converge to stable values as the number of items increases. Because STABLEVAL uses smooth, reliability-weighted aggregation rather than discrete thresholding, it mitigates ranking discontinuities caused by small annotator perturbations.

3. Key Contributions

The paper outlines four primary contributions:

STABLEVAL Framework: Introduction of a disagreement-aware framework that produces calibrated posterior item credit instead of hard labels, enabling graded and uncertainty-aware scoring.
Formalization of Stability: Definition of ranking stability as a formal evaluation criterion and theoretical analysis demonstrating the inherent instability of majority vote under annotator subsampling.
Distinction of Objectives: Clarification that label denoising (recovering ground truth) and evaluation stability are distinct goals; recovering latent labels does not guarantee robust system rankings.
Empirical Validation: Extensive validation across synthetic stress tests and real-world benchmarks showing that STABLEVAL yields more stable and statistically grounded rankings under annotator heterogeneity and adversarial noise compared to majority vote and Dawid–Skene.

4. Experimental Results

The authors evaluated STABLEVAL against Majority Vote (MV) and Dawid–Skene (DS) across synthetic configurations and four real-world datasets (MT-Bench, ConvAbuse, QAGS, MSLR).

4.1 Synthetic Stress Tests

Label Recovery vs. Stability: DS achieved the lowest Mean Squared Error (MSE) regarding latent ground truth, consistent with its denoising objective. However, low MSE did not guarantee stable rankings.
Heterogeneity and Noise: Under adversarial conditions (up to 40% adversarial annotators) and heterogeneous strict/lenient populations, MV exhibited increasing score error and ranking instability. Posterior Expected Credit (PEC) consistently outperformed MV in both error reduction and ranking consistency (Kendall's $\tau$ ).
Hard Items: In regimes with tight agent quality gaps or "hard" items, discrete MV thresholding caused sharper ranking sensitivity. PEC maintained near-perfect ranking stability (Kendall's $\tau \approx 1.0$ ) in scenarios where MV degraded.

4.2 Real-World Benchmarks

High-Disagreement Regimes (MT-Bench): With only 16.6% unanimous agreement, MT-Bench showed substantial instability for MV. PEC produced the most stable rankings under annotator subsampling, while DS was intermediate. Annotator diagnostics revealed significant variation in reliability, which STABLEVAL successfully modeled.
High-Consensus Regimes (ConvAbuse, QAGS): In datasets with high agreement (e.g., 78.4% unanimous in ConvAbuse), all methods converged to identical rankings and perfect stability. This indicates STABLEVAL does not introduce instability when disagreement is minimal.
Limited Annotator Regimes (MSLR): In small-annotator settings, all methods showed increased variability. While MV and DS showed slightly lower variability than PEC in this specific constrained case, PEC provided richer item-level ambiguity estimates.

5. Significance and Claims

The paper claims that modeling disagreement is essential for robust and reproducible AI evaluation, particularly when annotator variability is substantial.

Reframing Evaluation: The work reframes stability as a first-class objective, distinct from label recovery. It argues that collapsing uncertainty into hard labels discards signal necessary for stable comparisons.
Contextual Value: The authors modestly note that STABLEVAL's value is most pronounced in high-disagreement regimes (e.g., subjective tasks like toxicity detection or complex reasoning). In high-consensus settings, simpler methods like majority vote may suffice, and STABLEVAL does not claim universal superiority in all contexts.
Practical Implications: By preserving posterior uncertainty and modeling annotator behavior, STABLEVAL provides a principled foundation for evaluation pipelines that are less sensitive to the specific composition of the annotator pool. This is critical for high-stakes domains such as safety assessment, content moderation, and medical benchmarking, where fragile rankings could lead to suboptimal model selection.

The authors conclude that as AI evaluation increasingly relies on human or model-based judges, moving beyond majority vote to explicitly model disagreement and uncertainty is necessary for achieving robust and reproducible system comparisons.