Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment

Technical Summary: Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment

Problem Statement
Hallucination, defined as the generation of information inconsistent with reality or unsupported by provided evidence, remains a critical challenge for Large Language Models (LLMs), particularly in Retrieval-Augmented Generation (RAG) and agentic systems. While existing literature relies on standardized benchmarks (e.g., QAGS-C, SummEval) with human-annotated ground truth to evaluate factual consistency, this study questions the adequacy of these fixed labels. Recent evidence suggests that human annotations often exhibit systematic variation and that frontier LLMs may outperform current benchmarks in detecting subtle inconsistencies. The core problem addressed is whether single-pass human annotations are sufficient for evaluating state-of-the-art LLMs, or if the reliance on potentially noisy or ambiguous benchmark labels leads to an underestimation of model performance.

Methodology
The authors propose a disagreement-aware, adjudication-based evaluation framework to re-evaluate hallucination detection in abstractive summarization tasks. The methodology proceeds in three main stages:

1. LLM Inference: Two state-of-the-art models, GPT-5 Mini and Gemini 2.5 Flash, were tasked with evaluating document-summary pairs from the SummEval and QAGS-C datasets. Unlike standard binary classification, these models were prompted to provide structured JSON outputs containing a detailed reasoning justification and a specific text span indicating the location of any hallucination (or "none" if consistent).
2. Disagreement Identification: The study identified a subset of samples where the original dataset labels (derived from expert/crowd annotations) conflicted with the predictions of the LLMs. Specifically, cases where the two LLMs agreed with each other but disagreed with the human label, or where the LLMs and human labels were in total disagreement, were flagged.
3. Human Adjudication: A panel of two human adjudicators (cross-cultural, blinded to each other's decisions) re-evaluated the conflicted samples. Crucially, these adjudicators were provided with the original dataset label alongside the LLM-generated reasons and span-based evidence. They were instructed to make a final determination based on all available information, effectively acting as a "judge" between the original human annotation and the LLM's evidence-backed claim. In cases of persistent disagreement between adjudicators, a majority voting scheme involving the original label, both LLM predictions, and both adjudicator labels was used to resolve the final ground truth.

Key Results
The re-evaluation process yielded significant shifts in performance metrics and agreement rates:

• Increased Agreement: The "triple agreement" rate (where both LLMs and the adjudicated human label align) increased substantially. For QAGS-C, triple agreement rose by 6.38% (from 82.55% to 88.93%), and for SummEval, it increased by approximately 7.62% (from 84.75% to ~92.37%).
• Improved Model Accuracy: When evaluated against the newly adjudicated ground truth, both LLMs showed improved accuracy. GPT-5 Mini accuracy increased by 4.25% on QAGS-C and 2.34% on SummEval. Gemini 2.5 Flash showed larger gains, improving by 8.51% on QAGS-C and 3.80% on SummEval.
• Adjudicator Preference for LLM Evidence: In conflict cases, human adjudicators frequently sided with the LLMs, particularly when the models provided explicit reasoning and span-based evidence. In QAGS-C conflict cases, adjudicators endorsed LLM hallucination judgments 82.93% of the time; in SummEval, this was 73.00%.
• Revised Hallucination Rates: The adjudication process revealed that the original benchmarks underestimated hallucination prevalence. The hallucination rate in SummEval increased from 18.4% to 20.5%, and in QAGS-C, it rose from 51.9% to 56.17%.

Significance and Claims
The paper argues that for ambiguity-prone tasks like hallucination detection, single-pass annotations may be insufficient and can lead to the systematic underestimation of LLM capabilities. The study claims that:

1. Benchmark Limitations: Fixed benchmark-based evaluations are limited because they treat human annotations as static oracles, ignoring the systematic nature of annotation disagreement and the potential for LLMs to detect inconsistencies missed by humans.
2. Value of Adjudication: An adjudication-based protocol that incorporates LLM-generated reasoning and span-level evidence yields more reliable and semantically grounded benchmarks. The presence of explicit evidence (reasons and spans) significantly influences human judgment, suggesting that richer annotation schemes could reduce ambiguity.
3. Generalizability: While the experiments focused on summarization, the authors posit that this phenomenon likely extends to other domains such as question answering, RAG, and agentic AI, where errors propagate through complex reasoning steps.
4. Future Direction: The findings advocate for a shift in evaluation protocols toward iterative or model-assisted re-evaluation, moving beyond binary correctness to consider localization quality and the utility of span-level outputs. The authors suggest that future work should focus on developing fine-grained datasets that include both LLM and human-generated annotations with explicit reasoning.

The paper concludes that treating a single annotation pass as definitive is misleading for nuanced natural language tasks and that integrating disagreement analysis and LLM-assisted adjudication offers a more robust alternative for evaluating factual consistency.