Multi-LLM Disagreement as a Scalable Detector of Human Annotation Errors in Structured Data from Clinical Free-Text

This study demonstrates that disagreement among multiple locally hosted large language models serves as a highly accurate, scalable, and GDPR-compliant signal to prioritize human review of clinical annotation errors, effectively identifying the small subset of low-agreement cases that contain the majority of mistakes.

The Gist

Imagine you are running a massive library where thousands of books (medical reports) need to be cataloged. You hire a team of student assistants to read each book and fill out a simple card with five key facts: where a specific item was found, how big it was, how it was removed, and so on.

Because there are so many books and the work is repetitive, the students sometimes make mistakes. They might misread a number, skip a detail, or get confused by messy handwriting. Checking every single card manually would take forever and cost a fortune.

This paper proposes a clever, automated way to spot the cards that are most likely to be wrong, so you only have to check the ones that matter.

The "Committee of Experts" Analogy

Instead of just trusting the student assistant, the researchers brought in four different "AI experts" (Large Language Models) to read the same books and fill out the same cards. These AI experts are like four different specialists who have read millions of medical reports.

Here is the core idea: If the student and all four AI experts agree on the answer, it's probably right. But if the student says "Red" and the four AI experts all say "Blue," something is likely wrong.

The researchers didn't just look at one AI; they looked at the disagreement between the four AIs and the human student. They created a "Disagreement Score":

• Score 4: All four AIs agree with the human. (Safe to ignore).
• Score 0: None of the AIs agree with the human. (Highly suspicious!).

The "Needle in a Haystack" Discovery

The most exciting finding is that you don't need to check the whole haystack.

• The researchers found that the "low agreement" cases (where the AIs and the human disagreed) made up only 6.5% of the total work.
• However, this tiny slice contained about 80% of all the actual mistakes.

It's like having a metal detector that only beeps when you are standing on a pile of gold coins, ignoring the thousands of empty spots in the sand. By focusing their human review only on that small 6.5% where the AIs and the human disagreed, they could catch almost all the errors without doing the heavy lifting of checking everything.

The Results in Plain English

• Accuracy: When the AIs and the human disagreed, the human was wrong about 76% of the time. When they all agreed, the human was almost never wrong.
• Efficiency: Using this "Disagreement Score" allowed them to filter out the safe cases and zoom in on the risky ones. The system was incredibly good at predicting errors, with a score of 0.99 out of 1.0 (where 1.0 is perfect).
• Privacy: All these AI experts ran on the hospital's own computers (locally), not on the public internet. This means patient data never left the building, keeping it safe and private.
• Language: The study was done on German medical reports. This proves the method works even when the language is different from English, which is where most AI research usually happens.

Why This Matters

Traditionally, to ensure quality, you might have to double-check every single card (which is slow) or just randomly pick a few to check (which might miss the bad ones).

This paper suggests a smarter approach: Let the AI committee argue with the human. If they all agree, move on. If they fight, send that specific case to an experienced expert for a final look. This saves time, saves money, and ensures the data used for medical research is much cleaner and more reliable.

In short, the paper shows that using a group of AI models to "vibe check" human work is a powerful, scalable, and privacy-safe way to catch mistakes before they become a problem.

Technical Summary

Technical Summary: Multi-LLM Disagreement as a Scalable Detector of Human Annotation Errors in Structured Data from Clinical Free-Text

Problem Statement
The generation of high-quality structured datasets from clinical free-text is fundamental to medical research and machine learning but relies heavily on human annotators. This process is prone to two primary error sources: repetitive fatigue leading to lapses or oversights, and the delegation of tasks to less experienced annotators (e.g., medical students) who may lack specific clinical context. Traditional quality control methods, such as double annotation or exhaustive expert review, are often impractical at scale due to the high workload and cost. While Large Language Models (LLMs) have been explored as replacements for human annotators, prior work has largely focused on comparing single-model accuracy against human labels or using LLMs to correct human errors. This paper addresses the gap in scalable, privacy-compliant quality control mechanisms that can prioritize human review for the highest-risk annotations without requiring cloud-based inference or replacing the human annotator entirely.

Methodology
The study utilized a retrospective dataset of 910 German-language colonoscopy reports from University Hospital Mannheim (2010–2014), describing endoscopic mucosal resections. The workflow involved the following steps:

1. Human Annotation: A medical student extracted five structured variables for the largest polyp in each report: anatomical location, primary diameter, secondary diameter, resection technique (en-bloc vs. piecemeal), and the presence of multiple polyps.
2. Multi-LLM Extraction: Four locally hosted, open-weight LLMs (Gemma 3 27B, DeepSeek-R1 70B, GPT-OSS 120B, and Mistral Large 3) independently extracted the same five variables from the reports. All models were deployed on institutional GPU servers to ensure GDPR compliance and data privacy.
3. Agreement Scoring: For each annotation, an agreement score (0–4) was calculated based on the number of LLMs that matched the human label.
4. Stratified Sampling and Adjudication: To evaluate the predictive value of disagreement, a stratified sample was drawn, oversampling low-agreement strata (scores 0–2) and undersampling high-agreement strata (scores 3–4). A blinded, experienced medical reviewer adjudicated these cases by comparing the human annotation against the output of a single reference LLM (Gemma 3 27B, chosen for its high accuracy on the prompt development set).
5. Statistical Analysis: Error rates were calculated under two definitions: "strict" (unambiguously incorrect) and "broad" (including indeterminate cases). Prevalence-adjusted Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves were generated to assess the ability of the multi-LLM disagreement score to detect human errors.

Key Results

• Error Concentration: Human error rates were strongly inversely correlated with LLM agreement. At an agreement score of 0 (no LLMs matched the human), the human error rate was 76.3%. Conversely, at scores of 3 and 4, human error rates dropped to 0%.
• Efficiency of Targeted Review: The lowest-agreement stratum (score 0) constituted only 6.5% of the total 4,550 annotations but concentrated an estimated 80% of all errors.
• Predictive Performance: The multi-LLM disagreement score achieved a prevalence-adjusted AUC-ROC of 0.991 (95% CI 0.987–0.994) and an AUC-PR of 0.893 (95% CI 0.851–0.929) for detecting human errors under the broad definition. This significantly outperformed a single-model binary classifier (Gemma 3 27B alone), which achieved an AUC-ROC of 0.944.
• Model Scaling: Experiments with smaller model variants (e.g., Gemma 3 1B to 27B) indicated that match rates increased substantially with model size, particularly between 1B and 4B parameters, with diminishing returns observed beyond 12B–14B parameters.
• Error Types: Qualitative analysis of LLM failures identified four primary error modes: incorrect size identification (e.g., misinterpreting ranges), misclassifying re-resections as new polyps, failing to detect polyps that were resected but not retrieved, and incorrect location identification.

Significance and Claims
The paper claims that multi-LLM disagreement serves as a robust, scalable, and privacy-compliant signal for prioritizing human annotation review. By shifting from uniform quality control to risk-stratified review, the framework allows researchers to focus expert effort on the small subset of cases (approx. 6.5%) where errors are most likely, while safely accepting the majority of high-agreement annotations.

Key contributions highlighted by the authors include:

1. Real-World Validation: Moving beyond curated benchmarks to real-world, German-language electronic health records, demonstrating the method's utility in a complex, non-English clinical setting.
2. Privacy Compliance: Demonstrating that locally hosted, open-weight models are sufficient for effective error detection, avoiding the need for cloud-based APIs and ensuring patient data remains within institutional infrastructure.
3. Graded Operating Points: Unlike binary single-model checks, the multi-LLM agreement score provides a graded signal (0–4), allowing institutions to tune the sensitivity and positive predictive value of their review processes based on available resources.
4. Generalizability: The authors posit that the framework is language- and task-agnostic, making it applicable to various clinical domains beyond gastroenterology, provided multiple independent LLMs can be deployed.

The study concludes that this approach enables the creation of large, high-quality clinical datasets with substantially reduced review effort, offering a practical solution to the bottleneck of manual data abstraction in medical research.