💬 NLP

Consensus Measures for Unstructured Biomedical Text Annotations

This paper investigates methods for quantifying inter-rater reliability in unstructured biomedical text annotations, demonstrating that while semantic equivalence measures like embeddings and large language models have distinct limitations, natural language inference offers a promising compromise for assessing agreement.

Original authors: Pascal Wullschleger, Christian Kreis, Martin A. Walter, Marc Pouly, Jennifer Foster

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pascal Wullschleger, Christian Kreis, Martin A. Walter, Marc Pouly, Jennifer Foster

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery by reading thousands of old, handwritten letters. In the world of medicine, scientists do something similar: they read millions of research papers to find hidden clues about diseases, drugs, and treatments. But here's the tricky part. Sometimes, the clues they are looking for aren't listed in a checklist. Instead, they are hidden in free-flowing sentences, like a doctor writing "the patient felt exhausted and weak" instead of checking a box that says "fatigue."

To make sure these clues are real and not just a fluke, scientists often get two or more people to read the same letters and write down what they find. This is called "annotating." If both people write down the exact same words, it's easy to say they agree. But what if one writes "exhausted" and the other writes "tired"? Are they the same? In the old days, computers would say "no, those are different words," and the agreement score would be low. But in the real world of medicine, "exhausted" and "tired" mean the same thing. This paper is about teaching computers to understand that subtle difference, so we can trust the clues we find in medical literature.

The researchers in this paper, led by Pascal Wullschleger and his team, set out to solve a specific headache: how do we measure how well two people agree when they are writing free-form notes instead of picking from a list? They call this "soft inter-rater reliability." Think of it like grading a creative writing assignment. If you ask two teachers to grade an essay, they might use different words to describe the same good point. A "hard" grading system would fail them for not using the exact same words. A "soft" system understands that "brilliant" and "excellent" are close enough to count as agreement.

The team didn't just guess how to fix this; they built a giant digital playground to test it. They created thousands of fake medical scenarios where they knew the "right" answer in advance. They then pitted different computer tools against each other to see which one could best tell if two different descriptions meant the same thing. Some tools were like spell-checkers, counting how many letters were different (edit distance). Others were like smart dictionaries that looked at the meaning of words (embeddings). And the newest tools were like giant, super-smart AI brains (Large Language Models) that could read a sentence and decide if it meant the same thing as another.

Here is what they found, and it's a bit of a plot twist. The "spell-checker" tools were okay for simple things but failed miserably when the words were tricky. The "smart dictionary" tools were fast and good at spotting general similarities, but they had a blind spot: they couldn't tell the difference between two things that were similar but not the same. For example, they might think "a headache" and "a migraine" are the same because the words look alike, even though a migraine is a specific, stronger type of headache. This is a big problem in medicine where precision matters.

The giant AI brains were the most accurate at understanding the true meaning, but they had a different problem: they were too slow and expensive to run the massive number of tests needed to prove they were working correctly. It's like having a genius detective who solves the case perfectly but takes a month to do it, while you need an answer in a second.

So, what is the winner? The paper suggests that the best solution is a middle-ground tool based on "Natural Language Inference" (NLI). Think of NLI as a logic puzzle solver. Instead of just counting letters or guessing similarity, it asks: "If this sentence is true, does that sentence have to be true?" It's like asking, "If I say 'I have a red car,' does that mean 'I have a car'?" Yes. But if I say "I have a car," does that mean "I have a red car"? No. This logic-based approach was the most reliable in their tests. It was fast enough to be practical but smart enough to avoid the mistakes the other tools made.

The researchers also showed that this method works not just for single words, but for lists of notes. Imagine one doctor writes a list of three symptoms and another writes a list of four. The paper figured out how to match them up perfectly, like pairing socks from two different piles to see how many match, even if the piles aren't the same size.

In the end, this paper doesn't claim to have solved every problem in medical research. It suggests that for now, using these logic-based AI tools is the smartest way to measure agreement on free-text medical notes. It's a step forward in making sure that when we dig through the mountain of medical literature to find new cures, we aren't just counting words, but actually understanding the meaning behind them.

Technical Summary: Consensus Measures for Unstructured Biomedical Text Annotations

Problem Statement

Biomedical literature is increasingly repurposed for knowledge extraction beyond the original scope of the studies, such as identifying molecular targets or treatment toxicities. In these exploratory settings, target concepts are not known a priori, leading annotators to prefer open-ended, unstructured text labels over predefined categories. While Inter-Rater Reliability (IRR) is standard for assessing the quality of codified annotations using discrete categories (e.g., Cohen's $\kappa$ ), conventional measures fail in open-ended scenarios. Traditional IRR relies on identity (exact string matching), which is inappropriate when human experts or automated systems describe the same underlying biomedical concept using different terms (e.g., "Fatigue" vs. "Weakness" vs. "Asthenia"). This creates a methodological gap: there is no standardized, systematic way to quantify agreement on unstructured text annotations where semantic equivalence, rather than lexical identity, defines consensus.

Methodology

The authors propose and evaluate "Soft IRR," a framework that replaces binary agreement with a continuous semantic equivalence measure $\hat{P}(a \equiv b)$ . The study investigates two scenarios:

Single-Text IRR: Comparing two individual text annotations directly.
Set-Matching IRR: Comparing two sets of annotations (e.g., multiple concepts extracted from a single document), requiring an assignment problem solution to match elements between sets optimally.

Equivalence Estimators

The paper evaluates a wide range of semantic equivalence measures:

Set-Overlap: Jaccard coefficient and Measuring Agreement on Set-valued Items (MASI).
Edit-Distance: Levenshtein, Indel, and Longest Common Subsequence (LCS) distances.
Semantic Similarity: Cosine similarity using sentence embeddings (e.g., PubMedBERT, Qwen3-0.6B).
Natural Language Inference (NLI): Mutual entailment models (e.g., mDeBERTa-v3, PubMedBERT fine-tuned on MedNLI) that determine if one text entails another.
LLM-as-a-Judge: Using Large Language Models (e.g., MedGemma-4B, Qwen3.5-9B) to predict mutual entailment via log-probabilities or verbalized JSON outputs.

Experimental Design

The evaluation employs a two-pronged approach:

Synthetic Experiments: Using controlled biomedical terminologies (ICD-11, MeSH, MedDRA), the authors generate ground-truth agreement levels by sampling synonyms and introducing controlled "negative" examples (siblings, generalizations, and lexical overlaps). This allows for precise calculation of Mean Absolute Error (MAE) between estimated soft IRR and ground-truth IRR.
Real-World Experiments: Evaluation on two datasets with free-text annotations and known ground-truth mappings to standardized terminologies:
- Derm1M: Dermatology image captions mapped to skin concepts.
- REFLACX: Chest X-ray reports with radiologist annotations mapped to anomaly labels.

For set-matching scenarios, the authors compare greedy matching against the Hungarian algorithm for optimal assignment.

Key Contributions

Systematic Comparison: The paper provides the first systematic comparison of diverse semantic equivalence measures (embeddings, NLI, LLMs, edit distance) for soft IRR, analyzing their performance and specific failure modes against ground truth.
Set-Matching Extension: The authors extend soft IRR to set-valued annotations, proposing methods to solve the assignment problem between two sets of unstructured text and evaluating the impact of optimal (Hungarian) vs. greedy matching.
NLI-Based Proposal: The study introduces and validates the use of NLI models for calculating soft IRR, framing it as a paraphrase detection task via mutual entailment.
Failure Mode Analysis: Through controlled negative sampling, the paper identifies that embedding-based approaches struggle to distinguish between similar but non-equivalent concepts (e.g., generalizations), whereas NLI and LLM-based approaches are more robust.

Results

Performance: Deep learning-based measures (NLI, embeddings, LLMs) generally outperform exact matching, set-overlap, and edit-distance methods in synthetic settings, particularly in capturing semantic nuance.
Failure Modes: Embedding-based models show high error rates when faced with "generalization errors" (e.g., a specific concept vs. its parent category) because they conflate similarity with equivalence. In contrast, NLI and LLM-based measures maintain lower error rates across different types of disagreements.
Scalability vs. Accuracy: While LLMs (especially verbalized variants) achieve high accuracy, they are computationally expensive, making the estimation of chance agreement ( $p_e$ ) via Monte Carlo sampling difficult for large datasets. NLI models offer a "sensible compromise," balancing scalability with robust performance.
Real-World Validation: On real-world datasets (Derm1M, REFLACX), the best soft measures achieved MAEs between 0.005 and 0.065 against ground-truth IRR, demonstrating practical viability. NLI-based approaches were found to be the most robust across different datasets.

Significance and Claims

The paper claims to be the first to quantify IRR for unstructured text annotations in this manner, moving beyond the limitations of discrete category agreement. The authors argue that the choice of equivalence measure significantly impacts the estimated reliability of annotation tasks. They conclude that while embeddings are scalable and LLMs are accurate, NLI-based measures represent the best compromise for unstructured biomedical annotation tasks due to their ability to handle various types of semantic disagreements while remaining computationally feasible.

The study is modest in its scope, acknowledging limitations such as the restriction to biomedical datasets (which may not generalize to other domains), the non-exhaustive nature of the tested measures, and the potential for synthetic experiments to be easier than real-world scenarios due to pretraining on similar terminologies. The work does not propose new future applications but rather establishes a methodological foundation for assessing the reliability of open-ended biomedical data extraction.