📄 otolaryngology

Developing an open-source framework for LLM evaluation of patients using EHR clinical documentation; performance of LLMs relative to medical professionals

This study demonstrates that current large language models fail to achieve inter-rater reliability comparable to medical professionals when extracting structured clinical information from ENT electronic health records, suggesting they are best suited for initial data extraction requiring human verification rather than autonomous clinical deployment.

Original authors: Barrett, L., Joshi, N., North, A. S., Dimitrov, L., Maughan, E. F., Ross, T., Pankhania, R., Paramjothy, K., Minty, I., Farache-Trajano, L., Smith, S. L., Mason, K. A., Bhargava, E. K., Donnelly, C.

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Barrett, L., Joshi, N., North, A. S., Dimitrov, L., Maughan, E. F., Ross, T., Pankhania, R., Paramjothy, K., Minty, I., Farache-Trajano, L., Smith, S. L., Mason, K. A., Bhargava, E. K., Donnelly, C., Fatoum, H., Padiyar, A., Kader, Z., Chan, C. H. K., Schilder, A. G., Mehta, N.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Hospitals today generate a vast ocean of written records. Every patient visit, every test result, and every treatment decision is captured in digital notes known as electronic health records. These documents are rich with detail, containing the specific language doctors use to describe symptoms, diagnose conditions, and plan care. For decades, turning these unstructured stories into organized, searchable data has been a difficult task, often requiring humans to read and re-write the information by hand. Recently, a new type of computer program called a large language model has emerged. These systems are trained on massive amounts of text and can read, understand, and summarize human language with surprising fluency. They have shown they can answer medical questions and pass licensing exams, leading many to wonder if they can also read patient records and extract the vital facts inside them automatically.

This question is not just about saving time; it is about safety and accuracy. If a computer is to help doctors manage patient care, it must find the right information without making things up or missing critical details. A new study set out to test this capability directly. Researchers gathered nearly one hundred real patient records from ear, nose, and throat clinics and asked both human doctors and seven different large language models to read them. The task was to pull out specific facts, such as the patient's age, their symptoms, their diagnosis, and the treatments they received. To ensure everyone was speaking the same language, the researchers required that every piece of information be translated into a standardized code used by hospitals worldwide. This allowed for a precise comparison of how well the machines performed against the humans.

The results revealed a clear gap between human expertise and current artificial intelligence. When the fourteen doctors in the study read the same records, they agreed with each other on the extracted information about 81 percent of the time. This high level of agreement showed that the doctors were interpreting the notes consistently. However, when the computer models were compared to the doctors, the agreement dropped significantly to about 66 percent. In other words, the machines were not yet as reliable as the humans in identifying the same facts from the same text. The study tested models of different sizes, from massive systems with hundreds of billions of connections to smaller ones that could run on local computers. None of them reached the level of consistency shown by the medical professionals.

The performance varied depending on what kind of information was being extracted. The models were quite good at finding treatments and test results, which are often written in clear, standard ways. They were less successful with signs and diagnoses, which often require a deeper understanding of the clinical context. Interestingly, the computers tended to find more information than the doctors did, particularly regarding risk factors. While doctors often focus on the immediate problem, the models seemed to flag every possible risk mentioned in the text. This suggests the machines are not just copying human behavior but are processing the text differently, sometimes catching details a human might overlook, but also potentially flagging things that are not clinically relevant.

Despite these differences, the machines showed they could be useful tools if used correctly. The study found that when the models did identify a piece of information, they were usually right, with a precision rate of 97 percent. This means they rarely made up facts that were not there. However, they did miss some information about 15 percent of the time. This pattern points to a specific role for these tools in the future: they are best suited to act as a first pass, quickly scanning records to pull out likely candidates for the doctor to review. The final decision and verification would still need to come from a human. The researchers concluded that while these models are powerful, they are not yet ready to work alone in a clinical setting. They are best viewed as assistants that can help organize the flood of medical data, provided a human expert is always there to check their work.

Technical Summary: Evaluation of LLMs in Clinical Information Extraction from ENT EHRs

Problem Statement
While Large Language Models (LLMs) have demonstrated proficiency in medical knowledge assessments (e.g., licensing exams), their ability to extract structured, standardized clinical information from real-world Electronic Health Records (EHRs) remains uncertain. Specifically, there is a lack of rigorous evaluation regarding whether LLMs can achieve inter-rater reliability comparable to medical professionals when processing unstructured clinical narratives. This uncertainty poses a barrier to the deployment of AI in clinical documentation workflows, particularly in specialized domains like Ear, Nose, and Throat (ENT) care, where complex terminology and the need for deep phenotyping are critical.

Methodology
The study employed a cross-sectional design to evaluate the performance of seven LLMs against fourteen qualified medical professionals (authors of the study) in extracting clinical data from 98 publicly available, anonymized ENT EHRs sourced from MTSamples.

Dataset and Annotation: The dataset consisted of 98 ENT clinical documents. Annotations covered seven categories: socio-demographics, signs, symptoms, diagnoses, treatments, risk factors, and test results.
Standardization: To ensure objective comparison, all extracted entities were mapped to SNOMED-CT codes (UK database version 39.0.0). Annotators provided exact text, context, and qualifiers (e.g., laterality, presence status).
Models Evaluated: Seven LLMs were tested, representing different architectures and deployment modes:
- Proprietary Cloud: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro.
- Open-Source Cloud: Llama 3.1 (405B, 70B, 8B).
- Local Deployment: Gemma 3 (12B).
- All models utilized a standardized five-step chain-of-thought prompting strategy with one-shot learning and strict parameter controls (fixed random seed, deterministic sampling).
Statistical Analysis:
- Reliability: Inter-rater agreement was measured using Cohen's Kappa ( $\kappa$ ).
- Hypothesis Testing: Bayesian hierarchical modeling was used to test non-inferiority. The study tested whether the agreement between medical professionals and LLMs (Medic-LLM) was non-inferior to the agreement between medical professionals (Medic-Medic). Non-inferiority margins ( $\delta$ ) of 0.05, 0.10, and 0.15 were assessed with 95% posterior probability thresholds.
- Accuracy: Classification metrics (Precision, Recall, Specificity, False Positive Rate) were calculated using medical professionals' annotations as the ground truth, employing a permutation approach to account for human variability.

Key Results

Inter-Rater Reliability:
- Medic-Medic: Achieved substantial agreement with a mean Cohen's Kappa of 0.752 (95% CI: 0.710–0.794). Bayesian posterior mean was 0.813.
- Medic-LLM: Achieved only fair agreement with a mean Cohen's Kappa of 0.391 (95% CI: 0.362–0.420). Bayesian posterior mean was 0.659.
- LLM-LLM: Showed substantial agreement among models ( $\kappa = 0.795$ ), suggesting consistent biases across different architectures.
Non-Inferiority: The hypothesis that LLMs perform equivalently to medical professionals was rejected at all tested margins ( $\delta = 0.05, 0.10, 0.15$ ). The probability that Medic-LLM agreement falls within 0.10 of Medic-Medic agreement was only 4.5%.
Category Performance:
- Agreement varied by clinical category. The smallest differences between humans and LLMs were observed in test results and treatments.
- The largest discrepancies were found in signs and diagnoses, which require nuanced clinical interpretation.
- GPT-4o (the best-performing model in this cohort) achieved 97.0% precision and 84.9% recall, with a 7.5% false positive rate.
Extraction Volume: LLMs consistently extracted more clinical entities per document than human annotators, particularly in "risk factors" (LLM mean: 7.33 vs. Human mean: 1.01), indicating a divergence in how models and clinicians prioritize information.

Significance and Claims
The paper claims to provide the first systematic evaluation of LLMs against medical professionals using standardized inter-rater reliability metrics in a specialized clinical domain. The primary significance lies in establishing evidence-based guidance for the deployment of AI in clinical documentation:

Current Limitations: Current LLMs do not achieve inter-rater reliability levels comparable to medical professionals for clinical information extraction. They are not yet ready for autonomous operation in this context.
Recommended Workflow: Given the high precision (97.0%) but moderate recall and non-negligible false positive rate (7.5%), the authors suggest LLMs are best suited for initial extraction with mandatory human verification, rather than autonomous decision-making.
Methodological Framework: The study establishes a rigorous framework using SNOMED-CT and Bayesian hierarchical modeling to evaluate clinical AI, offering a reproducible methodology for future research.
Domain Specificity: The performance gaps highlight that while models may excel at standardized tasks, they struggle with the nuanced interpretation required for signs and diagnoses, suggesting that prompt engineering alone is insufficient without domain-specific architectural innovations or training.

The authors explicitly state that these findings should not be used to guide clinical practice directly, as the study relies on publicly available, educational documents (MTSamples) which may not fully represent the heterogeneity of real-world institutional EHRs.

Technical Summary: Evaluation of LLMs in Clinical Information Extraction from ENT EHRs

More like this