MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations

Technical Summary of MedErrBench

Problem Statement
The paper addresses the critical need for robust evaluation of Large Language Models (LLMs) in detecting, localizing, and correcting medical errors within clinical text. Inaccuracies in clinical documentation, such as misdiagnoses or incorrect treatment suggestions, can lead to severe adverse patient outcomes. While LLMs are increasingly deployed in healthcare, comprehensive benchmarks for medical error correction remain scarce. Existing resources, such as the MEDEC dataset, are primarily monolingual (English) and lack a diverse taxonomy of error types that reflect the complexity of real-world clinical practice. Furthermore, the absence of multilingual datasets hinders the development of equitable AI systems for global healthcare contexts.

Methodology
The authors introduce MedErrBench, a fine-grained, multilingual benchmark developed in collaboration with experienced clinicians. The methodology involves three core components:

1. Taxonomy Construction: The authors distilled ten representative clinical error categories, expanding upon existing frameworks. These include: Diagnosis, Management, Treatment, Pharmacotherapy, Causal Organism/Pathogen, Lab/Serum Value Interpretation, Physiology, Histology, Anatomy, and Epidemiology.
2. Dataset Construction: The benchmark covers English, Chinese, and Arabic.
   • English and Chinese: Samples were adapted from MedQA, with expert-verified errors injected into clinical notes.
   • Arabic: Samples were adapted from MedArabiQ and MedAraBench.
   • Error Injection: For each instance, a correct answer was preserved, and a plausible but incorrect alternative was inserted to create an erroneous clinical note. For error types missing from source datasets, clinicians contributed real-world cases.
   • Annotation: Beyond error categorization, instances were annotated with difficulty levels (Easy, Medium, Hard), reasoning types (Factual Recall, Single-hop, Multi-hop), and important clinical words.
   • Quality Control: A rigorous two-stage review process was employed. First, native-speaking NLP researchers verified content and formatting. Second, two independent clinicians reviewed all instances for medical validity, correcting transformation errors and validating labels.
3. Evaluation: The benchmark supports three tasks: Error Detection (binary classification), Error Localization (identifying the erroneous sentence), and Error Correction (generating the revised text). The authors evaluated a diverse set of models, including general-purpose LLMs (e.g., GPT-4o, LLaMA), language-specific LLMs (e.g., Qwen, ALLAM), and medical-domain LLMs (e.g., MedGemma, HuatuoGPT).

Key Contributions

• Clinician-Informed Taxonomy: The establishment of a 10-category error typology that reflects real-world clinical challenges, providing a foundational schema for future dataset construction.
• First Multilingual Benchmark: The introduction of MedErrBench as the first fine-grained benchmark for medical error detection and correction across English, Chinese, and Arabic, rigorously validated by domain experts.
• Comprehensive Evaluation: A systematic benchmarking of various LLM families across multiple languages and tasks, accompanied by in-depth analyses regarding the impact of error type definitions, example difficulty, and reasoning types.

Results
The evaluation reveals significant performance gaps across models and languages:

• Model Performance: Language-specialized models (e.g., Deepseek-R1, Deepseek-V3, Doubao-1.5) generally outperformed general-purpose and medical-domain models. Notably, medical-specific models like MedGemma and HuatuoGPT did not consistently outperform general-purpose models, suggesting that domain-specific training does not automatically translate to superior error detection and correction capabilities.
• Language Disparities: Performance was notably lower in non-English settings, particularly for Arabic, where the ALLAM-7B model struggled significantly. This highlights the need for language-aware systems.
• Task Difficulty: Localization and correction tasks proved more challenging than detection across all models.
• Prompting and Context: Providing error type definitions consistently improved performance, especially in zero-shot settings. Few-shot learning generally outperformed zero-shot, though the impact of adding error type labels without definitions varied by model.
• Difficulty and Reasoning: Model performance varied by difficulty level and reasoning type. For instance, some models performed better on "Hard" examples in few-shot settings, while others showed a decline.

Significance and Claims
The paper claims that MedErrBench addresses critical limitations in current medical NLP resources by providing a diverse, high-quality, and clinically validated dataset. The authors emphasize that their findings highlight the challenges current LLMs face in understanding clinical errors, particularly in non-English contexts and for complex reasoning tasks. By making the dataset and evaluation protocols publicly available, the authors aim to advance multilingual clinical NLP research, promoting the development of safer and more equitable AI-based healthcare systems globally. The work serves as a foundation for future research into fine-grained clinical reasoning, severity stratification, and multi-agent systems for clinical tasks.