Technical Summary: MedicalBench

Problem Statement

Medical concept extraction from electronic health records (EHRs) is critical for downstream applications such as cohort identification, outcome prediction, and automated documentation. However, a significant challenge persists: medically meaningful concepts (e.g., diagnoses like anemia or obesity) are frequently implicit rather than explicitly stated in clinical narratives. For instance, a note may list "low hemoglobin" or "BMI 37" without explicitly using the terms "anemia" or "obesity."

Existing benchmarks for medical concept extraction often rely on human-annotated evidence spans but predominantly focus on explicitly stated concepts. They offer limited coverage of cases requiring inference from indirect evidence. Furthermore, most current benchmarks evaluate concept presence at the document level without providing insight into why a concept is warranted or where the supporting evidence resides, failing to test the reasoning capabilities required for robust medical extraction.

Methodology

Dataset Construction (MedicalBench)

The authors constructed MedicalBench, a benchmark designed to evaluate medical concept extraction with explicit evidence grounding and implicit reasoning. The dataset is derived from MIMIC-IV discharge summaries and associated ICD-10 codes.

1. Data Source & Candidate Concepts: Discharge summaries were mapped to ICD-10 diagnoses and procedures. These billing codes, reviewed by professional coders, served as the ground truth for clinical concepts.
2. LLM-Based Triage (Positive Candidates): To identify challenging cases where models might fail, a two-stage triage process was employed:
   • Stage 1: A non-reasoning LLM (GPT-4o-mini) classified note-concept pairs as Explicit, Implicit, or Unrelated.
   • Stage 2: A reasoning LLM (o3) re-evaluated cases where the first model judged a concept as "Unrelated" despite the concept being present in the ICD-10 codes. These "putative positives" formed the basis for challenging implicit examples.
3. Negative Sampling: To ensure difficulty, two strategies were used:
   • Prevalence-weighted negatives: Sampling based on concept frequency in MIMIC-IV.
   • Semantically similar negatives: Selecting concepts that are medically related but distinct (e.g., "Obesity" vs. "BMI 30–39") to test fine-grained reasoning.
4. Annotation Pipeline:
   • Expert Annotation: 9 independent medical experts labeled pairs as Related or Unrelated. For Related cases, they identified sentence-level evidence spans and provided clinical rationales.
   • Quality Control: Disagreements were adjudicated by a subject matter expert (SME). Automated programs checked for consistency.
   • Final Dataset: The resulting dataset contains 823 high-quality examples (352 gold positives, 471 gold negatives), emphasizing implicit reasoning, semantic ambiguity, and cases where LLMs disagree with experts.

Evaluation Tasks

MedicalBench defines two complementary tasks:

1. Medical Concept Extraction: A binary classification task (Does the note document the concept?). Metrics: Micro-averaged Precision, Recall, and F1.
2. Sentence-Level Evidence Retrieval: For positive cases, the system must retrieve the specific sentences supporting the concept. Metric: Macro-averaged sentence recall.

Key Results

Model Performance

The authors evaluated state-of-the-art Large Language Models (LLMs) and a supervised baseline (PLM-ICD).

• Overall Difficulty: Performance remains modest across all models. The highest extraction F1 score achieved was 0.59 (Gemini-3-pro-preview), with most models scoring below 0.60. This underscores the inherent difficulty of the task.
• Model Comparison:
  • Reasoning vs. Non-Reasoning: Reasoning models (e.g., o3, GPT-5) generally outperformed non-reasoning models (e.g., GPT-4.1).
  • Parameter Scale: Larger models (e.g., Claude-opus-4.5) outperformed their smaller counterparts (Sonnet, Haiku).
  • Baseline: While the supervised PLM-ICD established a traditional discriminative frontier, several LLMs (Gemini-3-pro-preview, GPT-5, Grok-4) surpassed it even in zero-shot settings, achieving a better balance of precision and recall.

Impact of Reasoning and Prompting

• Evidence Retrieval Correlation: Models that better retrieved supporting evidence tended to make more correct concept extractions.
• Reasoning Cues: Explicitly incorporating reasoning cues (paraphrased evidence) into the prompt significantly improved performance. For GPT-5, adding reasoning cues increased F1 from 0.55 to 0.71, driven by gains in both precision (+12%) and recall (+18%).
• Implicit Evidence Extraction: Prompting models to explicitly consider implicit evidence was critical. Removing this instruction caused GPT-5's F1 to drop from 0.55 to 0.32, with recall nearly halving. This indicates that models fail to capture hidden relationships without specific guidance to infer them.
• Note Length Bias: Unlike previous findings in text extraction, MedicalBench showed no significant correlation between note length and extraction accuracy (r = 0.02). This suggests the benchmark successfully isolates reasoning difficulty rather than superficial confounders like context length.

Case Studies

The paper highlights three specific failure modes even in high-performing models:

1. Implicit Inference: Failure to link medication patterns (e.g., Valganciclovir, Furosemide) to implied conditions (e.g., kidney failure).
2. Temporal Context: Confusing current acute events with past conditions (e.g., misinterpreting "myocardial infarction" in a differential diagnosis as a confirmed "old myocardial infarction").
3. Scattered Evidence: Failure to integrate evidence separated by large token distances (e.g., connecting a procedure in the "Procedures" section with a pathology result in the "Assessment" section).

Significance and Claims

The paper positions MedicalBench as the first systematic benchmark for implicit, evidence-grounded medical concept extraction. Its primary contributions are:

1. Shifting the Paradigm: It moves the field from "extraction-as-classification" toward "extraction-as-reasoning," requiring models to justify predictions with transparent, medically faithful evidence.
2. Diagnostic Capability: By including implicit positives, semantically confusable negatives, and LLM-expert disagreements, the benchmark isolates reasoning difficulties that current models struggle with.
3. Foundation for Explainability: The dataset provides a foundation for developing medical language models that can not only identify relevant concepts but also articulate why they are relevant, a prerequisite for trustworthy deployment in real-world medical settings.
4. Empirical Insight: The results demonstrate that while current LLMs struggle with implicit reasoning, performance can be substantially improved by explicitly prompting for reasoning cues and implicit evidence extraction, highlighting a clear path for future model development.

The authors conclude that MedicalBench reveals the limitations of current systems in handling nuanced, implicit medical documentation and establishes a rigorous standard for evaluating the interpretability and reasoning capabilities of future medical AI.