🤖 AI

Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction

This paper systematically evaluates large language models against supervised baselines for extracting post-discharge clinical actions, revealing that while LLMs excel at detecting actionability, they struggle with fine-grained classification due to a lack of clinical reasoning, thereby highlighting the critical need for datasets annotated with rationales rather than just labels.

Original authors: Shivali Dalmia, Ananya Mantravadi, Prasanna Desikan

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shivali Dalmia, Ananya Mantravadi, Prasanna Desikan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Handoff" Problem

Imagine a patient's journey through the hospital as a relay race. When a runner (the patient) finishes their leg in the hospital and hands the baton to the next runner (the patient going home), the "baton" is actually a discharge summary. This document tells the patient and their family doctor what to do next: take this pill, get that test, or see a specialist.

The problem? These discharge notes are often messy, long, and written like a novel. They mix up what happened yesterday with what needs to happen tomorrow. It's like trying to find a specific recipe in a cookbook that has been filled with random grocery lists and travel stories. If a doctor misses a crucial instruction hidden in that text, the patient could get sick again.

What This Paper Did

The researchers wanted to see if Large Language Models (LLMs)—the super-smart AI chatbots we hear about in the news—could act as a "smart scanner" to read these messy notes and pull out the specific instructions (like "take aspirin" or "get an X-ray").

They compared two types of AI:

The "Specialized Intern" (Supervised BERT models): These are older, specialized AI models that have been trained on thousands of examples by humans. They are like a medical student who has memorized a specific textbook.
The "Genius Generalist" (Modern LLMs): These are the big, general-purpose AI models (like GPT-5, Gemini, Claude) that haven't been specifically trained on this exact task. They are like a brilliant polymath who knows a little bit about everything and can figure things out just by reading the instructions (prompting).

The Two-Step Strategy (The "Filter and Sort" Method)

The researchers realized that asking the AI to just "find the instructions" was too hard. So, they built a two-stage framework, which is like a two-step sorting machine in a factory:

Stage 1: The "Is this important?" Filter.
The AI first reads a sentence and asks: "Does this sentence actually tell someone to do something?"
- Example: "The patient was on aspirin." (No, that's just history. Discard.)
- Example: "Stop taking the aspirin." (Yes, that's an action. Keep.)
- Result: The modern LLMs were amazing at this. They were better than the specialized "Interns" at spotting what was important and what was just background noise.
Stage 2: The "What kind of action is it?" Sorter.
Once the AI knows a sentence is an action, it has to sort it into a specific box: Is it a Medication? A Lab Test? An Appointment?
- Result: Here, the specialized "Interns" (BERT models) won. They were better at putting the items in the exact right box. The "Genius Generalists" (LLMs) were good, but they made more mistakes in the fine details.

The "Gotcha": The AI vs. The Annotation Rules

The most interesting part of the paper is the "Error Analysis." The researchers found that sometimes the AI was actually right, but the test said it was wrong.

Think of it like a teacher grading a student's essay.

The Student (AI): Writes a sentence that is logically correct and medically sound.
The Teacher (Dataset Rules): Says, "No, you got it wrong because you didn't follow the strict formatting rule in the textbook."

For example, if a note says "Stop taking the pill" inside a section called "Discharge Instructions," the dataset rules might say, "This is just a 'Patient Instruction'." But the AI correctly thinks, "Wait, this is also a 'Medication Change'!" The AI is clinically right, but the dataset's rigid rules penalize it for being too smart.

The paper argues that the "ground truth" (the answer key) has some inconsistencies. Sometimes it labels things based on where they appear in the document (structure) rather than what they actually mean (semantics).

The Verdict

For finding the needle in the haystack (Stage 1): The modern, general-purpose AI models are the winners. They are better at understanding the context and filtering out the fluff without needing to be retrained.
For sorting the needles into tiny boxes (Stage 2): The old-school, specialized models still have the edge. They are more consistent with the specific rules of the dataset.
The Medical Specialist AI: Interestingly, a model specifically trained for medicine (MedGemma) did worse than the general ones. The researchers think this is because it was trained to answer questions, not to follow strict formatting rules to extract data.

The Bottom Line

The paper concludes that we shouldn't just pick one AI and hope for the best. Instead, we should build a hybrid team:

Use the Genius Generalist (LLM) to do the heavy lifting of reading the note and finding the important actions.
Use the Specialized Intern (BERT) or a human to double-check the specific categories.

This combination creates a safety net that is both flexible and accurate, ensuring that when a patient goes home, no critical instruction is left behind. The paper also suggests that to make these AI tools truly reliable, we need to update our "answer keys" (datasets) to include the reasoning behind why something is an action, not just the action itself.

Technical Summary: Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction

Problem Formulation

The paper addresses the critical challenge of extracting actionable clinical follow-up tasks from unstructured hospital discharge summaries. Effective care transitions from hospital to home rely on the clarity of these documents; however, discharge notes are often lengthy, complex, and inconsistently structured, containing historical data that obscures future care instructions. This lack of structure imposes a cognitive burden on Primary Care Providers (PCPs), potentially leading to adverse drug events, delayed care, and preventable readmissions.

The specific task is defined as identifying sentence-level spans corresponding to fine-grained, patient-specific action items (e.g., follow-up appointments, lab tests, medication adjustments) that must be executed post-discharge. The authors utilize the CLIP dataset, which provides expert-annotated discharge notes from the MIMIC-III database with labeled action spans across seven predefined clinical categories: Appointment, Medication, Lab, Imaging, Procedure, Patient Instructions, and Other.

Methodology

The authors propose a two-stage extraction framework designed to decompose the complex task of clinical action extraction into manageable steps, operating under strict data-privacy constraints (Zero Data Retention).

1. Data Preprocessing and Context Batching

Sentence-Level Conversion: Original character-span annotations from the CLIP dataset are aggregated into sentence-level ground truth. A sentence is labeled as "Actionable" if it contains any annotated span; otherwise, it is "Non-Actionable."
Context Batching: To address contextual dependencies (e.g., section headers or references to previous sentences), sentences are processed in contiguous batches of size $k$ (optimized via successive halving among $k \in \{10, 15, 30\}$ ). This allows the model to utilize surrounding context while making independent predictions for each sentence.

2. Two-Stage Prompt Chaining

The extraction process is split into two distinct reasoning stages to reduce false positives from descriptive narratives:

Stage 1: Actionability Filtering: The model determines whether a sentence contains a concrete clinical instruction requiring post-discharge action. It distinguishes between "Actionable" (e.g., "hold ASA," "repeat tests in 2 weeks") and "Non-Actionable" (e.g., "Discharged on Atorvastatin," "follow up as needed").
Stage 2: Clinical Action Category Classification: Only sentences deemed actionable in Stage 1 are passed to this stage. The model assigns one or more of the seven clinical action categories. This stage utilizes few-shot prompting with explicit definitions, inclusion/exclusion criteria, and strict JSON formatting requirements.

3. Experimental Setup

Models Evaluated: Five general-purpose and medical-specialized Large Language Models (LLMs) were benchmarked in zero-shot and few-shot settings: GPT-5.2, Gemini-3-Flash, Claude Sonnet 3.5, DeepSeek-V3.2, and MedGemma-27B-it.
Baselines: The LLMs were compared against two state-of-the-art supervised BERT-based models from the original CLIP work: MIMIC-DNote-BERT+Context and TTP-BERT+Context (250k).
Evaluation Metrics: Performance was measured using Precision, Recall, and F1 scores at both the binary actionability level (Stage 1) and the multi-label category level (Stage 2), alongside JSON compliance rates.

Key Contributions

Two-Stage Framework: Introduction of a decomposition strategy that separates actionability detection from category classification to improve robustness in clinical text.
Systematic Benchmarking: A comprehensive evaluation of modern generative LLMs against established supervised baselines for safety-critical clinical extraction, a gap previously unaddressed in literature.
Privacy-Compliant Evaluation: Demonstration of effective LLM deployment for clinical tasks under Zero Data Retention protocols, ensuring sensitive health data is not retained.
Annotation Analysis: Identification of systematic inconsistencies in the ground-truth annotations, specifically regarding structural labeling rules versus semantic clinical reasoning.

Results

Stage 1: Actionability Filtering

Modern LLMs significantly outperformed supervised baselines in distinguishing actionable instructions from descriptive text.

GPT-5.2 achieved the highest Macro F1 of 0.895, representing a 23% relative improvement over the best supervised baseline (TTP-BERT+Context at 0.668).
Gemini-3-Flash followed closely with a Macro F1 of 0.888.
Notably, the medical-specialized model, MedGemma-27B-it, underperformed significantly (Macro F1: 0.459), suggesting that instruction-tuning for QA does not necessarily translate to structured extraction tasks without specific fine-tuning.

Stage 2: Action Category Classification

Supervised BERT models retained a meaningful advantage in fine-grained, multi-label classification.

TTP-BERT+Context achieved the highest overall Macro F1 of 0.668.
GPT-5.2 achieved the best LLM performance with a Macro F1 of 0.570, falling approximately 15% short of the supervised baseline.
Category-Specific Performance: While LLMs generally lagged, they showed competitiveness in specific areas. For instance, Gemini-3-Flash matched TTP-BERT on Lab-related follow-up (F1 0.754 vs. 0.745), and Claude Sonnet 3.5 showed strong performance in Medication-related follow-up among LLMs.

Error Analysis and Limitations

Qualitative analysis revealed that many model "failures" were actually misalignments between model reasoning and dataset annotation conventions rather than true clinical reasoning deficits:

Structure vs. Semantics: The CLIP dataset often applies labels based on document sections (e.g., labeling all sentences in "Discharge Instructions" as "Patient Instructions") rather than semantic content. Models correctly identifying the semantic category (e.g., "Medication") were penalized as false positives for deviating from the structural label.
Implicit Actions: Models struggled with sentences describing past inpatient actions that imply future follow-up (e.g., "Warfarin was initiated" implying "monitor INR"), often filtering them out as non-actionable due to the lack of explicit future directives.
Annotation Inconsistencies: The ground truth showed inconsistencies in labeling medication lists, where some chronic medications were labeled as actionable while others were not, confusing the models.

Significance and Conclusion

The paper concludes that while contemporary LLMs demonstrate superior semantic reasoning for binary actionability detection, they currently lack the consistency of task-specific supervised models for fine-grained multi-label classification in safety-critical contexts.

The authors argue that reported performance metrics may underestimate the true clinical reasoning capabilities of LLMs due to rigid annotation conventions that prioritize document structure over clinical semantics. They posit that advancing clinical NLP requires:

Reasoning-Annotated Datasets: Datasets that document why a span is actionable, not just which spans are labeled.
Hybrid Architectures: Combining the flexibility of LLMs for initial extraction with specialized classifiers for category assignment.
Human-in-the-Loop Validation: Integrating clinician review to distinguish between genuine model errors and annotation ambiguities.

Ultimately, the work suggests that while LLMs offer a flexible alternative to traditional pipelines, a hybrid approach leveraging the strengths of both generative models and supervised classifiers is necessary for reliable, deployment-ready medical agent workflows.