💬 NLP

Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications

This paper addresses the underexplored challenges of structured tabular reporting from nurse dictations and medical order extraction from consultations by evaluating various large language models, proposing an agentic pipeline for generating synthetic data, and releasing the first open-source datasets (SYNUR and SIMORD) to support these critical clinical documentation tasks.

Original authors: Jean-Philippe Corbeil, Asma Ben Abacha, George Michalopoulos, Phillip Swazinna, Miguel Del-Agua, Jerome Tremblay, Akila Jeeson Daniel, Cari Bader, Yu-Cheng Cho, Pooja Krishnan, Nathan Bodenstab, Thoma

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jean-Philippe Corbeil, Asma Ben Abacha, George Michalopoulos, Phillip Swazinna, Miguel Del-Agua, Jerome Tremblay, Akila Jeeson Daniel, Cari Bader, Yu-Cheng Cho, Pooja Krishnan, Nathan Bodenstab, Thomas Lin, Wenxuan Teng, Francois Beaulieu, Paul Vozila

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a hospital as a bustling, high-stakes kitchen. The chefs (doctors) are busy cooking up complex treatment plans, and the sous-chefs (nurses) are running around taking orders, checking temperatures, and noting down exactly what's happening with every dish (patient).

Right now, after a chaotic shift, both the chefs and the sous-chefs have to stop, sit down, and type out long, messy notes into a computer. This is the "documentation burden." It takes time away from actually caring for the patient.

This paper is about building a super-smart, digital assistant (powered by Large Language Models, or LLMs) that listens to the chefs and sous-chefs as they speak, and instantly turns their rambling, spoken words into neat, organized computer forms.

Here is the breakdown of their two main projects, explained with some kitchen metaphors:

1. The "Sous-Chef" Project: Turning Nurse Speech into Checklists

The Problem: Nurses often dictate notes like this: "Uh, the patient's breathing is a bit heavy, and they're using that little tube in the nose, and oh, they weigh about 98 kilos."
In the hospital computer system, this needs to be entered into a giant spreadsheet (called a "flowsheet") with specific boxes: Breathing Pattern: Labored, Oxygen Device: Nasal Cannula, Weight: 98 kg.

The Challenge:

The "Lost in the Middle" Problem: The spreadsheets are so huge (thousands of rows) that the AI gets overwhelmed, like a student trying to read a whole encyclopedia to find one word.
The "Messy Speech" Problem: Nurses speak with pauses, "ums," and corrections. The AI needs to understand that "uh, the breathing is... um, heavy" means "Labored."

The Solution:
The team built a three-step process:

Chopping it up: They split the long speech into small, logical chunks.
Finding the right menu: They use a "search engine" (RAG) to find only the relevant rows in the giant spreadsheet that match the current chunk of speech.
Filling the form: The AI fills in the specific boxes.

The Secret Sauce (SYNUR):
They didn't have enough real nurse recordings to train the AI (because patient privacy is strict). So, they created SYNUR, a "fake" dataset. They used an AI to imagine realistic nurse scenarios, then had real nurses verify them. It's like a flight simulator for nurses: the AI practices on fake flights so it's ready for the real thing.

2. The "Head Chef" Project: Turning Doctor Conversations into Orders

The Problem: A doctor talks to a patient for 20 minutes. Somewhere in that conversation, they say, "Okay, let's get a blood test for your diabetes and maybe start you on a heart pill."
The AI needs to extract these as formal "Medical Orders" with specific details: Type: Lab, Description: Hemoglobin A1c, Reason: Type 1 Diabetes.

The Challenge:

The "Needle in a Haystack": The orders are buried in long, casual conversations.
The "Free-Form" Problem: Doctors don't speak in perfect JSON code. They say things like, "I think we should probably do a CT scan, maybe tomorrow, to check that infection." The AI has to figure out the what, the why, and the where (which part of the conversation it came from).

The Solution:
They created SIMORD, a dataset of simulated doctor-patient conversations with perfect "gold standard" answers. They tested different AIs to see who could best act like a medical scribe.

The Results:

The Heavyweights: The biggest, most expensive AI models (like GPT-4o) did a great job, but they are slow and costly.
The Underdogs: Surprisingly, a smaller, open-source model (MediPhi) performed just as well as the giant models when given a few examples to learn from. This is huge because it means hospitals could run this on their own computers without paying huge fees to tech giants.
The "Reasoning" Models: Newer AI models that "think" before they speak (like o1) were amazing at finding where in the conversation the order came from (Provenance), but sometimes they got confused on the actual details.

Why This Matters

Think of this paper as a blueprint for automating the paperwork so healthcare workers can stop being data-entry clerks and start being caregivers again.

For Nurses: It turns a 10-minute typing session into a 10-second voice note.
For Doctors: It ensures no treatment order gets lost in the noise of a busy conversation.
For Everyone: By releasing these "fake" but realistic datasets (SYNUR and SIMORD) to the public, the authors are handing the keys to the whole world. Now, any researcher can build better tools without needing to steal private patient data.

In short: They taught computers to listen to the chaos of a hospital, understand the medical jargon, and neatly file everything away, giving doctors and nurses back the most valuable resource they have: time with their patients.

1. Problem Statement

Despite the success of Large Language Models (LLMs) on standardized medical benchmarks (e.g., MedQA, MMLU-Medical), two high-impact, real-world clinical Natural Language Processing (NLP) tasks remain underexplored due to data scarcity, sensitivity, and complexity:

Structured Reporting from Nurse Dictations: Converting unstructured, often disfluent nurse speech into structured Electronic Health Record (EHR) flowsheet entries. This is challenging due to long contexts, "overflowing" flowsheets (hundreds of rows), and the need to map spoken language to specific schema keys.
Medical Order Extraction from Doctor-Patient Consultations: Extracting structured medical orders (medication, lab, imaging, follow-up) with associated reasons and provenance from long, complex doctor-patient dialogues. This involves generating free-form fields (description/reason) alongside structured labels.

The lack of public, high-quality datasets for these specific tasks has hindered the evaluation and deployment of AI solutions that could significantly reduce documentation burdens on healthcare providers.

2. Methodology

The authors propose a dual approach involving the creation of new datasets and the evaluation of various LLM architectures (open-weight and closed-weight).

A. Data Creation

1. SYNUR (Synthetic Nursing Dataset):
To address the scarcity of nurse dictation data, the authors developed a six-stage agentic pipeline to generate realistic, non-sensitive synthetic data:

Seed Generation: Started with 80 human-verified, fabricated dictations.
Ontology Mining & Expansion: Used LLMs to extract 547 observations, distill them into 97 concepts, and expand the ontology to 193 concepts (validated by expert nurses).
Scenario & Dictation Synthesis: Generated patient case rationales and combined them with the ontology to create 456 unique scenarios. These were passed to an LLM (GPT-4o) to generate realistic nurse dictations, including natural speech disfluencies and corrections.
Annotation: Expert nurses annotated the synthetic data, resulting in 223 synthetic dictations with 3,000 observations.
Validation: The synthetic data was validated via "LLM-as-a-Judge" (scoring ~3/4 for quality and realism) and Inter-Annotator Agreement (86.8% accuracy, $\kappa=0.634$ ).

2. SIMORD (Simulated Orders Dataset):

Constructed by re-annotating high-quality conversations from existing datasets (ACI-Bench and PriMock57).
Medically trained annotators extracted medical orders (medication, lab, imaging, follow-up) including descriptions, reasons, and provenance (line numbers).
The dataset includes 64 training samples and 100 test samples (255 total orders).

B. Task Methodologies

Task 1: Nurse Observation Extraction

Pipeline: A three-step process to handle long contexts and schema constraints:
1. Segmentation: An LLM splits the streaming transcript into medically coherent, non-overlapping segments.
2. RAG Filtering: Reduces the massive flowsheet schema (hundreds of rows) to the top- $N$ relevant candidates based on cosine similarity with the current segment.
3. Extraction: An LLM extracts the observation and value, canonicalizing it to the reduced schema.
Models Evaluated: Closed-weight models (GPT-4o, 4o-mini, 4.1, 4.1-mini).

Task 2: Medical Order Extraction

Pipeline: Direct prompt-based extraction of JSON objects containing order_type, description, reason, and provenance.
Models Evaluated:
- Closed-weight: GPT-4o, GPT-4.1, o1-mini, o1-prev, o3-mini.
- Open-weight: Phi-3.5-mini-instruct, MediPhi-Instruct (3.8B), Llama3-8B-instruct, Llama3-Med42-8B.
Settings: Zero-shot and Few-shot (1-2 examples).

3. Key Contributions

SYNUR Dataset: The first open-source synthetic nursing dataset for extracting structured observations from spoken dictations, addressing data privacy and scarcity.
SIMORD Dataset: The first open-source dataset for extracting medical orders from doctor-patient simulated transcripts.
Agentic Data Pipeline: A novel framework for generating realistic, annotated synthetic clinical data using an iterative loop between domain experts and LLMs.
Comprehensive Benchmarking: The first systematic evaluation of both open- and closed-weight LLMs on these specific clinical extraction tasks, highlighting the viability of lightweight models.

4. Results

Nurse Observation Extraction (SYNUR & Proprietary Data)

Model Performance: GPT-4.1 consistently outperformed other closed-weight models across proprietary hospital datasets (F1 scores ranging from 88.2% to 93.6%).
Synthetic vs. Real: GPT-4o performed best on the synthetic SYNUR dataset (likely due to training data overlap), while GPT-4.1 led on real hospital data.
Few-Shot Learning: Including examples in the prompt significantly improved F1 scores across all models.
Error Analysis: Models struggled with mapping values to specific medical scales and inferring missing identifiers.

Medical Order Extraction (SIMORD)

Closed-Weight Models: No single model dominated all metrics.
- GPT-4o excelled in Description and Type extraction.
- o1-mini showed superior performance in Provenance (identifying source lines), likely due to its reasoning capabilities.
- Reasoning Models (o1 series): Showed a massive improvement (+25.7%) in Provenance but did not consistently outperform others in Description or Type.
- Parsing Issues: Models like o1-preview and GPT-4.1 frequently produced malformed JSON (run-on sequences), whereas 4o, o1-mini, and o3-mini maintained <1% parsing errors.
Open-Weight Models:
- MediPhi-Instruct (3.8B): With two-shot prompting, it surpassed GPT-4o in Description and achieved parity in Match and Type metrics, demonstrating the viability of small, open-weight models for this task.
- Llama3-8B: Achieved Description and Reason scores comparable to GPT-4o in few-shot settings.
Metric Gaps: There is a significant gap (>30%) between Match scores (order existence) and Description scores, indicating that while models find the order, generating the precise free-text description remains a challenge.

5. Significance and Limitations

Significance:

Workflow Integration: The proposed pipelines (especially the nurse observation extraction) are designed to integrate directly into existing EHR flowsheets without altering established routines, offering a practical path to reducing documentation burden.
Democratization: By releasing SYNUR and SIMORD, the authors provide critical resources for the research community to advance clinical NLP beyond standardized benchmarks.
Model Efficiency: The results suggest that lightweight, open-weight models (e.g., 3.8B parameters) can rival massive proprietary models in specific clinical extraction tasks when properly prompted, offering cost-effective deployment options.

Limitations:

Synthetic Data Fidelity: While high-quality, synthetic data (SYNUR) cannot fully capture the richness and variability of real clinical language.
Dataset Size: SIMORD is relatively small (64 training samples), limiting its utility for fine-tuning models.
Annotation Variability: Natural noise in annotations (e.g., span boundaries, informal phrasing) sets a ceiling on achievable performance.
Code Access: The full nursing extraction codebase cannot be shared due to proprietary constraints, though simplified prompts are provided.

In conclusion, this paper demonstrates that LLMs are ready to tackle complex, unstructured-to-structured clinical tasks, provided that robust data strategies (like synthetic generation) and tailored pipelines (like RAG and segmentation) are employed to handle the specific constraints of clinical workflows.