💬 NLP

Domain-Adapted Small Language Models for Reliable Clinical Triage

This study demonstrates that a domain-adapted, fine-tuned open-source small language model (Qwen2.5-7B) significantly outperforms both baseline models and advanced proprietary large language models in accurately assigning Emergency Severity Index scores from clinical triage narratives, offering a reliable and privacy-preserving decision-support tool for emergency departments.

Original authors: Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy emergency room as a giant, chaotic airport terminal. Every few minutes, a new passenger (patient) arrives with a story about why they are there. The job of the "gate agents" (nurses) is to quickly decide how urgent their trip is. They use a special five-star rating system called the Emergency Severity Index (ESI):

5 Stars: "I just need a band-aid; I can wait."
1 Star: "My plane is on fire; I need to get on the runway right now."

The problem is that the stories patients tell are messy, long, and full of confusing details. Sometimes, the gate agents get tired or overwhelmed and give the wrong rating. This can mean someone who needs immediate help waits too long, or someone with a minor issue gets rushed to the front, blocking the runway for those who truly need it.

This paper is about building a smart, lightweight assistant to help the gate agents get these ratings right, without needing a super-computer or sending patient secrets to the cloud.

The Big Idea: Small is Beautiful

The researchers asked: Do we need a giant, expensive "super-brain" (a massive AI) to do this, or can a smaller, smarter "pocket-brain" do the job just as well?

They tested several "Small Language Models" (SLMs)—think of these as compact, efficient tools that can fit on a hospital's own secure computer (like a laptop in the back office) rather than needing to call a giant server farm. This keeps patient data private and makes the tool fast.

The Experiment: How to Feed the Assistant

The team tried different ways to give information to the AI, like trying different recipes to see which one makes the best cake:

The Raw Story: Feeding the AI the messy, unedited notes the nurse wrote. (Result: The AI got confused by the noise.)
The Structured List: Feeding the AI a checklist of facts (heart rate, age, symptoms). (Result: The AI missed the "vibe" of the story.)
The "Clinical Vignette": This was the winner. The AI first read the messy story and summarized it into a short, clear, professional paragraph—like a news headline that captures the most important facts. When the AI read this clean summary, it got the rating right much more often.

The Star Player: Qwen2.5-7B

Out of all the models they tested, one stood out: Qwen2.5-7B.

Why it won: It was the perfect balance. It was fast (making a decision in less than a second, faster than a blink), accurate, and didn't "hallucinate" (make up fake medical facts).
The Training: To make it even better, the researchers gave it a "crash course" using thousands of real, anonymized examples from a children's hospital. They didn't just show it the rules; they showed it real-world scenarios. This is like taking a student pilot and letting them practice on thousands of simulated flights before letting them fly a real plane.

The Results: A Clear Winner

After this training, the Qwen2.5-7B model became a triage expert:

Fewer Mistakes: It made far fewer errors than the untrained models and even beat some of the giant, expensive "super-brains" (like GPT-4o) that are usually considered the best.
Safety First: It was particularly good at avoiding "under-triage" (missing a serious case).
Speed: It was so fast it could keep up with the busiest emergency room without slowing anyone down.

What Didn't Work

The researchers also tried some fancy tricks, but they didn't help:

The "Committee" Approach: They tried having multiple AI agents debate the answer and vote on the best one. Instead of getting smarter, this just made the system slower and more confused.
The "Library" Approach: They tried giving the AI a digital copy of the ESI rulebook to look up answers while it worked. This actually made things worse, adding confusion and slowing the system down. The paper suggests that for this specific job, a well-trained brain is better than a brain that has to stop and look things up.

The Bottom Line

This study shows that you don't need a massive, expensive, cloud-based AI to help run an emergency room. A small, specialized, and carefully trained AI model can live right on the hospital's own computers. It can read messy notes, summarize them into clear stories, and help nurses decide who needs help first, all while keeping patient data safe and private.

Important Note: The paper emphasizes that this was tested on retrospective data (looking back at old records) in a children's hospital. While the results are promising, the study did not test this system in a live, real-time emergency room with real patients yet. The goal was to prove the technology works, not to say it's ready to replace nurses tomorrow.

1. Problem Statement

Accurate assignment of the Emergency Severity Index (ESI) in emergency departments (EDs) is critical for patient outcomes and resource allocation but remains highly variable and subjective.

Current Challenges: Human triage accuracy is low (often 30–50%), leading to undertriage (delaying life-saving care) or overtriage (overburdening resources).
Limitations of Existing AI: While Large Language Models (LLMs) show promise, proprietary models raise HIPAA/privacy concerns due to external API calls, require massive compute resources, and often produce inconsistent outputs. Furthermore, complex inference strategies (like multi-agent debates or Retrieval-Augmented Generation) have not been proven to improve performance on rule-based clinical tasks.
The Gap: There is a need for privacy-preserving, on-premise, and computationally efficient decision-support tools that can reliably automate ESI assignment without compromising safety or reproducibility.

2. Methodology

The study proposes a framework for adapting Small Language Models (SLMs) (≤7B parameters) to clinical triage using a specific data pipeline and fine-tuning strategy.

Data Sources & Preparation

Dataset: Retrospective data from Children's National Hospital (CNH) pediatric ED (approx. 117k encounters).
Filtering: A high-quality subset ( $n=353$ ) was curated for evaluation, requiring complete Chief Complaints, Vital Signs, Physical Exams, and Past Medical History.
Training Data:
1. Expert-Curated: 245 vignettes from the ESI Handbook (Gold Standard).
2. Silver-Standard Dataset: 117,247 clinical vignettes automatically generated by prompting Qwen2.5-7B to summarize structured triage fields, retaining the nurse-assigned ESI as the ground truth.

Model Selection & Architecture

Base Models: Evaluated multiple open-source SLMs (Qwen2.5, MedGemma, MedLLaMA2) and larger models (GPT-OSS).
Target Model: Qwen2.5-7B was selected as the primary candidate due to its balance of performance and efficiency.
Fine-Tuning Strategy:
- QLoRA (Quantized Low-Rank Adaptation): Used for memory-efficient training on multi-GPU setups.
- Sequential Chunking: The silver dataset was split into 10 chunks. The model was trained sequentially on each chunk (Chunk 1 $\to$ Chunk 10) to monitor convergence and stability.
- Input Formats: Tested six prompting pipelines, including raw notes, structured data, and clinical vignettes (concise summaries of patient narratives).

Evaluation Metrics

Performance was measured against nurse-assigned ESI levels using:

Total Discordance: Overall misclassification rate.
Undertriage/Overtriage: Directional errors.
Significant Errors: Critical misclassifications (e.g., predicting ESI 3–5 for a true ESI 2, or ESI 1–2 for a true ESI 3–5).
Inference Time: Latency per patient encounter.

Exploratory Experiments

Explainability: Token-level attention maps were analyzed to identify which clinical cues (e.g., respiratory distress vs. administrative terms) drove predictions.
Multi-Agent Systems: Tested ensembles of 3–4 agents with different personas (safety-first, guideline-strict) using voting and debate mechanisms.
Retrieval-Augmented Generation (RAG): Tested injecting the ESI Handbook into the context window.
Robustness: Evaluated performance when key data (Vital Signs or Physical Exam) was artificially removed.

3. Key Contributions

Demonstration of SLM Viability: Proved that a 7B-parameter open-source model, when domain-adapted, can outperform larger proprietary LLMs (e.g., GPT-4o) and other SLMs in clinical triage.
Optimization of Input Representation: Identified that clinical vignettes (concise summaries) are the superior input format, significantly outperforming raw triage notes or structured data alone.
Superiority of Fine-Tuning over Complex Inference: Showed that targeted fine-tuning on institution-specific data yields better results than complex inference strategies like multi-agent debates or RAG, which often introduced noise and latency.
Privacy-Preserving Deployment: Established a pathway for on-premise, HIPAA-compliant triage support using lightweight models that run in sub-second latency.

4. Key Results

Model Performance Comparison

Best Model: Fine-tuned Qwen2.5-7B achieved the best balance of accuracy and speed.
- Total Discordance: Reduced to 25.85% (compared to ~50–60% for base models and ~57% for GPT-4o).
- Significant Errors: Achieved low rates of significant undertriage (6.25%) and overtriage (2.56%).
- Inference Speed: 0.15–0.16 seconds per encounter, making it suitable for real-time workflows.
Comparison to Larger Models: Larger models (e.g., GPT-OSS-120B, MedGemma-27B) offered no accuracy gains but were 20–150x slower. GPT-4o had a discordance of ~57% and took ~6 seconds per encounter.

Impact of Strategies

Fine-Tuning vs. RAG: Fine-tuning alone reduced discordance significantly. Adding RAG (retrieving ESI guidelines) increased error rates and latency, suggesting that for rule-based tasks, internalized knowledge via fine-tuning is superior to external retrieval.
Multi-Agent Systems: Ensembles (3–4 agents) degraded performance (discordance rose to ~33–43%) and increased latency (21–56s), likely due to conflicting reasoning and variability.
Robustness to Missing Data: The fine-tuned Qwen2.5-7B maintained low discordance (~29%) even when Vital Signs or Physical Exams were missing, outperforming both base Qwen and GPT-4o in incomplete data scenarios.

Explainability Findings

Correct Predictions: Attention focused on clinically meaningful markers (respiratory rates, pain descriptors, age-specific context).
Incorrect Predictions: Attention drifted toward administrative terms ("management," "process") or superficial lexical cues, indicating areas for future improvement in training data curation.

5. Significance and Conclusion

This study establishes that domain-adapted Small Language Models are a practical, safe, and highly effective solution for clinical triage support.

Clinical Impact: By reducing discordance and significant errors, these models can assist nurses in standardizing ESI assignments, potentially reducing delays in care for high-acuity patients.
Operational Feasibility: The sub-second inference time and on-premise deployment capability address the critical barriers of latency and data privacy that have hindered the adoption of LLMs in healthcare.
Strategic Insight: The findings suggest that for structured, rule-based medical tasks, high-quality fine-tuning on institution-specific data is more effective than scaling model size or employing complex inference architectures.

Future Work: The authors recommend expanding to real-time, noisy data, including excluded populations (mental health, ESI 1), and developing outcome-anchored benchmarks to validate triage accuracy against actual patient outcomes rather than just nurse assignments.