💬 NLP

Benchmarking and Adapting On-Device LLMs for Clinical Decision Support

This paper benchmarks and adapts various open-source on-device large language models for clinical decision support, demonstrating that they can achieve diagnostic accuracy comparable to or exceeding state-of-the-art proprietary models while offering superior privacy and resource efficiency, particularly when enhanced through fine-tuning.

Original authors: Alif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alif Munim, Jun Ma, Omar Ibrahim, Alhusain Abdalla, Shuolin Yin, Leo Chen, Bo Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant medical detective, but instead of living in a giant, cloud-based skyscraper where it needs a supercomputer to think, you want to keep this detective right in your pocket or on your local computer. This paper is about testing a new generation of these "pocket detectives" (called on-device Large Language Models) to see if they are smart enough to help doctors make decisions without needing to send sensitive patient data to the internet.

Here is the story of their journey, explained simply:

The Problem: The "Cloud" vs. The "Local"

Currently, the smartest medical AI models are like giant, expensive libraries that only exist on the internet. To use them, a doctor has to send patient notes to a remote server. This raises two big red flags:

Privacy: Sending patient data off-site can break strict hospital rules.
Cost & Access: These giant libraries require massive, expensive computers to run, which many clinics can't afford.

The open-source community tried to build smaller, local versions, but the best ones were still too heavy (like trying to carry a full encyclopedia in a backpack). This paper asked: Can we build a "detective" small enough to fit on a standard computer, but smart enough to solve medical mysteries?

The Experiment: The Three Trials

The researchers put several new "pocket detectives" (specifically models named gpt-oss, Qwen3.5, and Gemma 4) through three different tests to see how they compared to the "giant cloud libraries" (like GPT-5 and Gemini).

1. The Generalist Test (The ER Doctor)

The Task: The AI had to look at patient history and X-ray/MRI reports and pick the correct disease from a list.
The Result: The small, local detectives performed surprisingly well. One model, Gemma 4, was the star of the show, scoring 86.5% accuracy. This was better than the "mini" version of the giant cloud model (GPT-5-mini) and almost as good as the top-tier cloud model.
The Analogy: It's like a local mechanic who, without needing to call a master engineer in another country, can correctly identify 86 out of 100 car engine problems just by looking at the dashboard.

2. The Specialist Test (The Eye Doctor)

The Task: The AI had to answer tricky multiple-choice questions about eye diseases and how to treat them.
The Result: A larger local model (gpt-oss-120b) actually beat the top cloud models in this specific area. It showed that local models can be experts in specific fields, not just generalists.

3. The Judge Test (The Reviewer)

The Task: Instead of diagnosing, the AI had to act as a supervisor, grading the work of other AI models to see if their advice was good.
The Result: The local models were excellent judges. They agreed with human experts almost perfectly, showing they understand the rules of medical reasoning, not just the facts.

The Magic Trick: "Fine-Tuning"

The researchers realized that while the local models were good, they could be great. They took two of the local models and gave them a "crash course" (called fine-tuning) using thousands of past medical cases.

The Analogy: Imagine a smart student who knows general science. If you give them a specific textbook of past exam questions and answers, they don't just memorize; they learn how to think about those specific problems.
The Result: After this training, the local models jumped in performance. One model (Qwen3.5) went from being "good" to being 87.9% accurate, which is nearly identical to the most powerful, expensive cloud model (89.4%).
Key Insight: This proves you don't need a massive model to get top results; you just need a smaller model that has been specifically trained on the right data.

The "Safety Net" Analysis

The researchers also looked at the mistakes the models made. They found something very reassuring:

No Wild Guesses: When the models got it wrong, they rarely made up fake diseases (hallucinations). Instead, 87% of the time, they picked a different real disease that was actually a reasonable possibility.
The Analogy: If a detective guesses the wrong suspect, they usually guess a suspect who could have done it, rather than guessing a ghost or a tree. This means the errors are "clinically plausible," which is much safer than random guessing.

The Bottom Line

This paper claims that we no longer need to rely on giant, expensive, cloud-based AI to get high-quality medical help.

Privacy: You can run these models on a single computer in a hospital room without sending data anywhere.
Performance: With a little bit of specific training, these small models can match or beat the performance of the biggest, most expensive AI systems.
Future: This opens the door for hospitals to have their own private, powerful AI assistants that respect patient privacy and work even when the internet is down.

In short: Small, local models can now do the heavy lifting of medical diagnosis, provided they get the right training.

1. Problem Statement

While Large Language Models (LLMs) show promise in clinical decision support, their real-world deployment is hindered by three primary barriers:

Privacy and Governance: Proprietary, cloud-hosted models require transmitting sensitive patient data to external servers, violating data-governance policies in many healthcare institutions.
Resource Constraints: State-of-the-art open-source models (e.g., DeepSeek-R1 with 671B parameters) are too large for local deployment on standard clinical hardware, making fine-tuning and inference prohibitively expensive.
Cost and Control: Proprietary models lack transparency and incur high operational costs, while current open-source alternatives often require massive computational resources (multi-GPU clusters) that are unavailable in typical clinics.

The paper addresses the question: Can smaller, on-device LLMs (running on single consumer GPUs) achieve clinical reasoning performance comparable to frontier proprietary models, and can they be effectively adapted via fine-tuning for specific clinical tasks?

2. Methodology

A. Model Selection

The study benchmarks three families of on-device models against leading proprietary and open-source baselines:

On-Device Models:
- gpt-oss: 20B and 120B parameters (optimized for 16GB and 80GB consumer GPUs, respectively).
- Qwen3.5: 9B, 27B, and 35B parameters.
- Gemma 4: 31B parameters.
Baselines:
- Proprietary: GPT-5.1, GPT-5-mini, Gemini 3.1 Pro.
- Open-Source: DeepSeek-R1 (671B).

B. Benchmark Tasks & Datasets

Three representative clinical scenarios were evaluated using zero-shot inference (k=3 self-consistency):

LLM-as-a-Generalist (General Diagnosis):
- Dataset: 207 radiological case reports from the Eurorad library (2025 cases to prevent data leakage).
- Task: Select the correct diagnosis from a provided differential list based on clinical history and imaging findings.
LLM-as-a-Specialist (Ophthalmology):
- Dataset: 130 multiple-choice questions (39 diagnosis, 91 management) covering six ophthalmic sub-specialties.
- Task: Multi-label classification (select all correct answers).
LLM-as-a-Clinical-Judge:
- Dataset: 1,315 cases with human expert scores across five specialties (Internal Med, Neurology, Surgery, Gyn, Peds).
- Task: Simulate human experts to score existing diagnoses/treatment plans on a 1–5 Likert scale.

C. Adaptation Strategy (Fine-Tuning)

To test adaptability, the authors fine-tuned gpt-oss-20b and Qwen3.5-35B:

Data Curation: Used a larger, architecturally matched model from the same family (gpt-oss-120b and Qwen3.5-122B) to generate high-quality Chain-of-Thought (CoT) reasoning traces for 1,895 historical Eurorad cases.
Technique: Low-Rank Adaptation (LoRA) with specific targeting of linear layers and expert layers.
Inference Optimization:
- gpt-oss-20b: Used Group Beam Search (13 beams) to explore diverse reasoning paths.
- Qwen3.5-35B: Used standard autoregressive generation with temperature 0.7.

D. Error Analysis

Taxonomy: Incorrect predictions were classified into five categories: surface-form mismatch, same disease family, plausible differential, off-topic/hallucination, and empty/refusal.
Upper-Bound Analysis: Calculated "pass@3" accuracy to determine the theoretical ceiling if the best answer from three runs was selected.

3. Key Contributions

Comprehensive Benchmarking: The first systematic evaluation of on-device LLMs (5B–120B range) against frontier proprietary models across general diagnosis, specialty care, and clinical judgment simulation.
Adaptability Framework: Demonstrated that fine-tuning on-device models using reasoning traces from larger, family-matched models significantly boosts performance without increasing memory footprint.
Error Characterization: Provided a granular analysis showing that most errors are "plausible differentials" (clinically sound but incorrect choices) rather than hallucinations, suggesting the models are reasoning correctly but lack specific knowledge.
Privacy-Preserving Pathway: Validated that high-accuracy clinical AI can run locally on consumer hardware (single GPU), eliminating the need for cloud data transmission.

4. Key Results

Performance Comparison (Zero-Shot)

General Diagnosis:
- Gemma 4 31B achieved the highest base on-device accuracy (86.5%), outperforming GPT-5-mini (84.5%) and approaching GPT-5.1 (89.4%).
- Qwen3.5-35B reached 84.1%, matching GPT-5-mini and exceeding DeepSeek-R1 (81.6%).
- gpt-oss-20b achieved 80.7%, comparable to DeepSeek-R1 despite being ~30x smaller in parameter count.
Ophthalmology (Specialist):
- gpt-oss-120b achieved 77.7% overall accuracy, surpassing GPT-5.1 (76.9%) and GPT-5-mini (67.7%).
- Models consistently performed better on diagnosis tasks than management tasks.
Clinical Judgment:
- On-device models (Qwen3.5 and Gemma 4) achieved a median diagnostic error of 0.00, perfectly aligning with human expert consensus, matching the best proprietary models.

Impact of Fine-Tuning

gpt-oss-20b: Accuracy increased from 80.7% to 86.5%, surpassing DeepSeek-R1 and GPT-5-mini.
Qwen3.5-35B: Accuracy increased from 84.1% to 87.9%, narrowing the gap to GPT-5.1 (89.4%) to just 1.5 percentage points.
Efficiency: Fine-tuning improved accuracy without increasing the memory footprint (remained ~18–20 GB for 4-bit quantization), whereas DeepSeek-R1 requires ~386 GB.

Error Analysis Insights

Plausible Differentials: 87.2% of all errors across models were "plausible differentials" (selecting a valid but incorrect diagnosis from the list). Only 5% were surface-form mismatches.
Hallucinations: Off-topic/hallucinated errors were restricted to the smallest models (Qwen3.5 9B/27B). All models $\ge$ 20B parameters (including fine-tuned ones) produced zero off-topic errors.
Recoverable Performance: Upper-bound analysis showed that up to 93.2% accuracy is attainable for Qwen3.5-35B via better answer selection strategies.
Hard Cases: The remaining unsolved cases (approx. 11%) were rare histopathological subtypes or syndromes, representing a "knowledge frontier" rather than an architectural failure.

5. Significance

Feasibility of Local AI: The study proves that high-fidelity clinical decision support does not require massive cloud infrastructure. Compact models (5–20 GB) can rival or exceed the performance of much larger proprietary systems when properly adapted.
Privacy and Sovereignty: By enabling on-device inference, healthcare institutions can maintain strict data sovereignty, complying with regulations (e.g., HIPAA, GDPR) without compromising diagnostic quality.
Cost-Effectiveness: The ability to fine-tune small models using reasoning traces from larger family members offers a scalable, low-cost pathway for hospitals to customize AI for their specific patient demographics and disease prevalences.
Reliability: The finding that larger on-device models produce zero hallucinations and align perfectly with human expert scoring suggests they are safe candidates for "second-opinion" systems and automated auditing in clinical workflows.

In conclusion, the paper establishes that on-device LLMs are a practical, accurate, and privacy-preserving alternative to cloud-based proprietary systems for clinical decision support, provided they are selected carefully (e.g., Gemma 4, Qwen3.5) and adapted via targeted fine-tuning.