🤖 machine learning

Safety and accuracy follow different scaling laws in clinical large language models

The paper introduces the SaFE-Scale framework and RadSaFE-200 benchmark to demonstrate that clinical LLM safety is not a passive byproduct of scaling but a deployment-dependent property where high-quality evidence significantly reduces dangerous errors, whereas standard retrieval and increased compute fail to replicate these safety gains.

Original authors: Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Bigger Isn't Always Safer

Imagine you are hiring a team of medical students to help diagnose patients. You might assume that if you hire the "smartest" student (the largest AI model) or give them a massive library to read (more data), they will automatically make fewer dangerous mistakes.

This paper argues that this assumption is wrong. In the high-stakes world of medicine, simply making a model "smarter" or "bigger" doesn't guarantee it will be safe. A model can be very accurate on average but still make a few terrifyingly confident, dangerous mistakes.

The researchers built a new testing ground called RadSaFE-200 (think of it as a "Safety Driving Test" for medical AI) to see what actually makes these AI systems safe. They tested 34 different AI models under six different scenarios.

The Six Scenarios: How We Tested the AI

The researchers didn't just ask the AI questions; they changed how the AI was allowed to answer. Imagine these as six different ways a student might take a test:

Closed-Book: The student has to answer from memory alone. No notes allowed.
Clean Evidence: The student is given a perfect, clear, doctor-written summary of the facts right before the test.
Conflict Evidence: The student is given the perfect summary, but someone sneaks in one confusing or contradictory sentence.
Standard Search (RAG): The student is allowed to look up answers in a messy, uncurated encyclopedia (Radiopaedia) but doesn't know how to filter the noise.
Agent Search (Agentic RAG): The student has a smart assistant who searches the encyclopedia, reads the results, and summarizes them for the student.
Max Context: The student is given the entire encyclopedia dumped into their lap, hoping they can find the right page amidst the chaos.

The Surprising Results

1. The "Clean Evidence" Superpower

The biggest discovery was that giving the AI high-quality, doctor-written facts was the only thing that truly fixed the safety problems.

The Analogy: Imagine a student taking a test. If you give them a cheat sheet written by a Nobel Prize-winning doctor, they get almost everything right and rarely make dangerous mistakes.
The Data: When the AI got this "Clean Evidence," accuracy jumped from 73.5% to 94.1%. More importantly, dangerous errors dropped from 12% to just 2.6%.

2. The "Search" Trap

The researchers thought that giving the AI access to the internet (Search/RAG) or a smart assistant (Agent) would make it safer.

The Analogy: It's like giving a student a messy library. They might find the right book, but they might also read a confusing blog post and get confident about the wrong answer.
The Result: The "Search" methods improved accuracy slightly, but they did not fix the safety issues. The AI still made dangerous mistakes, and sometimes it became more overconfident about those wrong answers.

3. The "Big Brain" Myth

They tested models ranging from tiny to massive (some with hundreds of billions of parameters).

The Analogy: You might think a genius PhD student is safer than a high schooler. But in this test, the PhD student still made the same specific, dangerous mistakes as the high schooler if they didn't have the right notes.
The Result: Making the model bigger didn't automatically make it safer. The quality of the information given to the model mattered far more than the size of the model's brain.

4. The Confidence Trap

One of the scariest findings was about confidence.

The Analogy: Imagine a student who gets a question wrong but says, "I am 100% sure I am right!" with a straight face.
The Result: The AI was often dangerously overconfident when it was wrong. It didn't matter if the model was small or large; when it made a high-risk medical error, it was usually very confident about it. You cannot trust the AI's "confidence meter" to tell you if it's safe.

5. The "Groupthink" Problem

They tried putting three different AIs together to vote on an answer (an "Ensemble"), hoping they would catch each other's mistakes.

The Analogy: It's like a committee of three doctors. You'd think if one is wrong, the others would correct them.
The Result: Sometimes, all three AIs made the exact same wrong mistake at the same time. This is called "synchronized failure." When a group of AIs agrees on a wrong answer, it can trick humans into thinking the answer is definitely correct, making the situation even more dangerous.

The Bottom Line

The paper concludes that safety is not a passive result of making AI bigger or faster.

Scaling up (making models bigger) helps a little, but it doesn't solve the safety problem.
Adding more compute (letting the AI think longer or vote multiple times) doesn't fix the core issues.
The only thing that worked: Feeding the AI clean, high-quality, doctor-curated evidence right before it answers.

The Takeaway: If you want a safe medical AI, don't just buy the biggest model. Instead, focus on building a system that feeds it the best, most reliable information possible. The quality of the "notes" the AI reads is more important than the size of the "student" reading them.

Technical Summary: Safety and Accuracy Follow Different Scaling Laws in Clinical Large Language Models

Problem Statement
Clinical Large Language Models (LLMs) are increasingly deployed for decision support, often under the assumption that scaling model size, context length, or inference-time compute will inherently improve safety. However, in clinical domains like radiology, safety is not merely a function of average accuracy. A model may achieve high accuracy while retaining a small subset of "high-risk" errors—incorrect answers that could alter clinical management, contradict evidence, or are delivered with dangerous overconfidence. Conventional benchmarks often treat all incorrect answers as equivalent, failing to distinguish between benign errors and those with severe clinical consequences. The authors posit that safety-relevant errors are sparse, asymmetric, and clinically structured, meaning that scaling strategies that improve average performance may not mitigate these specific, high-stakes failure modes.

Methodology
To address this gap, the authors introduce SaFE-Scale (Safety-Focused Evaluation of Scaling), a framework for measuring how clinical LLM safety changes across model scale, evidence quality, retrieval strategy, context exposure, and inference-time compute.

Benchmark Construction (RadSaFE-200): The study utilizes RadSaFE-200, a radiology safety-focused evaluation benchmark comprising 200 multiple-choice questions. The dataset is pooled from previously published sets (RadioRAG, RaR) and newly curated cases. Crucially, each question includes:
- Clinician-defined "clean evidence" (concise, correct clinical reasoning).
- "Conflict evidence" (clean evidence plus a distracting or partially conflicting sentence).
- Option-level safety labels indicating whether a specific answer choice constitutes a high-risk error (plausible clinical harm), an unsafe answer (contradicts standard of care), or an evidence contradiction.
Experimental Design: The authors evaluated 34 locally deployed LLMs across seven families (Qwen, Llama, Gemma, MedGemma, DeepSeek, Mistral, OpenAI-OSS) spanning 0.5B to 685B parameters. The evaluation followed a fully factorial design across six deployment conditions:
- Closed-book: Zero-shot prompting without external evidence.
- Clean evidence: Prompting with curated, clinician-written evidence.
- Conflict evidence: Prompting with evidence containing a distractor.
- Standard RAG: Retrieval-Augmented Generation using a fixed pipeline and Radiopaedia as the source.
- Agentic RAG: Retrieval-and-Reasoning (RaR) framework involving multi-step query reformulation and synthesis.
- Max-context: Prompting with the maximum possible context window.
Secondary experiments investigated inference-time compute via self-consistency (majority voting over 20 samples) and fixed three-model ensembles (majority voting across heterogeneous models).
Metrics: Primary outcomes included high-risk error rate, unsafe answer rate, contradiction rate, and dangerous overconfidence (defined as an incorrect, high-risk answer with entropy-normalized confidence $\ge$ 0.80). Accuracy was reported as a secondary endpoint.

Key Results
The study demonstrates that safety and accuracy decouple significantly across deployment conditions:

Evidence Quality Dominates: The most significant improvement in both accuracy and safety came from providing clean, clinician-written evidence. This shifted the model-averaged accuracy from 73.5% (closed-book) to 94.1% and reduced high-risk errors from 12.0% to 2.6%. Dangerous overconfidence dropped from 8.0% to 1.6%.
Retrieval and Agentic Reasoning Limitations: While Standard RAG and Agentic RAG improved accuracy over closed-book prompting (76.0% and 78.1%, respectively), they failed to reproduce the safety profile of clean evidence. Agentic RAG, despite improving accuracy over Standard RAG, resulted in higher dangerous overconfidence (8.0% vs. 5.7%), indicating that improved evidence integration does not automatically eliminate confident, high-risk errors.
Scaling and Compute Inefficacy:
- Model Scale: Within model families, larger parameters improved closed-book baselines but did not eliminate high-risk errors. Clean evidence compressed the variance between families, suggesting evidence quality is a stronger determinant of safety than model size.
- Context Length: Max-context prompting increased latency (27.0s vs. 18.0s) without closing the safety gap, yielding accuracy similar to closed-book prompting (74.0%).
- Inference-Time Compute: Self-consistency provided only marginal gains in accuracy (+0.39%) and safety, failing to significantly reduce high-risk errors.
- Ensembles: While ensembles improved aggregate accuracy relative to the panel average, they introduced synchronized failure, where all three members selected the same wrong option. Many ensemble high-risk errors were unanimous, and ensembles did not consistently outperform the safest individual member.
Confidence as a Signal: Confidence was found to be an unreliable safety filter. Models maintained high confidence (often >85%) even when producing high-risk or unsafe errors. The gap between confidence on correct answers and high-risk errors was too small to serve as a deployment filter.
Worst-Case Concentration: Clinically consequential failures were not random; they concentrated in a small subset of questions. The top 15 highest-risk questions produced high-risk errors in >50% of the model panel, with some questions causing 100% failure rates across all 34 models.

Significance and Claims
The paper claims that clinical LLM safety is not a passive consequence of scaling (model size, context, or compute) but a deployment property shaped by evidence quality, retrieval design, and context construction.

Decoupling of Metrics: The authors argue that accuracy scaling laws do not apply to safety. A system can become more accurate while remaining unsafe if the deployment strategy (e.g., retrieval) does not address the specific nature of high-risk errors.
Evidence is Critical: The primary lever for safety is the quality of the information provided to the model. Curated, clinician-written evidence is the only condition tested that consistently moved models into a "safe deployment region" (accuracy $\ge$ 90%, high-risk error $\le$ 5%).
Re-evaluating Deployment Strategies: The findings suggest that relying on retrieval, agentic reasoning, or ensembling alone is insufficient for clinical safety. These methods may improve average performance but leave residual, high-confidence errors that pose clinical risks.
Evaluation Shift: The paper advocates for a shift in clinical LLM evaluation from single-metric accuracy reporting to multi-dimensional safety profiling that explicitly measures high-risk errors, contradictions, and synchronized failures. RadSaFE-200 and SaFE-Scale are presented as instruments to enable this type of granular, deployment-specific safety analysis.

The authors conclude that safer clinical AI systems require explicit safety measurement at the deployment level, prioritizing the reduction of high-risk errors through evidence quality controls rather than assuming safety will emerge from scaling model parameters or inference-time compute.