On the robustness of medical term representations in locally deployable language models

This study evaluates the representational robustness of 15 locally deployable large language models on neurological terminology, revealing that while performance generally scales with model size, neither size nor medical fine-tuning guarantees clinical reliability due to significant variations based on terminological complexity and subdomain.

The Gist

Imagine you are trying to build a tiny, private library inside your own hospital basement. You want this library to hold all the medical knowledge needed to help doctors, but you can't connect it to the giant, public internet because patient privacy laws (like HIPAA) forbid it.

To make this work, you need to shrink the "brain" of the computer (the AI model) down so it fits on a standard server, rather than needing a massive supercomputer. But here's the scary question: If you shrink the brain, does it forget the important stuff?

This paper is like a rigorous stress test for 15 different "shrunken brains" (AI models) to see if they can handle complex medical terms without getting confused.

Here is the breakdown of what they found, using some everyday analogies:

1. The Test: "The Medical Logic Puzzle"

Instead of just asking the AI, "What is a headache?" (which is easy), the researchers gave them a tricky logic puzzle.

• The Setup: They gave the AI a medical term (like "Miller-Fisher syndrome"), its parent category ("a type of Guillain-Barré syndrome"), and a "distractor" (a different disease that sounds similar, like "Charcot-Marie-Tooth").
• The Challenge: The AI had to answer four specific questions correctly:
  1. Is the child part of the parent? (Yes)
  2. Is the parent part of the child? (No)
  3. Is the child the same as the distractor? (No)
  4. Is the distractor the same as the child? (No)
• The Rule: If the AI got any of these four wrong, it failed. This proves the AI isn't just guessing or recognizing patterns; it actually understands the relationship.

2. The Big Myth: "Bigger Isn't Always Better"

You might assume that a 70-billion-parameter model (a giant brain) would always beat a 20-billion-parameter model (a medium brain).

• The Reality: Not necessarily.
• The Analogy: Think of it like hiring a librarian. You might assume the "Giant Library" (70B model) knows more than the "Medium Library" (20B model). But the study found that a specific Medium Library (GPT-OSS 20B) actually knew the medical terms better than the Giant Library and even better than a specialized "Medical Library" that had been trained specifically on medical books.
• The Lesson: Just because an AI is huge or has been "medical-fine-tuned" doesn't mean it's safe for clinical use. Sometimes, a well-architected medium-sized model is smarter than a giant, clunky one.

3. The "Complexity Trap"

The researchers invented a "Complexity Score" (SCI) to measure how hard a word is.

• Easy Words: "Headache" or "Fever." (Highly common, low ambiguity).
• Hard Words: Rare, specific neurological syndromes with confusing names.
• The Trap: Most of the smaller AI models were like amateur chefs. They could cook a perfect burger (easy terms) but would burn the house down if you asked them to make a complex soufflé (rare medical terms). Their performance crashed hard when the words got difficult.
• The Winners: Only a few models (the "Master Chefs") could handle both the burger and the soufflé without failing. They maintained their accuracy even when the terms got very complex.

4. The "Specialist Training" Surprise

The team tested if giving the AI extra "medical school" training (fine-tuning) helped.

• The Tiny Brain (4B): It was like a toddler going to medical school. No matter how much they studied, they were still too small to understand the concepts. The extra training did nothing.
• The Medium Brain (27B): This was like a medical student. The extra training helped them significantly, boosting their accuracy from "okay" to "very good."
• The Lesson: You can't just "patch" a tiny AI with medical data and expect it to work. It needs to be big enough to hold that knowledge in the first place.

5. The "Diagnosis" vs. "Anatomy" Bias

The study found that the AIs were better at some topics than others.

• They were great at Diagnoses (naming a disease).
• They were terrible at Anatomy (naming specific body parts) and Symptoms.
• The Analogy: It's like a student who is great at memorizing the names of famous movies but gets confused when asked to describe the plot or the actors. If you use this AI to diagnose a patient, it might get lucky. If you use it to describe where the pain is or what the symptoms mean, it might hallucinate nonsense.

The Bottom Line for the Real World

If you are a hospital trying to run AI on your own servers to keep patient data safe:

1. Don't just pick the biggest model. Size doesn't guarantee safety.
2. Don't assume "Medical Training" fixes everything. If the model is too small, the training is wasted.
3. Test before you trust. You need to check if the AI can handle the hard words, not just the easy ones. If it fails on complex terms, it's a ticking time bomb for clinical errors.

In short: A small, smart, and well-tested AI is safer for your hospital than a giant, untested one. Don't let the "bigger is better" marketing fool you; in medicine, reliability is everything.

Technical Summary

1. Problem Statement

The healthcare sector is increasingly interested in locally deployable (on-premises) Large Language Models (LLMs) to ensure patient data privacy and compliance with regulations like HIPAA and GDPR. However, local deployment necessitates the use of lightweight models (typically 4B to 120B parameters) that can run on standard hardware, rather than massive frontier models.

The core problem addressed is whether these smaller, locally deployable models possess robust representations of medical terminology. While they may exhibit linguistic fluency, they might lack the precise, directional understanding of relationships between medical terms required for clinical safety. Current benchmarks often rely on multiple-choice questions (e.g., MedQA), which may not detect "shortcut learning" or superficial statistical associations. The authors argue that without verifying the foundational robustness of term representations, deploying these models in clinical settings poses significant safety risks, particularly for terms with low lexical frequency, high ambiguity, or low societal prominence.

2. Methodology

Dataset and Evaluation Framework

• Test Set: A custom dataset of 250 clinical neurology term triplets was created. Each triplet consists of a Child Term (A), a Parent Category (B), and a Distractor Term (C).
  • Example: (Miller-Fisher syndrome, Guillain-Barré variant, Charcot-Marie-Tooth variant).
• Robustness Definition: A term is considered "robustly represented" only if the model correctly answers four logical relationships for a given triplet:
  1. Affirmation: Correctly identifies that B is a parent of A.
  2. Reverse Rejection: Correctly identifies that A is not the parent of B.
  3. Distractor Rejection: Correctly identifies that A is not a variant of C.
  4. Reverse Distractor Rejection: Correctly identifies that C is not a parent of A.
• Protocol: Models were tested using a strict zero-shot protocol with three prompt variations per triplet to control for prompt fragility. This resulted in 750 unique evaluations per model.
• Models Evaluated: 15 open-weights LLMs ranging from 4B to 120B parameters, including general-purpose models (e.g., Llama, Qwen, Mistral) and medically fine-tuned variants (MedGemma). A frontier model (Google Gemini 3 Pro) served as a reference ceiling.

Semantic Complexity Index (SCI)

To quantify the difficulty of medical terms, the authors developed a novel composite metric, the Semantic Complexity Index (SCI), ranging from 0.2 (low complexity) to 0.7 (high complexity). It integrates four normalized variables:

1. Societal Prominence: Log-transformed annual Wikipedia views (inverted).
2. Lexical Rarity: Zipf frequency scores (inverted).
3. Semantic Ambiguity: Number of WordNet senses (polysemy).
4. Computational Fragmentation: Token count relative to a 10-token limit.

Statistical Analysis

• Correlation: Pearson correlation between robustness rates and log-transformed parameter counts.
• Subdomain Analysis: One-way ANOVA across five clinical subdomains (localisation, clinical features, investigations, diagnoses, treatments).
• Complexity Modeling: Locally Weighted Scatterplot Smoothing (LOWESS) to analyze performance degradation as SCI increases.
• Fine-tuning Impact: Chi-squared tests and Z-tests to compare base models vs. medically fine-tuned variants across complexity strata.

3. Key Contributions

1. Novel Evaluation Metric: Introduced a rigorous, logic-based "robust representation" test that moves beyond multiple-choice benchmarks to verify directional understanding of medical hierarchies.
2. Semantic Complexity Index (SCI): Developed a transferable metric to quantify the intrinsic difficulty of medical terms based on linguistic and societal properties.
3. Empirical Evidence on Scaling: Provided data challenging the assumption that larger models or medical fine-tuning automatically guarantee clinical safety in local deployment scenarios.

4. Key Results

A. Scaling Laws and Exceptions

• General Trend: Representational robustness follows a log-linear scaling law with model size ($r=0.736, p=0.002$).
• Notable Deviations:
  • GPT-OSS 20B achieved 84.2% robustness, outperforming significantly larger models (e.g., 70B+ models) and even some medically fine-tuned models.
  • Qwen 3 (32B) also showed exceptional performance (80.5%), exceeding the 70B+ baseline.
  • This suggests that architectural optimization and training quality can outweigh raw parameter count.

B. Impact of Medical Fine-Tuning

• Small Models (4B): Fine-tuning provided no significant benefit (14.7% vs. 15.7%, $p=0.67$). These models hit a performance floor.
• Medium Models (27B): Fine-tuning yielded significant gains, increasing robustness from 38.2% to 62.7% ($p<0.0001$).
• Complexity Interaction: The gains from fine-tuning were consistent across both low and high-complexity terms, indicating that fine-tuning is only effective when the base model has sufficient capacity to integrate specialized knowledge.

C. Semantic Complexity and "Complexity Invariance"

• Performance Degradation: Most local LLMs (12B–110B) exhibited a sharp decline in accuracy as SCI increased. For example, Llama 3.3 (70B) dropped from 70.6% on low-complexity terms to 30.8% on high-complexity terms.
• Complexity Invariance: Only the frontier model (Gemini 3 Pro) and GPT-OSS (20B and 120B) maintained "complexity invariance," with performance declines of less than 20% across the full SCI spectrum. GPT-OSS 20B maintained >80% accuracy even on the most complex terms.

D. Subdomain Variance

• Robustness varied significantly by clinical subdomain ($F=4.69, p=0.003$).
• Diagnoses were the most robustly represented (73.8%), significantly outperforming Localisation (47.9%) and Clinical Features (52.1%).
• This indicates that models struggle more with anatomical and symptom-based terminology than with diagnostic labels.

5. Significance and Implications

• Safety Warning: Neither model size nor medical fine-tuning is a reliable proxy for clinical safety. A larger model or a fine-tuned model may still fail catastrophically on complex, rare, or ambiguous medical terms.
• Deployment Strategy: Healthcare organizations cannot assume that "larger is better" for local deployment. The GPT-OSS 20B model demonstrated that a mid-sized, well-optimized generalist model can outperform larger, specialized counterparts in specific robustness metrics.
• Validation Framework: The authors propose that term-level robustness validation using complexity-aware frameworks (like the SCI) is a prerequisite for clinical adoption. Models should be validated against the specific complexity profile of their intended use case.
• Foundational Robustness: High-level clinical reasoning benchmarks may mask "shortcut learning." True clinical safety requires the "atomic" stability of term representations, which is currently fragile in most lightweight local LLMs when faced with high-complexity terminology.

In conclusion, the paper argues that safe local deployment requires moving beyond simple parameter counting and fine-tuning assumptions, advocating instead for rigorous, complexity-aware validation of term representations before clinical integration.