What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

Technical Summary: What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

Problem Statement
While multiple-choice question-answering (QA) datasets are standard for benchmarking Large Language Models (LLMs) in medicine, existing resources (e.g., MedQA, MedMCQA) lack fine-grained clinical specialty annotations. This limitation hinders the evaluation of LLMs in specialized hospital settings where performance within specific domains (e.g., cardiology vs. neurology) is critical. Furthermore, it remains unclear whether fine-tuning medical LLMs on specific specialty data injects new domain knowledge or merely shifts the model's distribution toward the medical domain.

Methodology
The authors introduce S-MedQA, a new English medical QA dataset designed to address the lack of specialty-level granularity. The construction and evaluation process involves three main stages:

1. Dataset Construction (S-MedQA):

   • Source: Derived from MedQA and MedMCQA, covering 15 clinical specialties (e.g., Cardiology, Neurology, Oncology) with over 24,000 QA pairs.
   • Annotation Strategy: A hybrid approach combining machine and expert verification.
     • Single-Specialty Labeling: GPT-3.5 generates predictions using five distinct prompts. A majority voting mechanism (requiring 3+ agreements) assigns a primary specialty. Human validation on a held-out set confirms accuracy between 90.8% and 97.8%.
     • Multi-Specialty Labeling: To handle cross-disciplinary questions, the authors employ a conformal prediction procedure. This method calibrates an LLM to predict multiple labels with coverage guarantees, achieving a set precision of 0.69 and recall of 0.52 against expert annotations.
   • Validation: Inter-annotator agreement (Krippendorff's alpha) among medical experts and students reached 83.6%.

2. Experimental Setup:

   • Models: Eight open-source LLMs were evaluated, including Llama-2, Llama-3, Mistral, Bio-Medical-LLaMA, and OLMo.
   • Fine-tuning: Models were fine-tuned using LoRA on training data from individual specialties (and a combined dataset).
   • Evaluation: To mitigate position bias, answer options were shuffled five times per sample. A post-hoc classifier (trained on Mistral-v0.2) was used to match model outputs to options, achieving 96.5% accuracy, rather than relying on first-token probabilities or simple string matching.

3. Analysis Techniques:

   • Cross-Specialty Evaluation: Models fine-tuned on one specialty were tested on all others to measure knowledge transfer.
   • Term Overlap Analysis: Using SNOMED-CT and HPO, the authors quantified the overlap of clinically relevant terms (disorders, findings, procedures) between train and test sets within and across specialties.
   • Token Probability Analysis: The study measured changes in the log-probabilities of clinically relevant tokens before and after fine-tuning to distinguish between domain shifting and knowledge injection.

Key Results

• Counter-Intuitive Transfer Patterns: Contrary to the expectation that fine-tuning on a specific specialty yields the best performance on that same specialty, the results showed that models fine-tuned on unrelated specialties often performed best. For instance, a model fine-tuned on Neurology data achieved the highest accuracy on the Cardiology test set. No model achieved its peak performance on the "diagonal" (training and testing on the same specialty).
• Limited Terminology Overlap: Analysis revealed that while training sets cover ~63.4% of the clinical terminology found in their corresponding test sets, the overlap across different specialties is only ~32.8%. This confirms that cross-specialty evaluations are not suffering from significant "knowledge leakage" due to shared terminology.
• Domain Shifting vs. Knowledge Injection: Token probability analysis showed that fine-tuning on medical data consistently increased the log-probabilities of clinically relevant terms across all specialties, regardless of the specific training specialty. Crucially, fine-tuning on non-medical data (social sciences from MMLU) resulted in a significant drop in these probabilities. This suggests that performance gains are driven primarily by domain shifting (aligning the model from general to medical) rather than the injection of specific, fine-grained clinical knowledge.

Significance and Contributions
The paper makes four primary contributions:

1. S-MedQA Release: The introduction of the first English medical QA dataset with high-quality, human-validated single and multi-specialty annotations, enabling fine-grained benchmarking.
2. Methodological Rigor: The establishment of a robust evaluation framework that accounts for cross-disciplinary questions and mitigates evaluation biases (e.g., answer ordering, token probability selection bias).
3. Empirical Evidence on Fine-Tuning: The study provides evidence challenging the assumption that fine-tuning injects specific specialty knowledge. Instead, it supports the hypothesis (aligned with Zhou et al., 2024) that improvements in medical QA tasks are largely derived from domain shifting.
4. Community Resource: The release of the dataset and all reproduction code to facilitate further research in clinical NLP.

Conclusion
The authors conclude that while clinical specialties possess distinct terminologies, the performance gains observed in medical LLMs are not necessarily due to the acquisition of new, specialty-specific facts during fine-tuning. Rather, the process appears to function as a domain shift, recalibrating the model's probability distribution toward the medical context. The paper calls for a rethinking of the role of fine-tuning data in the medical domain and suggests that future research should investigate the specific impact of QA complexity and difficulty on these transfer patterns.