💬 NLP

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

This paper introduces a practical, open-source evaluation framework that assesses small, locally-deployable LLMs for medical question answering by prioritizing output reproducibility alongside accuracy, revealing that even clinically fine-tuned models exhibit significant inconsistency across repeated runs—a critical safety gap overlooked by traditional single-pass benchmarks.

Original authors: Avi-ad Avraam Buskila

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Avi-ad Avraam Buskila

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a digital medical assistant to help people on online health forums. You want this assistant to be smart, but more importantly, you want it to be reliable.

This paper is like a safety inspector checking out three different "small" AI brains to see if they are ready for the job. The inspector realizes that being "smart on average" isn't enough. If an AI gives you the right answer today but a completely different (and potentially dangerous) answer tomorrow for the exact same question, it's not a doctor; it's a coin flip.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Fickle Chef"

Most people test AI by asking it a question once and checking if the answer is right. The authors say this is like hiring a chef, asking them to cook a steak, and only tasting it once. If the steak is good that one time, you hire them.

But what if that chef is a "fickle chef"?

Monday: They make a perfect medium-rare steak.
Tuesday: They burn it to a crisp.
Wednesday: They serve you a raw chicken breast instead.

If you only tasted the steak on Monday, you'd think the chef is great. But in a hospital, you can't afford a chef who changes their mind every time you order. The paper argues that for medical AI, consistency is just as important as correctness.

2. The Experiment: The "10-Try Test"

To test this, the researchers didn't just ask the AI one question once. They took 50 common medical questions (like "What are the symptoms of the flu?") and asked three different AI models the same question 10 times each.

Think of it like rolling a die 10 times. If it's a fair die, you expect a mix of numbers. But if you are testing a "reliable" die, you want it to land on the same number every time.

They tested three models:

Llama 3.1 (8B): A standard, general-purpose AI.
Gemma 3 (12B): A slightly larger, general-purpose AI.
MedGemma (4B): A smaller AI that was specifically "trained" on medical books and papers to be a doctor.

3. The Shocking Results: "The Chameleon Effect"

The results were surprising, even when the researchers told the AI to be very strict and not "creative" (a setting called "low temperature").

The "Unique" Problem: Even with strict settings, the AI models were like chameleons. Every single time they answered a question, they gave a slightly different version of the answer.
- For the smallest model (MedGemma), 93% of the answers were unique.
- For the largest model (Llama), 97% of the answers were unique.
- Translation: If you asked the AI the same question 10 times, you would get 10 different answers 9 out of 10 times.
The "Specialist" Surprise: You might think the model trained specifically on medicine (MedGemma) would be the best. But it actually performed the worst. It gave lower-quality answers and was even less consistent than the general models.
- Why? The researchers realized this might be unfair. MedGemma is tiny (4 billion "brain cells") compared to the others (8 and 12 billion). It's like comparing a smart 4th grader who read a medical textbook to a 12th grader who just read a lot of general books. The 12th grader (larger model) might still be smarter just because they have a bigger brain, not because they lack medical training.

4. The "Judge" vs. The "Scorecard"

The researchers used two ways to grade the answers:

The Scorecard (Lexical Metrics): Did the AI use the exact same words as the textbook? (Most AIs failed this because they rephrased things).
The Judge (LLM-as-Judge): They used another AI to read the answer and say, "Is this medically safe and correct?"
- The Catch: The "Judge" AI was also a bit fickle. Since they only asked the Judge once per answer, the scores might have some noise, but it still showed that the medical specialist model was struggling.

5. The Big Takeaway: "Don't Trust a Single Shot"

The main lesson for anyone wanting to use AI in healthcare is this: You cannot trust a single answer.

If you deploy an AI to help doctors or patients, and you only ask it once, you are gambling. The paper suggests that if you must use these models, you need a safety net:

The "Committee" Approach: Ask the AI the same question 10 times and take the answer that appears most often (majority vote).
The "Human Check": Always have a real human doctor review the AI's answer before showing it to a patient.

Summary in a Nutshell

This paper is a warning label for the future of medical AI. It says: "Just because an AI is smart doesn't mean it's stable."

Even the best small AI models are currently too "moody" to be trusted alone in a hospital. They change their minds too often. Before we let them loose on Reddit or in clinics, we need to build systems that force them to be consistent, or we need to accept that a human must always be in the loop to double-check their work.

The Golden Rule: In medicine, a "mostly right" answer that changes every time you ask is actually a "wrong" answer.

1. Problem Statement

The integration of Large Language Models (LLMs) into medical workflows, particularly in online health communities (e.g., Reddit), faces a critical gap in evaluation paradigms. Current benchmarks focus almost exclusively on single-pass accuracy against a ground-truth corpus. However, for safety-critical medical applications, output consistency (reproducibility) is equally vital.

The Risk: A model that returns a correct answer 70% of the time but a plausible but wrong answer 30% of the time, or one that generates a different correct answer every time it is queried, is unreliable. Instability erodes trust and hinders auditability.
The Gap: Existing clinical NLP benchmarks (e.g., MedQA, PubMedQA) do not measure run-to-run variance. Furthermore, it is unclear if lowering the generation temperature (e.g., $T=0.2$ ) is sufficient to ensure deterministic behavior in small, open-weight models.

2. Methodology

The authors propose a reproducibility-first evaluation framework designed for small, locally deployable open-weight LLMs.

Dataset and Setup

Data: 50 questions randomly sampled from MedQuAD (a dataset of 16k+ medical Q&A pairs from NIH resources).
Models Evaluated: Three open-weight models served via Ollama:
1. Llama 3.1 8B: General-purpose baseline.
2. Gemma 3 12B: Larger general-purpose model.
3. MedGemma 1.5 4B: Clinically fine-tuned model (hypothesis: domain specialization compensates for smaller scale).
Inference Protocol:
- Repetition: Each question is run 10 times per model ( $N=1,500$ total responses).
- Parameters: Fixed temperature $T=0.2$ (near-deterministic), $top\text{-}p=1.0$ , max tokens 512.
- Prompting: Identical system instruction ("You are a clinical expert...") to isolate model behavior from prompt engineering.

Evaluation Metrics

The framework computes eight metrics divided into three categories:

Quality Metrics (vs. Ground Truth):
- Lexical/Semantic: Exact Match, Token F1, String Similarity, BLEU, ROUGE-L, BERTScore.
- LLM-as-Judge: A locally hosted 20B open-source model scores responses (0–1) based on factual correctness and clinical safety using a structured rubric.
Reproducibility Metrics (Within-Model Consistency):
- Normalized Self-Agreement Rate: The fraction of runs matching the most frequent (modal) output. (1.0 = perfect consistency).
- Normalized Response Uniqueness Rate: The ratio of distinct outputs to total runs. (1.0 = every run is unique).
Efficiency: Latency and throughput (tokens/second).

Pipeline Architecture

A modular, open-source Python CLI (clinical-eval) with three stages: run (generation), score (metric calculation), and report (aggregation). The design separates generation from scoring to allow re-evaluation without re-querying models.

3. Key Contributions

Reproducibility-First Protocol: Introduces a standardized method to quantify within-model run-to-run consistency (self-agreement and uniqueness) alongside standard accuracy metrics.
Open-Source Pipeline: Releases a fully reproducible, resumable pipeline for evaluating medical LLMs, including data processing and reporting tools.
Empirical Evidence of Instability: Demonstrates that even at low temperatures ( $T=0.2$ ), small open LLMs exhibit high stochasticity in medical QA, a risk invisible to single-pass benchmarks.

4. Key Results

The study evaluated 1,500 responses across the three models.

Quality Performance

Semantic Alignment: All models showed similar BERTScore (0.847–0.852), indicating comparable semantic alignment with ground truth.
Lexical Quality: Llama 3.1 8B led in Token F1 (0.277).
Clinical Judgment: Gemma 3 12B achieved the highest Judge Score (0.600), followed by Llama 3.1 8B (0.592).
Surprise Finding: MedGemma 1.5 4B (clinically fine-tuned) underperformed significantly on the Judge Score (0.459) and lexical metrics compared to the larger general-purpose models.
Note: No model achieved an Exact Match (0.000) with ground truth, confirming all models paraphrase rather than recall.

Reproducibility (The Critical Finding)

Despite the low temperature setting ( $T=0.2$ ), models were highly inconsistent:

Self-Agreement: Ranged from 0.122 (Llama 3.1 8B) to 0.198 (Gemma 3 12B). This means even the "most consistent" model agreed with itself on fewer than 1 in 5 runs.
Uniqueness:
- Llama 3.1 8B: 97.4% of outputs were unique.
- MedGemma 1.5 4B: 93.6% unique.
- Gemma 3 12B: 86.8% unique.
Conclusion: The models occupy entirely disjoint output spaces; pairwise overlap between models was 0.000.

Trade-offs

Llama 3.1 8B: Fastest (43 tok/s) and highest lexical quality, but worst reproducibility.
Gemma 3 12B: Best balance of quality (Judge Score) and reproducibility, but slowest (25.5 tok/s).
MedGemma 1.5 4B: Underperformed on all metrics, though this is confounded by its smaller parameter count (4B vs. 8B/12B).

5. Significance and Implications

Low Temperature is Insufficient: Setting $T=0.2$ does not guarantee reproducible medical outputs. Practitioners cannot rely on single-pass generation for clinical safety.
Need for Stabilization Strategies: Medical deployment requires ensemble methods (e.g., majority voting across multiple runs), confidence gating, or mandatory human-in-the-loop review to mitigate variance.
Scale vs. Fine-Tuning: At these parameter counts, model scale appears to dominate domain fine-tuning. A 4B clinically fine-tuned model underperformed 8B and 12B general models. Future work must compare same-size models (e.g., Gemma 3 4B vs. MedGemma 4B) to isolate the true value of clinical specialization.
Benchmarking Standard: The paper argues that reproducibility metrics (self-agreement, uniqueness) must become a standard co-equal axis in LLM benchmarking alongside accuracy, especially for safety-critical domains.

6. Limitations & Future Work

Judge Variance: The LLM-as-judge itself is stochastic (run once per response), introducing unquantified variance in the quality scores.
Confounding Variables: The comparison between MedGemma and others is confounded by parameter count differences.
Future Directions: Planned work includes testing ensemble decoding strategies, evaluating on USMLE/MedQA benchmarks, and comparing same-size models to isolate the effect of fine-tuning.