MedScope: A Lightweight Benchmark of Open-Source Large Language Models for Medical Question Answering

This paper introduces MedScope, a lightweight benchmarking framework that systematically evaluates six open-source large language models on medical multiple-choice questions using multi-dimensional metrics and visual analyses, revealing significant performance heterogeneity and highlighting their current unsuitability for unsupervised high-risk clinical deployment despite their value as transparent baselines.

The Gist

Imagine you are a doctor trying to choose a new assistant to help you answer medical questions. You have a budget, so you can't hire the most expensive, super-intelligent AI (the "proprietary" ones that cost a fortune and live in the cloud). Instead, you want to test out the "lightweight" open-source assistants—smaller, free models that you can run on your own computer.

The paper "MedScope" is essentially a report card for six of these free, lightweight AI assistants. The researchers didn't just ask, "Who got the most questions right?" They built a whole new way to grade them, looking at speed, consistency, and how they handle different medical topics.

Here is the breakdown of what they did and what they found, using some everyday analogies:

1. The Setup: A "Speed Dating" for AI Models

The researchers took 1,000 medical questions (like a practice exam for doctors) and asked six different AI models to answer them. These models came from three famous "families": LLaMA, Qwen, and Gemma.

Think of these models like six different students taking a test:

• The "Big Brain" Student (LLaMA 3B): Studied hard, knows a lot, but takes a long time to think and sometimes gets confused about what the teacher is asking.
• The "Fast Runner" Student (Qwen 1.5B): Answers incredibly fast, almost instantly, but might not know as many deep facts.
• The "Balanced" Student (Gemma 4B): A middle ground. Not the fastest, not the absolute smartest, but very consistent and rarely makes silly mistakes.

2. The New Grading System: It's Not Just About the Score

In the past, people only looked at the final grade (Accuracy). MedScope says, "Wait, that's not enough!" They added four new ways to judge the students:

• The "Did They Even Answer?" Check (Invalid Rate): Sometimes, an AI gets confused and just writes a paragraph of nonsense instead of picking A, B, C, or D. The researchers checked how often this happened. The "Fast Runner" (Qwen) never failed to answer; the "Big Brain" (LLaMA) got confused about 15% of the time.
• The "Speed Test" (Inference Time): How long does it take to give an answer? If you are in an emergency room, you can't wait 10 seconds for an answer. The Qwen model was the fastest, finishing a question in less than a fifth of a second.
• The "Specialty Check" (Subject Variability): Medicine is huge. A model might be great at "Cardiology" (heart stuff) but terrible at "Dermatology" (skin stuff). The researchers found that no model was good at everything. Some were strong in microbiology but weak in surgery.
• The "Agreement Test" (Consistency): If you ask two different models the same question, do they agree? If they agree, it might mean they are both right, or it might mean they are both making the same mistake. The researchers found that models from the same family (like the two Qwen models) tended to agree with each other more than models from different families.

3. The Results: The "Trade-Off" Triangle

The most important finding is that there is no perfect model. It's like buying a car: you can have it fast, safe, or cheap, but usually not all three at once.

• If you want the highest accuracy: You pick the LLaMA 3B. It got the most questions right. But, it was the slowest and sometimes got confused by the instructions.
• If you want the most reliable answers: You pick Gemma 4B. It had the best balance of being smart and never giving a "garbage" answer.
• If you need speed (like for a quick app): You pick Qwen 1.5B. It was lightning fast and never failed to answer, even if it wasn't the absolute smartest.

4. The Big Warning: Don't Trust Them Yet

The authors give a very important warning: These models are not ready to replace doctors.

Imagine these models as junior medical students. They are smart, they can read textbooks, and they can pass a written test. But if you put them in a real hospital to treat a patient without a supervisor, they might make a dangerous mistake.

The paper concludes that while these lightweight models are great for:

• Research: Scientists can use them to test ideas without paying huge fees.
• Education: Helping students study.
• Privacy: Running on a local computer so patient data doesn't leave the building.

...they are not ready for high-stakes decisions like diagnosing a sick person. They are "assistants," not "doctors."

Summary in One Sentence

MedScope is a new, fairer way to test small, free AI models for medical questions, proving that while some are faster and others are smarter, none are perfect enough to work alone in a hospital yet, so we need to look at more than just their test scores to choose the right one.

Technical Summary

1. Problem Statement

While frontier proprietary Large Language Models (LLMs) have shown promise in medical applications, the empirical understanding of lightweight open-source LLMs in medical settings remains limited. Existing benchmarks often focus on massive proprietary models or rely solely on aggregate accuracy scores, failing to capture:

• The performance-efficiency trade-offs of models deployable on local hardware.
• Heterogeneity in performance across different medical specialties.
• Critical deployment metrics such as inference time, output validity, and prediction consistency.
• The need for transparent, reproducible, and resource-constrained evaluation frameworks suitable for academic and low-resource healthcare settings.

2. Methodology

The authors introduce MedScope, a lightweight benchmarking framework designed to systematically evaluate open-source LLMs on medical multiple-choice question answering (MCQA).

Dataset and Sampling

• Source: MedMCQA, a large-scale dataset derived from Indian medical entrance exams (194k+ questions, 21 subjects).
• Sampling: A subset of 1,000 questions was randomly sampled from the validation split using a fixed seed to ensure reproducibility.
• Preprocessing: Questions were standardized to a four-option format (A, B, C, D) with gold labels and subject/topic metadata.

Evaluated Models

Six lightweight open-source models from three representative families were evaluated under a unified local environment (macOS via Ollama runtime):

1. LLaMA: llama3.2:1b, llama3.2:3b
2. Qwen: qwen2.5:1.5b, qwen2.5:3b
3. Gemma: gemma3:1b, gemma3:4b

Evaluation Protocol

• Prompting: A unified prompt template constrained models to output a single label (A, B, C, or D). Rule-based post-processing extracted predictions; outputs failing to map to valid labels were marked as "invalid."
• Metrics:
  • Predictive: Accuracy and Macro-F1 (to account for class imbalance).
  • Reliability: Invalid Response Rate (controllability).
  • Efficiency: Average Inference Time per question.
  • Consistency: Pairwise model agreement (similarity in decision patterns).
  • Granularity: Subject-wise performance analysis across 21 medical disciplines.

Visualization Strategy

To move beyond single-score rankings, the framework employs multi-perspective visualizations:

• Clustered heatmaps (subject-wise performance).
• Pareto-style trade-off plots (accuracy vs. latency).
• Agreement matrices and network graphs.
• Radar charts for multi-dimensional profiling.
• Sankey diagrams for error flow analysis.

3. Key Contributions

1. MedScope Framework: Established a reproducible, lightweight evaluation setting for open-source medical LLMs that does not rely on proprietary APIs.
2. Multi-Dimensional Benchmarking: Conducted a systematic comparison across model families and scales, analyzing not just accuracy but also efficiency, validity, and subject-level robustness.
3. Interpretable Analysis: Provided a suite of visual tools to characterize model behavior, revealing that aggregate scores often mask significant variability in specific medical domains and deployment constraints.

4. Key Results

The study revealed substantial heterogeneity across models, indicating that no single model dominates all metrics:

• Performance vs. Efficiency Trade-off:
  • LLaMA (3B): Achieved the highest Accuracy (27.55%) but suffered from the slowest inference speed (0.87s/question) and the highest invalid response rate (14.7%).
  • Gemma (4B): Achieved the best Macro-F1 (20.76%) with 0% invalid responses, offering a strong balance between quality and stability.
  • Qwen (1.5B): Demonstrated superior efficiency (0.17s/question) with 0% invalid responses, though with lower accuracy (21.5%).
• Family-Level Trends:
  • LLaMA generally led in raw accuracy.
  • Gemma showed the best balance of accuracy, F1, and output validity.
  • Qwen excelled in inference speed, making it suitable for latency-sensitive applications.
• Subject-Wise Heterogeneity: Performance varied significantly across medical disciplines (e.g., Microbiology, Forensic Medicine). High aggregate scores did not guarantee competence in specific high-risk specialties.
• Agreement Analysis: Models from the same family showed higher agreement, but cross-family agreement was moderate. Crucially, high agreement does not equate to correctness; models often shared similar error patterns.

5. Significance and Implications

• Clinical Readiness: The results indicate that while lightweight open-source LLMs are valuable as transparent baselines for research and education, their current capabilities are insufficient for unsupervised deployment in high-risk clinical scenarios. Performance variability across specialties and the presence of invalid responses pose significant risks.
• Evaluation Paradigm Shift: The paper argues against relying on single-metric leaderboards. Effective medical AI evaluation must be multi-dimensional, considering efficiency, validity, and domain-specific robustness alongside accuracy.
• Deployment Strategy:
  • Resource-Constrained Settings: Lightweight models (especially Qwen and Gemma variants) offer viable options for local, privacy-preserving experimentation and educational tools.
  • Responsible Use: These models should be treated as assistive tools requiring human oversight rather than autonomous decision-makers.
• Future Directions: The authors suggest future work should explore ensemble methods to leverage model diversity, expand evaluation to open-ended dialogue and uncertainty quantification, and integrate clinician-in-the-loop validation.

Conclusion: MedScope provides a critical, accessible benchmark that highlights the nuanced trade-offs in lightweight medical LLMs, emphasizing that "smaller" does not mean "less capable" in all dimensions, but rather requires a more sophisticated, multi-faceted assessment for safe and effective deployment.