💬 NLP

Beyond Semantic Similarity: A Component-Wise Evaluation Framework for Medical Question Answering Systems with Health Equity Implications

This paper introduces VB-Score, a component-wise evaluation framework that reveals severe performance failures and significant health equity disparities in current medical LLMs, demonstrating that semantic similarity alone is insufficient for ensuring medical accuracy and safety.

Original authors: Abu Noman Md Sakib, Md. Main Oddin Chisty, Zijie Zhang

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abu Noman Md Sakib, Md. Main Oddin Chisty, Zijie Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a library and ask a very polite, well-spoken librarian for advice on how to treat a specific medical condition. The librarian smiles, speaks in perfect sentences, and gives you a long, flowing story about your illness. It sounds great! It feels like they know exactly what they are talking about.

But then, you realize: they forgot to tell you the name of the medicine, the exact dosage, or even if you should avoid certain foods. They gave you a beautiful, fluent story, but the actual medical facts were missing or wrong.

This is exactly what researchers found when they tested the world's most popular AI chatbots (like GPT-4, Claude, and Gemini) on medical questions. They built a new "report card" called VB-Score to see if these AIs are actually safe to use for health advice.

Here is the breakdown of their findings using simple analogies:

1. The "Smooth Talker" vs. The "Fact-Checker"

Most people judge an AI by how well it sounds. If it speaks smoothly and uses the right words, we assume it's smart.

The Old Way: Imagine grading a student only on their handwriting. If the handwriting is perfect, you give them an A.
The New Way (VB-Score): The researchers realized that in medicine, handwriting doesn't matter; the content does. They created a test that grades four things separately:
1. Did they get the specific names right? (e.g., "Ibuprofen" vs. just "painkiller").
2. Did they sound relevant? (Did they answer the question asked?)
3. Is the information actually true? (Did they contradict a doctor's guide?)
4. Is the information complete? (Did they list all the symptoms or just one?)

2. The "Fluency Trap" (The Big Gap)

The study found a shocking disconnect.

The Analogy: Imagine a tour guide who speaks perfect English and describes a city beautifully. But, when you ask, "Where is the nearest hospital?" they say, "Oh, there are many hospitals in this city!" without giving you a single address.
The Result: The AI chatbots were 51% good at sounding relevant (the tour guide's speech), but only 6% good at giving specific medical facts (the address).
The Danger: This means an AI can write a paragraph that sounds like a doctor, but if you follow its advice, you might miss the critical details (like the wrong drug name or missing a dangerous side effect).

3. The "Free vs. Paid" Surprise

Usually, we think "you get what you pay for." We assume the expensive, premium AI models are the safest and most accurate.

The Analogy: It's like buying a $500 suit that fits poorly, while the free t-shirt you found on the rack fits perfectly.
The Result: In this study, the free model (Gemini) actually gave better, safer medical advice than the expensive, paid models (GPT-4 and Claude). The free model was more careful about not lying, even if its writing style was a bit less "polished."

4. The "Unfairness" Problem (Health Equity)

The study also looked at whether the AI treated different types of diseases fairly.

The Analogy: Imagine a teacher who is great at teaching math (Infectious Diseases) but terrible at teaching history (Chronic Diseases).
The Result: The AI was significantly better at answering questions about Infectious Diseases (like the flu or COVID) than Chronic Diseases (like diabetes, heart disease, or mental health).
Why this matters: Chronic diseases often affect older people, lower-income communities, and minority groups the most. If the AI gives bad advice to these specific groups because it "doesn't understand" their complex, long-term conditions, it creates a double injustice: the people who need help the most get the worst advice.

5. Can We Just "Prompt" the AI to be Better?

People thought, "Maybe if we just ask the AI nicely, or give it a better instruction, it will fix itself."

The Analogy: It's like telling a car with a broken engine, "Please drive faster," and expecting it to work.
The Result: The researchers tried giving the AI special instructions (like "Be very specific with drug names"). It didn't really help. The AI's "brain" (its architecture) simply isn't built to extract precise medical facts yet. No amount of polite asking can fix a broken engine.

The Bottom Line

The paper warns us: Do not trust an AI just because it sounds smart.

Just because an AI can write a fluent, confident paragraph about your health doesn't mean it knows the facts. It might be "hallucinating" (making things up) or missing critical details like drug names and dosages.

The Takeaway: We need a new way to test medical AI that checks the facts, not just the fluff. Until then, we should treat AI health advice like a rough draft from a student: it might have good ideas, but you absolutely need a real doctor to check the work before you take it seriously.

1. Problem Statement

The increasing adoption of Large Language Models (LLMs) for patient-facing medical guidance poses significant risks to safety and health equity. Current evaluation methods for medical QA rely heavily on semantic similarity metrics (e.g., BLEU, ROUGE, BERTScore), which measure how closely a generated response matches the meaning of a reference text.

The Core Flaw: High semantic similarity does not guarantee medical accuracy. A model can generate a fluent, topically relevant response that lacks specific medical entities (dosages, drug names), contains factual contradictions, or omits critical structured information.
The Consequence: This "semantic-entity gap" allows models to produce dangerously misleading advice that appears correct to a layperson but fails clinical precision. Furthermore, existing evaluations often fail to detect algorithmic discrimination, where models perform worse on health topics affecting vulnerable populations (e.g., chronic conditions in elderly or minority groups).

2. Methodology

The authors propose a novel evaluation framework and conduct a rigorous benchmarking study.

A. The VB-Score Framework

The paper introduces VB-Score (Verification-Based Score), a composite metric designed to evaluate medical QA across four distinct, weighted dimensions:

Entity Recognition (30% weight): Measures the precision of extracting specific medical entities (medications, dosages, symptoms, procedures) using biomedical Named Entity Recognition (NER). This is heavily weighted due to the high risk of medication errors.
Semantic Similarity (30% weight): Assesses conceptual alignment using sentence-transformer embeddings to ensure the response addresses the clinical intent.
Factual Consistency (25% weight): Uses Natural Language Inference (NLI) models (RoBERTa-large-mnli) to detect contradictions between the generated output and authoritative references.
Structured Overlap (15% weight): Evaluates the completeness of structured information (e.g., bullet points, lists) using regex extraction and Jaccard similarity.

Formula:
$VB\text{-}Score = 0.30 \times \text{Entity\_F1} + 0.30 \times \text{Semantic\_Sim} + 0.25 \times \text{Factual\_Cons} + 0.15 \times \text{Structured\_Overlap}$

B. Dataset and Experimental Setup

Data: 48 public health topics sourced from authoritative bodies (CDC, WHO, NHS, Mayo Clinic), stratified into Infectious Diseases (19 topics) and Chronic Conditions (29 topics).
Models Evaluated: Three widely used LLMs:
- GPT-4 (Paid)
- Claude Sonnet 4.5 (Paid)
- Gemini 2.5 Flash (Free)
Prompting Strategies: The study tested Zero-Shot, Zero-Shot Strict (explicit entity instructions), Few-Shot, and Retrieval-Augmented Generation (RAG) configurations.
Evaluation: 576 total evaluations were performed. Statistical significance was tested using ANOVA and t-tests with Bonferroni correction.

3. Key Contributions

VB-Score Framework: The first component-wise evaluation framework for medical QA that explicitly separates entity precision, semantic alignment, factual consistency, and structural completeness, revealing trade-offs invisible to aggregate metrics.
Quantification of the Semantic-Entity Gap: The study empirically demonstrates a massive disconnect between fluency and precision in medical contexts.
Health Equity Analysis: The paper identifies "condition-based algorithmic discrimination," showing that models perform significantly worse on chronic diseases (affecting vulnerable populations) compared to infectious diseases.
Cost vs. Quality Challenge: The study challenges the assumption that paid models are inherently superior, finding that a free model (Gemini 2.5 Flash) outperformed paid alternatives in this specific benchmark.

4. Key Results

A. The Semantic-Entity Gap

Average Semantic Similarity: ~51.6% (Moderate relevance).
Average Entity Recognition (F1): 6.2% (Catastrophic failure).
The Gap: A 45.4 percentage point gap (8.3× ratio). Models generate fluent text that lacks actionable medical specifics (e.g., saying "take pain medication" instead of "take acetaminophen 500mg").
Failure Rates: 91–100% of samples failed to meet the minimum threshold for entity recognition (F1 < 0.10).

B. Model Performance Rankings

Contrary to expectations, Gemini 2.5 Flash (Free) achieved the highest overall VB-Score (0.340), outperforming GPT-4 (0.271) and Claude Sonnet 4.5 (0.229).

Gemini: Highest Factual Consistency (69.6%) but lower fluency and structure. It prioritized safety over polish.
GPT-4: Highest Semantic Similarity (53.8%) and Entity F1 (7.3%), but suffered from high factual contradiction rates (54.2%).
Claude: Best at Structured Overlap but had the worst Factual Consistency (17.9%), with 79.2% of responses contradicting authoritative sources.

C. Prompt Engineering and RAG

RAG (Retrieval-Augmented Generation): Improved overall VB-Scores by ~27% across all models but failed to fix entity extraction. Even with perfect ground-truth context, Entity F1 scores remained below 10%.
Prompt Sensitivity: Explicit instructions to "include medical entities" degraded performance for GPT-4 and Claude (by ~15-18%) but improved Gemini. This suggests no universal prompt template exists; models react differently to instruction following vs. factual accuracy.

D. Health Equity Disparities (Condition-Based Discrimination)

Performance Gap: Models performed 13.8% worse on Chronic Conditions (e.g., diabetes, hypertension) compared to Infectious Diseases (e.g., flu, HIV).
Driver: The disparity was primarily driven by lower Factual Consistency in chronic disease topics.
Equity Implication: Populations with higher burdens of chronic disease (elderly, low-income, minority groups) receive lower-quality AI health information, potentially exacerbating existing health inequities.

5. Significance and Implications

Safety Warning: Relying solely on semantic similarity is dangerous for medical AI. A model can be "fluent" yet "medically lethal" due to missing entities or factual contradictions.
Architectural Limitations: Prompt engineering and RAG cannot fully overcome the fundamental inability of current LLM architectures to extract precise medical entities. Specialized medical AI systems with built-in entity verification are required.
Equity and Access: The finding that a free model (Gemini) outperformed paid models challenges the notion that cost equals quality. This suggests that equitable access to high-quality medical AI is possible without financial barriers, provided the models are rigorously validated.
Policy Recommendation: The authors call for a shift from aggregate similarity metrics to component-wise, entity-centric evaluation with fairness-aware auditing before deploying LLMs in patient-facing healthcare applications.