Beyond Accuracy: Risk-Sensitive Evaluation of Hallucinated Medical Advice

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in patient-facing medical question answering (QA). While existing evaluation standards for hallucinations focus primarily on factual correctness (i.e., whether the output matches a reference answer or ground truth), this approach has critical limitations in safety-critical domains:

• Uniform Severity Assumption: Current metrics treat all errors as equally severe. However, in medicine, a benign factual error (e.g., wrong year of a drug discovery) poses little risk, whereas an unsupported treatment directive or urgency cue can lead to immediate patient harm.
• Misalignment with Clinical Risk: Standard benchmarks fail to capture "actionable" hallucinations. A model might generate factually incorrect but clinically safe text, or conversely, factually incorrect text that includes dangerous instructions (e.g., "stop taking your insulin").
• Subjectivity: Detecting hallucinations often requires expert annotation and is highly subjective, making automated evaluation difficult.

The paper argues that evaluation must shift from assessing correctness to assessing potential impact (risk) if the advice were acted upon.

2. Methodology: Risk-Sensitive Evaluation Framework

The authors propose a novel framework that quantifies hallucinations based on the presence of risk-bearing language rather than factual verification.

A. Risk-Bearing Language Categories

The framework identifies specific linguistic patterns known to be safety-critical in medical NLP. These categories include:

1. Treatment Directives: Instructions to start, stop, increase, or adjust medications/therapies.
2. Contraindications/Prohibitions: Statements asserting actions/medications should not be taken.
3. Dosage Expressions: Explicit quantities, schedules, or dose adjustments.
4. Urgency and Triage Cues: Recommendations to seek emergency care or, conversely, to avoid care.
5. High-Alert Medications: Mentions of high-risk drugs (e.g., anticoagulants, insulin).
6. Overconfident Assertions: Language indicating certainty without supporting context.

B. Risk-Sensitive Hallucination Score (RSHS)

The core metric is a continuous score, RSHS, which aggregates the presence and severity of these risk signals.

• Weighting: Each pattern within a category is assigned a weight ($w_{k,p}$) reflecting ordinal differences in potential harm (e.g., discouraging medical care is weighted higher than general urgency).
• Normalization: The score is normalized by the token length of the response to prevent bias toward verbose outputs.
• Formula:
  $$RSHS(x) = \frac{\sum_{k=1}^{K} \sum_{p \in P_k} w_{k,p} \cdot n(p, x)}{1 + \log(1 + |x|)}$$
  Where $n(p, x)$ is an indicator function for pattern $p$ in response $x$, and $|x|$ is the token length.

C. Risk $\times$ Relevance Analysis (QASim)

To distinguish between relevant but risky advice and nonsensical hallucinations, the authors introduce a Query-Response Relevance Score (QASim).

• Calculated as the cosine similarity between sentence embeddings of the patient query and the model response.
• Joint Analysis: The framework specifically targets the intersection of High Risk and Low Relevance, identifying failures where models generate dangerous, unsupported directives that are weakly grounded in the user's input.

3. Experimental Setup

• Models Evaluated: Three instruction-tuned models from the same family (FLAN-T5-small, FLAN-T5-base, FLAN-T5-large) and one decoder-only model (Phi-3-mini).
• Prompts: 200 synthetic, patient-facing prompts designed as "safety stress tests." These cover symptom triage, medication management, pregnancy, and chronic disease, phrased to elicit actionable advice.
• Goal: To observe how model scale and prompt framing influence the generation of risk-bearing language.

4. Key Results

• Divergent Risk Profiles: Models with similar surface-level behavior exhibited substantially different risk profiles. Larger models (FLAN-T5-large) showed a higher tendency to generate risk-bearing language (higher upper-percentile RSHS), particularly in treatment directives and urgency cues, compared to smaller models.
• Failure of Standard Metrics: Standard evaluation metrics failed to capture these distinctions, as they treated all responses equivalently regardless of the presence of dangerous instructions.
• Prompt Sensitivity: Risk scores increased significantly when prompts explicitly invited management considerations. This demonstrates that hallucination risk is highly dependent on task design and prompt framing.
• Risk-Relevance Failure Modes:
  • Small Models: Occasionally produced off-topic or nonsensical responses containing risk-bearing language (High Risk, Low Relevance).
  • Large Models: More often generated coherent but overly prescriptive guidance (High Risk, Moderate/High Relevance).
• Extension to Phi-3: The framework generalized to decoder-only models (Phi-3), which showed a mean RSHS of 0.36 with pronounced upper-tail behavior, confirming the framework's utility across architectures.

5. Key Contributions

1. Paradigm Shift: Proposes moving from "factual correctness" to "risk sensitivity" in hallucination evaluation for medical LLMs.
2. Novel Metric (RSHS): Introduces a lightweight, interpretable, and continuous score that quantifies the potential harm of an output based on specific safety-critical linguistic patterns.
3. Joint Analysis: Combines risk scoring with semantic relevance (QASim) to identify a specific, high-stakes failure mode: dangerous advice that is weakly grounded in the user's query.
4. Empirical Evidence: Demonstrates that model scale and prompt design critically alter risk profiles, suggesting that current benchmarks are insufficient for safety assessment.

6. Significance and Limitations

Significance:

• Safety-Critical Alignment: The framework aligns evaluation with real-world clinical safety, where the actionability of an error is more important than its factual accuracy.
• Diagnostic Tool: It serves as a diagnostic tool for developers to compare models and probe safety-critical conditions without requiring expensive expert annotation for every test.
• Prompt Design Awareness: Highlights that evaluation validity is dependent on how prompts are framed; neutral prompts may hide dangerous behaviors that emerge under management-oriented queries.

Limitations:

• Not a Deployment Tool: The framework is for evaluation only, not for real-time safety filtering or clinical decision support.
• Heuristic-Based: Risk identification relies on pattern-based heuristics (keywords/phrases), which may miss subtle unsafe guidance or overestimate risk in benign contexts.
• Synthetic Data: Experiments used synthetic stress tests rather than real clinical encounters, so results may not fully reflect deployment-ready behavior.
• No Clinical Verification: The study does not assess actual clinical correctness or patient outcomes.

Conclusion

The paper concludes that hallucination evaluation in medical AI must evolve beyond accuracy metrics. By incorporating risk sensitivity and analyzing the grounding of advice, researchers can better identify and mitigate the specific failure modes that pose the greatest threat to patient safety.