💬 NLP

Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation

This paper proposes a healthcare-specific LLM framework that combines a LoRA-fine-tuned summarization model trained on MIMIC-III with an independent, granular fact-checking module to significantly reduce hallucinations and ensure the reliability of clinical outputs.

Original authors: Musarrat Zeba, Abdullah Al Mamun, Kishoar Jahan Tithee, Debopom Sutradhar, Mohaimenul Azam Khan Raiaan, Saddam Mukta, Reem E. Mohamed, Md Rafiqul Islam, Yakub Sebastian, Mukhtar Hussain, Sami Azam

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Musarrat Zeba, Abdullah Al Mamun, Kishoar Jahan Tithee, Debopom Sutradhar, Mohaimenul Azam Khan Raiaan, Saddam Mukta, Reem E. Mohamed, Md Rafiqul Islam, Yakub Sebastian, Mukhtar Hussain, Sami Azam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Confident but Wrong" Doctor

Imagine a brilliant, fast-talking medical student who has read every book in the library. They can summarize a patient's entire hospital stay in seconds. However, this student has a dangerous habit: they sometimes make things up.

In the world of Artificial Intelligence (AI), this is called a "hallucination." The AI might confidently say a patient was given 50mg of a drug when they were actually given 10mg, or claim a patient had a surgery that never happened. In healthcare, these small lies can lead to big, dangerous mistakes.

The Solution: A Two-Step Safety System

The authors of this paper built a system to stop these lies. Think of it as a two-person team working together:

The Storyteller (The Generator): This is a specialized AI trained specifically on medical records. Its job is to write a summary of what happened to a patient.
The Strict Editor (The Fact-Checker): This is the star of the show. Unlike the Storyteller, the Editor does not use AI to check the work. Instead, it uses a rigid set of logic rules, like a detective with a magnifying glass.

How the "Strict Editor" Works

The Editor doesn't just read the summary and guess if it feels right. It breaks the story down into tiny, atomic facts called "propositions."

The Analogy: Imagine the patient's medical record (the source of truth) is a giant, organized filing cabinet. The summary is a handwritten note.
The Process: The Editor takes every single sentence from the handwritten note, breaks it into pieces (e.g., "Patient took 50mg of Lisinopril"), and then goes to the filing cabinet to find the matching file.

Once it finds the matching file, it runs a series of logic tests:

The Math Test (Numerical Check): Does the number match? If the note says "50mg" but the file says "10mg," the Editor slams a red stamp on it: FAIL.
The Time Travel Test (Temporal Check): Did the events happen in the right order? If the note says "The patient was discharged before the fever broke," but the file says the fever broke after discharge, the Editor flags it. FAIL.
The "Yes/No" Test (Negation Check): Did the note say "No antibiotics" when the file clearly says "Antibiotics given"? FAIL.
The Logic Test (Implication Check): If the note says the patient had pneumonia, logic dictates they must have been given antibiotics. If the note mentions pneumonia but forgets the antibiotics, the Editor catches the missing piece. FAIL.
The "Did You Forget?" Test (Presence Check): If the file mentions a major surgery but the summary doesn't mention it at all, the Editor flags the omission. FAIL.

If a fact passes all these tests, it gets a green stamp: Supported. If it fails even one, it gets a red stamp: Not Supported.

Why This is Special

Most other systems try to fix this problem by asking another AI to check the first AI. But that's like asking a student to grade their own homework; they might both make the same mistake or be too polite to catch errors.

This paper's system is unique because the Fact-Checker is "LLM-free." It doesn't guess or "feel" if something is right. It uses hard, mathematical logic and direct comparisons against the original medical records. It's like a calculator checking a math problem instead of a human guessing the answer.

The Results

The team tested this system on thousands of real patient records (from a database called MIMIC-III).

The Storyteller got very good at writing summaries that sounded natural and accurate (scoring high on standard language tests).
The Strict Editor was incredibly effective at catching lies. It correctly identified supported facts about 89% of the time and caught unsupported lies with high accuracy.

The Bottom Line

This paper presents a safety net for medical AI. It doesn't just hope the AI is telling the truth; it forces the AI to prove every single fact against the original medical records using strict, non-AI logic. This makes the final summary much safer for doctors to use when making decisions about patient care.

Note on Limitations: The authors admit that while this works great for common medical situations, it might struggle with extremely rare diseases or very complex, specialized cases where the "rules" of logic aren't as clear-cut. They also note that the system currently flags the errors but doesn't automatically fix them; a human still needs to review the red flags.

Technical Summary: Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation

1. Problem Statement

The integration of Large Language Models (LLMs) into healthcare offers significant potential for automating clinical tasks, such as summarizing Electronic Health Records (EHRs). However, a critical barrier to adoption is the propensity of LLMs to generate "hallucinations"—confidently stated but factually incorrect or unverified information. In safety-critical domains like medicine, even minor inaccuracies regarding medication dosages, diagnoses, or laboratory values can compromise patient safety and erode clinician trust.

Existing approaches to mitigate hallucinations often rely on LLM-based evaluators, retrieval-augmented generation (RAG), or complex neural verification pipelines. These methods frequently suffer from opacity, high computational costs, and a lack of transparency, as they often depend on the very probabilistic models they aim to verify. Furthermore, many evaluation frameworks operate at a coarse-grained level (e.g., PICO structures or sentence-level similarity), failing to detect specific, granular factual errors such as incorrect temporal sequences or numerical mismatches.

2. Methodology

The proposed framework addresses these limitations through a two-stage, modular pipeline: a domain-specific summarization generator and an independent, LLM-free fact-checking module.

2.1. Domain-Specific Summarization (Generator)

To generate clinically coherent summaries, the authors fine-tuned the LLaMA-3.1-8B model using Low-Rank Adaptation (LoRA).

Dataset: The model was trained on 26,104 discharge summaries extracted from the MIMIC-III database, covering over 40,000 patient records.
Technique: LoRA was employed to inject trainable low-rank matrices into the pre-trained weights, specifically targeting the query ( $W_q$ ) and value ( $W_v$ ) projection layers of the attention mechanism. This approach allows for efficient domain adaptation while keeping the majority of parameters frozen, reducing computational overhead and memory usage.
Optimization: The model was optimized using the AdamW optimizer with a cosine learning rate scheduler, mixed-precision training (bfloat16), and gradient checkpointing.

2.2. LLM-Free Fact-Checking Module (Verifier)

The core innovation is a verification system that operates independently of any LLM during the evaluation phase. It functions as a deterministic, rule-based layer that validates generated summaries against ground-truth EHR data.

Proposition Extraction: Both the generated summary and the source EHR are decomposed into atomic propositions defined as tuples: $p = (e, a, v, t)$ , representing Entity (e.g., medication), Attribute (e.g., dosage), Value (e.g., 50mg), and Time (e.g., "on day 2"). This decomposition is performed using deterministic rule-based pipelines and clinical Named Entity Recognition (NER), not generative models.
Alignment: Summary propositions are aligned with EHR propositions using cosine similarity on embeddings derived from domain-specific encoders (BioClinicalBERT and Sentence-BERT). This step identifies the most semantically similar candidate for verification but does not determine factual correctness on its own.
Deterministic Consistency Checks: Once aligned, pairs of propositions undergo a series of logical and numerical checks:
1. Negation Check: Detects contradictions where one proposition asserts presence and the other absence (e.g., "no antibiotics" vs. "treated with antibiotics").
2. Implication Check: Verifies clinically dependent relationships (e.g., "pneumonia" implies "antibiotics").
3. Temporal Consistency: Ensures the chronological order of events in the summary matches the EHR.
4. Mutual Exclusivity: Flags logically incompatible states occurring at the same time (e.g., "intubated" vs. "spontaneously breathing").
5. Numerical Consistency: Validates exact matches for quantitative values and units (e.g., creatinine levels).
6. Presence Check: Detects clinically significant omissions where a key EHR event is missing from the summary.
Verdict Generation: Each proposition receives a binary verdict: Supported (passes all checks) or Not Supported (fails one or more checks).

3. Key Contributions

The paper outlines three primary contributions:

LLM-Free Fact-Checking Module: A verification system that applies discrete logic (negation, implication, temporal, numerical, and presence checks) to evaluate propositions independently of LLMs, ensuring transparency and reproducibility.
Deterministic Verification Mechanism: A pipeline that breaks down text into atomic propositions for one-to-one comparison, enabling granular fact-checking that goes beyond surface-level semantic similarity.
Domain-Specific Adaptation: A fine-tuned summarization model using LoRA on MIMIC-III data, which generates clinically accurate summaries with reduced hallucination rates while maintaining computational efficiency.

4. Experimental Results

The framework was evaluated on a subset of 104 discharge summaries from MIMIC-III, comprising 3,786 extracted propositions.

4.1. Summarization Performance

The LoRA-fine-tuned LLaMA-3.1-8B model demonstrated strong alignment with reference summaries:

ROUGE-1: 0.5797
ROUGE-2: 0.5580
ROUGE-L: 0.5618
BERTScore (F1): 0.9120
These metrics indicate high lexical and semantic fidelity to the source clinical narratives.

4.2. Fact-Checking Performance

The independent verification module achieved the following metrics on the proposition-level test set:

Precision: 0.8904
Recall: 0.8234
F1-Score: 0.8556
Accuracy: 0.7913
Specificity (Unsupported): 0.6949

4.3. Ablation Study

An ablation analysis confirmed the necessity of all six consistency checks. Removing any single check (particularly Negation and Numerical consistency) resulted in a significant drop in performance (e.g., precision dropped by over 8% when the Numerical check was removed).

4.4. Expert Evaluation

Two clinical experts reviewed the generated summaries and propositions:

Clinical Consistency: 93.27% (97/104) of summaries were deemed clinically accurate.
Proposition Correctness: 95.8% of extracted propositions were correct.
Disagreements were minimal and primarily related to rare diseases or complex temporal dependencies.

4.5. Comparison with State-of-the-Art

The proposed module achieved an accuracy of 79.13% on the MIMIC-III claim verification task, outperforming several existing models, including:

Claude-2 (DOSSIER): 78.62%
Claude-2: 70.65%
CodeLlama-13B (DOSSIER): 65.76%
MedAlpaca 7B: 46.74%

5. Significance and Claims

The authors claim that this work provides a transparent, reproducible, and interpretable solution to LLM hallucinations in healthcare. By decoupling the verification process from generative models, the system avoids the "black box" nature of LLM-based evaluators. The framework is designed to serve as a post-hoc safety layer that can flag contradictions (e.g., mismatched dosages, omitted treatments) before a clinician signs off on a document.

The paper modestly acknowledges limitations, noting that the system's robustness may decrease in highly specialized domains (e.g., rare oncological subtypes or complex transplant cases) where training data is sparse or domain-specific ontologies are insufficient. The authors emphasize that while the system effectively identifies inconsistencies, it currently functions as a detector rather than a self-correcting agent, leaving the final decision and potential rewriting to human clinicians.

In conclusion, the study demonstrates that combining a parameter-efficient, domain-adapted generator with a deterministic, proposition-level fact-checker can significantly enhance the reliability of AI-assisted clinical summarization, offering a viable path toward safer deployment of LLMs in critical healthcare settings.