Technical Summary: Hallucination Detection-Guided Preference Optimization for Clinical Summarization

Problem Statement

Large Language Models (LLMs) demonstrate strong capabilities in text summarization but frequently generate "hallucinations"—content unsupported by the source document or inconsistent with world knowledge. In clinical settings, where models summarize patient records to support care delivery, these hallucinations manifest as fabricated or distorted medical statements (e.g., unsupported procedures, medications, or diagnoses). Even domain-adapted models often produce hallucinations using medically plausible terminology, necessitating expert review. Existing mitigation strategies face significant limitations: training-time methods (e.g., supervised fine-tuning, reinforcement learning) require vast amounts of high-quality domain data and often rely on factuality metrics that correlate weakly with clinical correctness. Inference-time techniques, such as retrieval-augmented generation or self-refinement, often incur high computational overhead or risk over-editing and stripping salient content.

Methodology

The authors propose a two-stage pipeline, HDSR (Hallucination Detection Guided Self-Refinement) and HDSR-PL (HDSR for Preference Learning), designed to reduce hallucinations while preserving summary fluency and coherence.

1. Hallucination Detection Guided Self-Refinement (HDSR)

HDSR is an inference-time method that integrates hallucination detection into an iterative refinement loop:

• Initial Generation: An LLM generates an initial summary from a clinical note (e.g., Brief Hospital Course).
• Detection: A hallucination detector (specifically MedCat or MedAlign) analyzes the summary against the source document to identify unsupported or contradictory spans.
  • MedCat links clinical concepts to biomedical ontologies to flag missing or unsupported content.
  • MedAlign uses a prompt-based approach following a clinically motivated taxonomy of error types (e.g., unsupported conditions, procedures, medications).
• Iterative Revision: The detector's feedback (highlighting specific error spans) is fed back to the LLM as a prompt. The model is instructed to revise the summary to correct these factual errors while preserving supported content and maintaining a professional tone.
• Termination: This process repeats until a fixed iteration limit is reached or no further hallucinations are detected.

2. Preference Learning from Detection-Guided Self-Refinement (HDSR-PL)

To amortize the computational cost of iterative refinement and enable faithful generation at inference time without additional steps, the authors propose HDSR-PL:

• Preference Pair Construction: The iterative HDSR process generates pairs of summaries for each source document: the initial (hallucinated) summary and the revised (factually corrected) summary.
• Training: These pairs are treated as preference data, where the revised summary is the "preferred" output and the initial summary is the "non-preferred" output.
• Optimization: The summarization model is fine-tuned using Direct Preference Optimization (DPO). This allows the model to internalize the detector-guided factual corrections, learning to generate faithful summaries directly during inference without requiring the iterative refinement loop.

Key Results

Experiments were conducted on the MIMIC-IV-Note v2.2 dataset, specifically the Hallucination-Generated-DI subset, using Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Gemma-3-4B-IT.

• Hallucination Reduction:
  • HDSR (Inference-time): Reduced hallucinations by approximately 24% in Llama-3.1-8B-Instruct (from 29 to 22 errors) compared to zero-shot prompting.
  • HDSR-PL (Training-time): Achieved the most significant reduction, lowering hallucinations by 48% (from 29 to 15 errors) in Llama-3.1-8B-Instruct.
  • In contrast, standard Supervised Fine-Tuning (SFT) on clinician-written references increased hallucinations nearly twofold (from 29 to 57 errors), highlighting the risk of SFT without specific factuality constraints.
• Summary Quality: Both HDSR and HDSR-PL preserved or improved summary quality metrics (Consistency, Coherence, Fluency, Relevance) as evaluated by human clinicians and LLM-as-judge. HDSR-PL showed a slight trade-off in fluency and coherence compared to HDSR, attributed to learning a general preference for factual alignment over fine-grained stylistic details.
• Error Type Analysis: The methods were particularly effective at reducing "Unsupported Conditions" and "Unsupported Procedures," which constitute the majority of errors in baseline models.

Significance and Claims

The paper claims that detection-informed refinement and preference learning offer an automated solution for improving factual faithfulness in clinical summarization. Key contributions include:

1. Demonstrating the failure of standard SFT: The work shows that simply fine-tuning on clinician-written summaries can exacerbate hallucinations in clinical contexts, necessitating specialized alignment strategies.
2. Automated Preference Data Generation: By leveraging hallucination detectors to guide self-refinement, the authors create high-quality preference pairs without manual human annotation, enabling DPO training for factual alignment.
3. Balancing Faithfulness and Fluency: The results indicate that it is possible to substantially reduce hallucinations (up to 48%) while maintaining the fluency and coherence required for clinical summaries, as validated by both human experts and automated metrics.

Limitations

The authors acknowledge several limitations:

• Detector Dependency: The efficacy of both HDSR and HDSR-PL relies on the accuracy of the hallucination detector. False positives (flagging supported content) or false negatives (missing subtle inconsistencies) can limit refinement effectiveness.
• Domain Specificity: The evaluation is restricted to summarizing Brief Hospital Courses into Discharge Instructions from MIMIC-IV. The transferability to other medical note types or non-clinical domains remains unexplored.
• Computational Cost: While HDSR-PL amortizes costs at inference time, the HDSR inference-time method incurs additional computational overhead due to multiple refinement iterations and detector calls, which may hinder deployment in latency-sensitive settings.