A Lightweight Retrieval-Grounded Framework for Hallucination Detection and Correction in Large Language Models

Technical Summary: A Lightweight Retrieval-Grounded Framework for Hallucination Detection and Correction in Large Language Models

Problem Statement
Large Language Models (LLMs) have become central to applications ranging from question answering to enterprise decision support. However, they remain prone to "hallucinations"—outputs that appear fluent and plausible but are factually incorrect, unsupported by evidence, or inconsistent with verified knowledge. This issue poses significant risks in high-stakes domains such as healthcare, finance, and legal reasoning. Existing mitigation strategies, including Retrieval-Augmented Generation (RAG), self-verification, tool-assisted fact-checking, and multi-agent debate frameworks, often suffer from specific limitations: they may focus solely on detection without offering correction, rely on repeated large-model inference which increases computational costs, or introduce complex orchestration that hinders reproducibility.

Methodology
The paper proposes a lightweight retrieval-grounded framework designed to detect and correct hallucinations without the computational overhead of live multi-LLM inference. The system is structured as a modular pipeline that decouples generation, retrieval, verification, and correction.

1. Modular Agent Simulation: Unlike conventional multi-agent systems that invoke multiple live LLM instances, this framework models agents as modular processing stages operating on pre-generated benchmark responses. This design isolates verification and correction from generation variability, ensuring controlled and reproducible evaluation.
2. Retrieval Module: The system utilizes TF-IDF (Term Frequency-Inverse Document Frequency) for evidence retrieval rather than semantic embeddings. This choice prioritizes computational efficiency, interpretability, and reproducibility, avoiding dependencies on external neural retrieval models. Given a query $q$, the system retrieves the top-$k$ relevant documents ($D$).
3. Verification Agent: This component evaluates the factual consistency of a generated response ($r_g$) against the retrieved evidence ($D$). It employs a weighted verification score combining two signals:
   • Answer–Evidence Similarity: The TF-IDF cosine similarity between the generated response and the retrieved documents.
   • Query–Evidence Relevance: The TF-IDF cosine similarity between the original query and the retrieved documents.
   • The score is calculated as: $Score = \alpha \cdot Sim(r_g, D) + (1 - \alpha) \cdot Sim(q, D)$, where $\alpha$ controls the relative importance of the two signals.
   • A threshold-based decision rule ($\tau$) classifies the response as hallucinated ($Score < \tau$) or non-hallucinated ($Score \geq \tau$).
4. Correction Agent: When a response is flagged as hallucinated, the correction agent revises the content. It removes unsupported claims and incorporates evidence-consistent information while preserving relevant parts of the original response. Crucially, this mechanism is designed to be evidence-guided and lightweight, avoiding additional LLM inference steps.

Key Contributions
The paper outlines four primary contributions:

1. A lightweight detection and correction framework that avoids expensive multi-LLM inference while preserving modular, agent-inspired reasoning.
2. A unified pipeline that extends beyond mere detection to produce evidence-guided corrected responses.
3. A threshold-based weighted verification formulation that explicitly combines answer–evidence similarity and query–evidence relevance.
4. A reproducible experimental evaluation on the MedHallu and TruthfulQA datasets, demonstrating strong detection performance and measurable hallucination reduction.

Experimental Results
The framework was evaluated on two datasets: MedHallu (9,000 samples representing medical-domain hallucinations) and TruthfulQA (817 samples representing adversarial factuality challenges).

• Detection Performance: Using an optimal threshold ($\tau = 0.55$), the framework achieved an F1-score of approximately 0.93 (specifically 0.931 on the combined set) and a recall of 0.891. On the MedHallu dataset specifically, the F1-score reached 0.942.
• Correction Performance: The correction stage reduced the hallucination rate in the MedHallu setting from 1.0000 to 0.5784, representing a 42.16% reduction in hallucinated responses.
• Baseline Comparisons: Experiments showed that retrieval alone is insufficient for detection, as it fails to explicitly validate the consistency between the answer and the evidence. The addition of the weighted verification module significantly improved detection metrics.
• Limitations Observed: On TruthfulQA, the correction stage showed a "regression rate" of 0.2485, indicating that similarity-based correction can sometimes unnecessarily alter factual responses. This suggests that while effective for domain-specific hallucinations, the current similarity-based approach requires careful gating for adversarial or misconception-driven queries.

Significance and Claims
The paper claims that its primary significance lies in offering a reproducible and computationally efficient alternative to complex multi-agent verification systems. By modeling agents as modular stages rather than live LLM calls, the framework eliminates the need for repeated large-model inference, making it suitable for resource-constrained environments.

The authors assert that explicit verification is essential; retrieval alone provides context but does not confirm factual support. Their weighted scoring formulation addresses this gap. Furthermore, the work emphasizes that mitigation should not stop at detection; the proposed correction stage actively improves the factual reliability of the final output. However, the paper modestly acknowledges that TF-IDF retrieval and lexical similarity are weaker for reasoning-heavy or adversarial cases, suggesting that future iterations should incorporate semantic embeddings and entailment-based verification.