Technical Summary: HallDetect – Decomposed Entailment for Factuality Checking

Problem Statement

The deployment of Large Language Models (LLMs) in professional and decision-critical contexts is hindered by hallucinations: confident yet factually unsupported statements. Existing detection methods face significant limitations:

• Surface-level metrics (e.g., ROUGE, embedding similarity) fail to capture logical inconsistencies.
• "LLM-as-a-judge" paradigms suffer from high computational costs, latency, reproducibility issues, and self-preference bias, particularly when deployed on consumer hardware or via proprietary black-box APIs.
• White-box approaches relying on internal signals (token probabilities) are inapplicable to closed-source models.
• Existing fact-checkers often require full-precision models, external knowledge bases (measuring extrinsic factuality rather than intrinsic faithfulness), or lack the granularity to localize specific errors within a generated text.

There is a need for a lightweight, reference-free, black-box framework that can detect intrinsic hallucinations (inconsistencies with a provided source) under strict computational budgets while providing an audit trail for specific errors.

Methodology: The HallDetect Framework

HallDetect is a resource-efficient, domain-agnostic pipeline that decomposes generated text into atomic claims and verifies them against a source document using a discriminative Natural Language Inference (NLI) model. The framework operates under a strict 4-bit quantized budget on consumer-grade hardware.

Pipeline Architecture

The process consists of four distinct phases:

1. Atomic Claim Extraction (Phase I):

   • A locally deployed, 4-bit quantized LLM (e.g., Llama 3.1, Gemma 2, or Mistral) decomposes the generated response $R$ into a set of atomic claims $\{c_1, \dots, c_n\}$.
   • The extraction is "copy-faithful," reproducing claims verbatim to ensure verification targets the model's actual assertions.

2. Multi-Scale Contextual Chunking (Phase II):

   • The source document $D$ is partitioned into a hierarchical context library $\mathcal{K}$ at five granularities ($m \in \{1, 2, 4, 8, 16\}$ chunks).
   • This dyadic scheduling ensures that evidence is captured at various scales (from sentence-level to whole-document) without incurring linear cost growth. The total library size is capped at 31 chunks per document.

3. Contrastive NLI Verification (Phase III):

   • A compact, encoder-based NLI model (DeBERTa-v3-Large, fine-tuned on MNLI, FEVER-NLI, etc.) verifies each claim $c_i$ against every chunk $k \in \mathcal{K}$.
   • Contrastive Scoring: For each claim, the system independently identifies the chunk with the highest Entailment probability ($P(E)$) and the chunk with the highest Contradiction probability ($P(C)$).
   • The score is calculated as: $Score(c_i) = \max_k P(E|k, c_i) - \max_k P(C|k, c_i)$.
   • The Neutral class is handled implicitly; claims where all chunks are predominantly neutral receive low scores for both entailment and contradiction, resulting in a near-zero score (indicating unsupported content).

4. Asymmetric Aggregation (Phase IV):

   • The final Factuality Entailment Decomposition score (FEDscore) aggregates per-claim scores using a geometric mean.
   • Asymmetry: The aggregation is designed for auditing. If any single claim has a score $\le 0$ (indicating confident contradiction), the entire response is flagged as hallucinated ($FEDscore = 1$). This prioritizes recall, ensuring that a single unsupported assertion invalidates the output.

Key Contributions

The authors present four primary contributions:

1. A Frugal Decomposition-and-Verification Pipeline: HallDetect couples atomic claim extraction with a contrastive, multi-scale NLI formulation. It requires only the source-generation pair and is compatible with arbitrary black-box generators, avoiding the need for generative "judges" during the verification phase.
2. Controlled Evaluation under Fixed Budgets: The framework is evaluated under strict resource parity (all components 4-bit quantized, running on identical consumer hardware). Under these constraints, HallDetect outperforms comparably resourced generative (LLM CoT) and embedding-based baselines on three of four benchmarks, while remaining competitive in the fourth.
3. Stability Across Backbones and Domains: By delegating verification to a discriminative encoder, HallDetect exhibits markedly lower variance than generative Chain-of-Thought (CoT) judges across different backbone families (Mistral, Gemma, Llama) and domains (news, dialogue, adversarial, biomedical).
4. Claim-Level Audit Trail: Unlike scalar scoring methods, HallDetect produces a mapping between atomic claims and source evidence spans, allowing for the localization of contradictions and unsupported additions.

Quantitative Results

Evaluated on four benchmarks (QAGS-CNN/DM, TofuEval, FalseSum, PubMedQA) using 4-bit quantized models:

• QAGS-CNN/DM (News Summarization): HallDetect achieved an F1 score of 0.771 (Gemma-2), outperforming the LLM CoT baseline (0.589) and the monolithic NLI baseline (0.638). It demonstrated superior recall (1.000) compared to baselines, attributed to the decomposition strategy preventing the "dilution" of localized errors.
• Stability: While LLM CoT performance collapsed significantly across different backbones (e.g., F1 dropping from 0.589 to 0.321 on Llama-3.1), HallDetect remained stable (F1 range 0.734–0.771).
• Adversarial Stress (FalseSum): HallDetect maintained strong performance (F1 0.731), outperforming the LLM CoT baseline (0.375) and the SelfCheck baseline (0.649).
• Domain Generalization (PubMedQA): HallDetect remained competitive but did not outperform the SelfCheck baseline (HallDetect F1 0.632 vs. SelfCheck F1 0.668). While specialized scorers occasionally matched or exceeded HallDetect's performance in this domain, HallDetect retained the unique ability to localize specific medical claims lacking grounding, a feature absent in scalar baselines.

Significance and Claims

The paper positions HallDetect not as a state-of-the-art detector surpassing full-precision or API-based systems, but as a robust, resource-efficient solution for scenarios where computational budgets are constrained (e.g., local deployment, consumer hardware).

• Robustness to Quantization: The study demonstrates that decoupling claim extraction from verification (using a discriminative encoder) shields the judgment from the decoding biases and instability that plague quantized generative models.
• Auditability: The framework provides a "claim-to-span" audit trail, localizing errors rather than providing a single scalar score, which is critical for human-in-the-loop auditing.
• Operational Feasibility: By operating entirely on 4-bit quantized models with sub-1GB VRAM requirements, HallDetect offers a practical alternative to expensive "LLM-as-a-judge" paradigms for intrinsic factuality checking.

The authors explicitly note limitations, including the lack of comparison to specialized discriminative fact-checkers like MiniCheck, the inability to verify extrinsic facts (claims not in the source), and the recall-oriented operating point which may lower precision in low-hallucination-rate scenarios.