How Far Can You Get Without a GPU? A Systematic Benchmark of Lightweight Hallucination Detection Across Question Answering, Dialogue, and Summarisation

Technical Summary: Systematic Benchmark of Lightweight Hallucination Detection

Problem Statement
The deployment of Large Language Models (LLMs) in high-stakes domains requires robust hallucination detection. However, the most accurate detection methods currently rely on resource-intensive requirements: GPU-accelerated inference, proprietary API calls, or white-box access to model internals (e.g., token probabilities). These constraints exclude resource-constrained researchers and practitioners. This paper addresses the question of how effectively hallucination detection can be performed using only lightweight, CPU-feasible methods built on publicly available models, without API calls or access to the generating model's internals.

Methodology
The authors conducted a systematic benchmark of five distinct detection methods across the three tasks of the HaluEval dataset: Question Answering (QA), Dialogue, and Summarisation. The study utilized a standard laptop CPU and public models.

1. Methods Evaluated:

   • ROUGE-L: A lexical overlap metric measuring the longest common subsequence between the source and candidate.
   • Semantic Similarity: Cosine similarity between source and candidate embeddings using all-MiniLM-L6-v2 (~22M parameters).
   • BERTScore: A token-level contextual F-measure using a DistilBERT backbone.
   • NLI Detector: A Natural Language Inference approach using a DeBERTa-v3-base model (~184M parameters) fine-tuned on MultiNLI, FEVER, and Adversarial NLI. The source (truncated to 800 characters) plus context serves as the premise, and the candidate as the hypothesis. The score is derived as $s = 1 - P(\text{entailment})$.
   • Ensemble: A score-level linear combination of the NLI and Semantic Similarity scores ($s = \alpha s_{NLI} + (1-\alpha)s_{sim}$), with weights ($\alpha$) optimized via grid search on a validation set.

2. Experimental Setup:

   • Data: 3,000 samples per task were drawn from HaluEval (seed 42).
   • Split: 1,000 instances for validation (threshold and weight calibration) and 2,000 instances for testing.
   • Calibration: Thresholds were selected on the validation set to maximize the Youden index (TPR - FPR) and applied unchanged to the test set.
   • Metrics: Accuracy, Precision, Recall, F1 (where positive = hallucinated), and AUC-ROC.

Key Results
Performance was found to be highly task-dependent, with no single method dominating across all scenarios.

• Question Answering (QA): This was the most tractable task. The Ensemble method achieved the best performance (F1 = 0.792, AUC-ROC = 0.873), outperforming the best single method by approximately 5 points. Contextual token overlap (BERTScore) proved a stronger signal than sentence-level cosine similarity.
• Dialogue: Performance degraded compared to QA. The NLI detector emerged as the strongest single method (AUC-ROC = 0.713, Recall = 0.781), indicating that entailment reasoning transfers effectively to conversational settings. Lexical and embedding methods clustered lower (AUC-ROC 0.61–0.66).
• Summarisation: All five methods degraded to near-random performance (AUC-ROC between 0.469 and 0.574). While some methods showed high F1 scores (e.g., BERTScore F1 = 0.658), this was driven by extreme recall (>0.99) with near-chance precision, effectively labeling almost everything as hallucinated. The authors attribute this to a structural failure: summarisation hallucinations are often subtle factual edits within long, otherwise faithful summaries. Overlap metrics are dominated by the faithful remainder, and the NLI model's 800-character premise truncation prevents it from accessing the full document context.

Computational Footprint
All experiments ran on a standard laptop CPU without a GPU.

• Throughput: Lexical and embedding methods are highly efficient (ROUGE-L: ~1,300 candidates/sec; Similarity/BERTScore: ~66 candidates/sec).
• Bottleneck: The NLI detector (~184M parameters) is the slowest at ~4 candidates/sec, scoring a 2,000-instance test set in approximately eight minutes. The ensemble adds no additional model cost beyond the NLI and similarity components.

Key Contributions

1. Systematic Benchmark: A comprehensive evaluation of five CPU-feasible methods across three distinct NLP tasks, calibrated on held-out validation data.
2. Task-Dependent Ranking: Demonstration that method effectiveness varies significantly by task; the Ensemble is optimal for QA, NLI for Dialogue, and no lightweight method is viable for Summarisation.
3. Identification of Systematic Failure: The paper maps the "practical frontier" of GPU-free detection, highlighting a clear structural limit where lightweight methods fail on summarisation due to the nature of the hallucinations and context constraints.
4. Practical Guidance: Recommendations for method selection based on computational constraints and task type (e.g., using NLI for dialogue ranking, avoiding lightweight methods for summarisation).

Significance and Claims
The paper claims to provide a realistic performance baseline for the large community working without specialized hardware. It argues that comparing a single expensive method against weak baselines obscures the specific strengths and failure modes of accessible methods.

The authors conclude that while GPU-free detection is viable and effective for QA and Dialogue, it faces a structural limit on Summarisation. This failure is not a fault of a specific algorithm but a limitation of the lightweight regime itself, which lacks the capacity for claim-level decomposition or long-context modeling required for summarisation tasks. The work offers practitioners without specialized hardware a clear foundation for selecting methods and a realistic understanding of where accessible detection succeeds and where it breaks down.