Technical Summary: BenHalluEval

Problem Statement

Despite Bengali being the sixth most spoken language globally, there has been no systematic evaluation of hallucinations in Large Language Models (LLMs) for this language. Existing benchmarks for hallucination detection (e.g., TruthfulQA, FActScore, HaluEval) focus primarily on English or other high-resource languages like Arabic and Hindi. Furthermore, current evaluation methods often rely on single-track protocols that test only on hallucinated candidates, failing to account for models that may exhibit uniform response biases (e.g., always predicting "hallucination" or "no hallucination"). Additionally, the specific behavior of LLMs on Bangla-English code-mixed inputs—a common usage pattern in Bengali digital communication—remains unexamined.

Methodology

The authors introduce BenHalluEval, a multi-task hallucination evaluation framework designed specifically for Bengali and Bangla-English code-mixed inputs. The framework operates through a three-stage pipeline:

1. Seed Dataset Selection and Filtering

The framework constructs a benchmark using four distinct tasks, drawing from existing datasets and applying specific filtering strategies to identify "hard" samples prone to hallucination:

• Generative Question Answering (GQA): Uses TyDiQA-GoldP (Bengali). Samples were filtered by prompting three models (Qwen2.5-14B, Gemma-2-9B, DeepSeek-R1-14B) without context; instances where all three failed were selected as the seed set (1,000 samples).
• Bangla-English Code-Mixed QA: The 1,000 GQA seed questions were converted to Bangla-English (Roman script) using GPT-5.4, while retaining original Bengali answers.
• Summarization: Uses BanglaCHQ-Summ (1,800 medical query-summary pairs). Samples were filtered based on low BERTScore F1 similarity when summarized by three models, selecting the 1,000 most difficult instances.
• Reasoning: Uses SOMADHAN (8,792 Bengali math problems). Samples were stratified by reasoning chain length (500 longest, 500 shortest) to ensure diverse difficulty levels.

2. Hallucinated Sample Generation

Using GPT-5.4, the authors generated 12,000 hallucinated candidates across the four tasks. The generation process employed task-specific hallucination taxonomies:

• GQA & Code-Mixed QA (4 types): Factualness (fabricating facts), Comprehension (misunderstanding intent), Specificity (inappropriate detail level), and Inference (unsupported conclusions).
• Summarization (3 types): Fabricated Content, Non-Factual Addition, and Direct Contradiction.
• Reasoning (5 types): Arithmetic Slip, Formula Misapplication, Invalid Deduction, Hallucinated Intermediate Fact, and Semantic Drift.

3. Dual-Track Evaluation Protocol

To address the limitations of single-track evaluation, the authors propose a dual-track protocol evaluated on seven LLMs (spanning reasoning-oriented, multilingual, and Bengali-centric categories):

• Track A: Presents ground-truth correct answers. The expected verdict is "No" (not hallucinated). This measures the False Positive Rate.
• Track B: Presents the generated hallucinated candidates. The expected verdict is "Yes". This measures the Hallucination Detection Rate.

4. BenHalluScore (BHS)

The authors introduce BenHalluScore, a calibration metric designed to jointly penalize both failure modes (false positives on ground truth and missed detections on hallucinated data).
$$\text{BenHalluScore} = \frac{1}{2} \left( \frac{W_A}{N_A} + \frac{W_B}{N_B} \right) \times 100\%$$
Where $W$ represents wrong verdicts and $N$ represents total instances per track. A lower score indicates better calibration; a score of 50% implies random or uniformly biased behavior.

Key Results

The evaluation of seven models (including DeepSeek-R1-14B, GPT-4.1 mini, Qwen2.5-32B, Gemma-2-27B, Mistral-nemo-12B, LLaMA-3.1-8B, and TigerLLM-9B) revealed significant variations in performance:

• Performance Range: BenHalluScores ranged from 7.72% (Qwen2.5-32B on Summarization) to 55.42% (Mistral-nemo-12B on Code-Mixed QA).
• Task Variance: Summarization tasks generally yielded lower (better) scores than GQA or Reasoning, suggesting that access to full source documents aids factual consistency. Reasoning tasks showed the widest performance spread, indicating high difficulty in detecting reasoning hallucinations in Bengali.
• Model Categories:
  • GPT-4.1 mini achieved the best balance on GQA (15.56%), maintaining low errors on both tracks.
  • TigerLLM-9B (a Bengali-centric model) outperformed significantly larger multilingual models on GQA and Summarization, suggesting that language-specific pretraining is a stronger predictor of calibration than parameter count.
  • Mistral-nemo-12B and LLaMA-3.1-8B exhibited severe systematic biases. For instance, Mistral-nemo-12B flagged nearly all ground-truth answers as hallucinations (87.70% Track A error), while LLaMA-3.1-8B failed to detect most hallucinations (81.63% Track B error).
• Code-Mixed vs. Native Script: Some multilingual models (Qwen2.5, Gemma-2, DeepSeek-R1) performed better on Bangla-English code-mixed inputs than on native Bengali script, likely due to stronger English-anchored representations. Conversely, TigerLLM-9B and GPT-4.1 mini performed better on native Bengali.
• Mitigation (Chain-of-Thought): Applying Chain-of-Thought (CoT) prompting did not consistently improve hallucination discrimination. In several cases, CoT merely shifted a model's bias from a uniform "No" to a uniform "Yes" (or vice versa), without genuinely improving detection accuracy.

Significance and Contributions

The paper claims the following contributions and significance:

1. First Dedicated Benchmark: BenHalluEval establishes the first dedicated hallucination benchmark for Bengali, covering both native and code-mixed inputs.
2. Dual-Track Calibration: The work highlights the inadequacy of single-track evaluation for low-resource languages. By introducing Track A and Track B, it reveals failure modes (uniform response bias) that would otherwise be masked by high detection rates on hallucinated data alone.
3. Language-Specific Optimization: The results challenge the assumption that larger parameter counts guarantee better performance in low-resource settings, demonstrating that Bengali-centric pretraining (TigerLLM-9B) can yield superior calibration compared to larger multilingual models.
4. Prompting Limitations: The study provides empirical evidence that CoT prompting is not a universal mitigation strategy for hallucinations in Bengali, as it often alters prediction distributions without improving discrimination capabilities.

The authors conclude that reliable hallucination evaluation for Bengali requires specialized, dual-track frameworks and that current prompting strategies alone are insufficient for mitigating hallucinations in this domain. The dataset and code are made available to facilitate further research.