Technical Summary: HalluVerse-M3

Problem Statement

Large Language Models (LLMs) frequently generate coherent text that lacks factual grounding, a phenomenon known as hallucination. This issue poses significant risks in mission-critical domains such as healthcare, law, and executive decision-making. While recent research has produced datasets to analyze hallucinations, existing resources suffer from three primary limitations:

1. Language Bias: They predominantly focus on English, limiting the understanding of hallucination patterns in other linguistic contexts.
2. Coarse Granularity: Most treat hallucination as a binary phenomenon (factual vs. hallucinated), obscuring the diversity of error types (e.g., entity substitution vs. relation distortion).
3. Task Specificity: Resources typically address single generation tasks, hindering systematic comparison across different application contexts like question answering and summarization.

Consequently, there is a lack of unified frameworks capable of analyzing fine-grained hallucinations across multiple languages and tasks.

Methodology

The authors introduce HalluVerse-M3, a dataset designed to enable systematic analysis of hallucinations through a controlled construction pipeline.

Dataset Construction

• Languages and Tasks: The dataset covers four languages (English, Arabic, Hindi, Turkish) and two generation tasks: Question Answering (QA) and Dialogue Summarization (DS).
• Source Data:
  • QA: Derived from a factual QA dataset by Lin et al. [2022], using reference answers grounded in verifiable evidence.
  • DS: Derived from the DialogSum dataset [Chen et al., 2021], focusing on summaries of concrete events and entities.
• Multilingual Preparation: English serves as the source. Parallel outputs for Arabic, Hindi, and Turkish were generated via automatic translation (Google Translate API) followed by rigorous manual verification by native speakers to ensure grammatical correctness and semantic faithfulness.
• Hallucination Injection: A controlled editing process was employed to create hallucinated outputs ($\tilde{y}$) from reference outputs ($y^*$). This process involves:
  • Removing or altering specific reference propositions ($\Delta^-$).
  • Introducing exactly one new, unsupported proposition ($\Delta^+$) such that $|\Delta^+| = 1$.
  • Ensuring the new proposition is not supported by the reference ($\Delta^+ \cap P(y^*) = \emptyset$).
• Fine-Grained Annotation: The dataset distinguishes between three mutually exclusive hallucination types based on the nature of the injected proposition $p^+$:
  1. Entity-level: The predicate and attributes match the reference, but at least one entity argument differs.
  2. Relation-level: The entity arguments match, but the predicate (relation) or attributes (e.g., temporal order, causality) differ.
  3. Sentence-level: The proposition has no aligned reference proposition; it is an unanchored statement.
• Validation: Two native speakers per language independently annotated the data. Instances with grammatical errors or unclear hallucinations were labeled as null (0) and excluded. Inter-annotator agreement was substantial to near-perfect (Cohen's $\kappa$ ranging from 0.74 to 0.83).

Experimental Setup

The authors evaluated a diverse set of 14 contemporary LLMs (both open-source and proprietary) on the task of fine-grained hallucination detection.

• Models: Included Phi-4, Mistral-7B, Qwen-2.5 variants, Gemma-2, LLaMA-3.3, DeepSeek-V2.5, PaLM 2, Claude-3.5, Gemini-2.5, and GPT-4 variants.
• Protocol: Evaluation was conducted using prompting only (no fine-tuning). Models were tasked with identifying the hallucination type (Entity, Relation, or Sentence) given a pair of original and hallucinated outputs.
• Metrics: Accuracy was reported as the primary metric, analyzed across tasks, languages, and hallucination types.

Key Results

The evaluation yielded several consistent trends regarding model performance:

1. Task Difficulty: Question Answering (QA) is consistently easier for models to evaluate than Dialogue Summarization (DS). This is attributed to the localized nature of errors in QA versus the diffuse, context-dependent nature of hallucinations in summarization.
2. Hallucination Type Difficulty: A clear performance gradient exists across hallucination types:
   • Entity-level: Easiest to detect. Frontier models (e.g., GPT-4o) achieved >90% accuracy in QA.
   • Relation-level: Moderately difficult, requiring deeper semantic reasoning.
   • Sentence-level: Most challenging. Even the strongest models struggled, with accuracy often dropping to 50–65% in summarization tasks.
3. Language Effects: Performance is highest in English and degrades in lower-resource languages. Hindi consistently exhibited the lowest detection accuracy, particularly in summarization tasks.
4. Model Comparison: Proprietary models generally outperformed open-source systems, though larger open-source models (e.g., DeepSeek-V2.5, LLaMA-3.3-70B) performed competitively in QA. GPT-4o achieved the strongest overall performance, likely benefiting from its involvement in the hallucination injection process.

Significance and Claims

The paper claims that HalluVerse-M3 provides a realistic and challenging benchmark for studying hallucinations in multilingual, multi-task settings. Its significance lies in:

• Granularity: Moving beyond binary classification to distinguish between entity, relation, and sentence-level errors.
• Multilingual Scope: Offering a rare resource for analyzing hallucination patterns across typologically diverse languages (English, Arabic, Hindi, Turkish).
• Task Diversity: Enabling controlled comparisons between grounded tasks (QA) and abstractive tasks (Summarization).

The authors conclude that while recent frontier models show strong performance in detecting entity-level errors in QA, sentence-level hallucinations in generative tasks remain an open problem, even for the most advanced models. The dataset is released to support future research in detection and mitigation, emphasizing the need for methods that move beyond surface-level cues toward deeper semantic grounding.

Ethical Considerations

The dataset contains deliberately fabricated information. The authors note the risk of misuse for training systems to generate misleading content. To mitigate this, the dataset is released under a CC BY-NC 4.0 license for non-commercial research only, with explicit labeling of hallucinated instances and their types. Content safety screening indicated a low presence of offensive language.