💬 NLP

MUCH: A Multilingual Claim Hallucination Benchmark

The paper introduces MUCH, the first multilingual benchmark for claim-level uncertainty quantification that features a highly efficient deterministic segmentation algorithm and released generation logits to enable fair, reproducible, and realistic evaluation of Large Language Model reliability across four European languages.

Original authors: Jérémie Dentan, Alexi Canesse, Davide Buscaldi, Aymen Shabou, Sonia Vanier

Published 2026-02-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jérémie Dentan, Alexi Canesse, Davide Buscaldi, Aymen Shabou, Sonia Vanier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, very fast robot librarian (a Large Language Model, or LLM) who can answer any question you ask. The problem is, sometimes this librarian is so eager to please that they make things up. They might tell you the capital of France is "Paris, but also a giant cheese wheel," and they say it with such confidence that you believe them. This is called a hallucination.

The paper "MUCH" introduces a new way to test how well we can catch these lies before they cause trouble. Here is the story of how they did it, explained simply.

1. The Problem: The "Whole Cake" vs. The "Bad Slice"

Previously, if you asked a robot librarian a long question and got a long answer, existing tests would give the entire answer a single grade.

The Old Way: If the librarian wrote a 10-page essay and got one tiny fact wrong on page 7, the whole essay gets an "F." That's unfair if the other 9 pages were perfect.
The New Way (MUCH): The authors wanted to slice the cake. They wanted to check every single "claim" (every distinct idea) in the answer individually. Did they get the price right? Did they get the date right? This is called Claim-Level Uncertainty Quantification.

2. The Solution: Building the "MUCH" Benchmark

The team built a giant testing ground called MUCH (Multilingual Claim Hallucination Benchmark). Think of it as a massive "Hallucination Gym" for AI.

Here are the four special tools they built for this gym:

A. The Multilingual Playground

They didn't just test in English. They built the gym in four languages: English, French, Spanish, and German. This ensures the robots aren't just good at lying in one language but are tested everywhere. They gathered nearly 5,000 questions and answers from four different AI models.

B. The "X-Ray Glasses" (Logits)

Most AI tests only look at the final text the robot writes. But the authors realized that to build better lie detectors, you need to see what the robot was thinking while it wrote.

The Analogy: Imagine a magician pulling a rabbit out of a hat. Most people just see the rabbit. The authors kept the "magic trick" logs (called logits). These logs show the probability the robot assigned to every single word it considered.
Why it matters: This allows researchers to build "white-box" detectors that can peek inside the robot's brain to see if it was unsure about a word before it even said it.

C. The "Super-Fast Slicer" (much_segmenter)

To check individual claims, you first have to cut the long answer into small, bite-sized pieces.

The Old Problem: Previous methods used other AIs or humans to cut the text. This was slow, expensive, and sometimes inconsistent (like using a shaky hand to slice bread).
The MUCH Solution: They built a new tool called much_segmenter. It's a simple, rule-based algorithm (like a very precise knife) that cuts the text based on punctuation and common words.
The Speed: It is incredibly fast. It takes only 0.2% of the time it takes the robot to write the answer. It's like slicing a loaf of bread in the blink of an eye while the baker is still kneading the dough. This makes it possible to check for lies in real-time.

D. The "Silver Standard" Judges

Ideally, humans would check every single fact. But with 20,000+ claims, that would take years.

The Trick: They used two super-smart AI judges (GPT-4o and GPT-4.1) to check the facts against Wikipedia.
The Safety Net: They only kept the answers where both AI judges agreed. If the judges disagreed, they threw that answer out. This created a "Silver Standard"—a dataset that is almost as good as human-checked data but was created much faster.

3. The Results: The Robots Are Still Lying

The authors tested the best existing "lie detector" methods on their new gym.

The Good News: The methods work better than random guessing.
The Bad News: They are still not good enough for real life.
- Accuracy: The best method only caught about half of the lies when it was trying to be very careful not to cry wolf (false alarms).
- Speed: The best method was actually slower than the robot writing the answer in the first place! It took 124% longer to check the answer than to write it. That's like hiring a security guard who takes longer to check your ID than it takes for you to walk through the door.

4. The Takeaway

The paper concludes that we have a great new playground (MUCH) and a super-fast knife (much_segmenter) to slice up AI answers. However, the "lie detectors" we have today are still too slow and not accurate enough to trust in the real world.

In short: We finally have a way to test AI lies in detail and in multiple languages, but we still need to invent better, faster lie detectors before we can trust these robots with our most important information.

1. Problem Statement

Large Language Models (LLMs) are prone to "hallucinations" (generating plausible but non-factual content). While Uncertainty Quantification (UQ) techniques exist to estimate model confidence, current benchmarks suffer from two critical limitations that hinder the development of robust, real-world UQ methods:

Lack of White-Box Data: Existing benchmarks do not release generation logits (probability distributions) for tokens. This prevents the evaluation of "white-box" UQ methods that rely on internal model states.
Unrealistic Segmentation: Current claim-level benchmarks rely on manual annotation or LLM-based segmentation to split generations into distinct claims. These methods are non-deterministic, computationally expensive (often requiring as much time as the generation itself), and difficult to reproduce, making them unsuitable for real-time monitoring in production environments.

2. Methodology: The MUCH Benchmark

The authors introduce MUCH, a multilingual benchmark designed for fair, reproducible, and efficient evaluation of claim-level UQ.

A. Dataset Construction

Source: Questions are filtered from the Mu-SHROOM dataset, covering four European languages: English, French, Spanish, and German.
Generation: 4,873 samples were generated using four instruction-tuned open-weight models (Llama 3.1 8B, Llama 3.2 3B, Ministral 8B, and Gemma 3 4B) at two different temperatures (0.7 and 1.0).
Logits: Crucially, the dataset includes 24 logits per generated token, enabling the development and testing of white-box UQ methods without re-generating data.
Annotation: Factuality is annotated at the claim level (distinct ideas) rather than the response level.
- Silver Standard: Claims were automatically annotated using GPT-4o and GPT-4.1 against corresponding Wikipedia pages.
- Quality Control: Samples were retained only if both GPT models agreed on the factuality of every claim. This filtering process removed ambiguous cases, resulting in a final set of 4,873 samples containing 20,751 claims.
- Verification: A subset was manually annotated by humans, showing high agreement ( $\kappa \approx 0.92$ ) with the automated labels after filtering, validating the "silver standard."

B. The `much_segmenter` Algorithm

To address the segmentation bottleneck, the authors propose much_segmenter, a novel, deterministic algorithm:

Mechanism: It is a rule-based system using keyword and punctuation cues (stopwords, punctuation marks) to identify claim boundaries. It utilizes nltk's TreebankWordTokenizer to split text into words and maps character indices to LLM token indices.
Advantages:
- Deterministic: Produces identical results every time, unlike LLM-based segmentation.
- Efficiency: It is extremely fast, requiring only 0.2% of the computation time needed for LLM generation.
- Token Alignment: It directly maps claims to the specific tokens generated by the target LLM, solving the alignment issues common in prompt-based segmentation.

3. Key Contributions

MUCH Dataset: A multilingual (EN, FR, ES, DE) benchmark with 4,873 samples and 20,751 claim-level factuality annotations.
Logit Release: The first claim-level benchmark to release 24 logits per token, facilitating white-box research.
much_segmenter: A fast, reproducible, and language-independent (for Latin-script European languages) segmentation tool that enables real-time UQ evaluation.
Comprehensive Baselines: A rigorous evaluation of state-of-the-art UQ methods (CCP, SAR, Token Likelihood, Token Entropy, Max Likelihood) under realistic constraints.

4. Results and Evaluation

The authors evaluated five baseline methods on MUCH, measuring both performance (ROC-AUC, PR-AUC) and efficiency (runtime relative to generation).

Performance:
- CCP (Claim Consistency Prompting) achieved the highest performance with a ROC-AUC of 0.772.
- Other methods (SAR, Token Entropy, etc.) scored slightly lower (0.73–0.75).
- Language Gap: Performance was significantly higher in English than in other languages, largely because the baselines rely on NLI models (e.g., DeBERTa) optimized primarily for English.
Efficiency Trade-off:
- CCP was computationally prohibitive, adding 124% overhead to the LLM generation time (due to the NLI model).
- Lightweight methods (Token Likelihood, Max Likelihood) added only 0.3–0.5% overhead but offered lower accuracy.
- Segmentation: The introduction of much_segmenter reduced the segmentation overhead to negligible levels (0.2%), a massive improvement over previous LLM-based approaches.
Current Limitations: Even the best method (CCP) only achieves a True Positive Rate of ~48% at a 10% False Positive Rate, indicating that current UQ methods are not yet reliable enough for high-stakes real-time deployment.

5. Significance and Future Directions

Real-World Applicability: MUCH shifts the focus from theoretical benchmarks to realistic deployment constraints by emphasizing efficiency and determinism. It proves that claim-level UQ can be monitored in real-time if segmentation is efficient.
White-Box Research: By releasing logits, the authors enable a new wave of research into internal model states for hallucination detection, moving beyond black-box text-only analysis.
Call to Action: The paper highlights that future methods must improve precision (low false positives) and efficiency (minimal overhead) to be viable. It also identifies the need for better multilingual NLI models to close the performance gap in non-English languages.

In summary, MUCH provides the necessary infrastructure (data, tools, and protocols) to transition claim-level uncertainty quantification from an academic exercise to a practical tool for monitoring LLM reliability.