A multilingual hallucination benchmark: MultiWikiQHalluA
This paper introduces MultiWikiQHalluA, a multilingual hallucination benchmark covering 306 languages, and uses it to demonstrate that hallucination rates are significantly higher in lower-resource languages and smaller models, with the largest models performing best across evaluated European languages.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very smart, multilingual robot that loves to read Wikipedia articles and answer questions about them. You ask it, "What is the capital of Iceland?" and it replies, "Reykjavik." Great! But sometimes, this robot gets a little too creative. It might say, "Reykjavik is located on the moon," or "The capital is a city called 'Ice-berg'." Even though the sentence sounds smooth and confident, it's completely made up. In the world of AI, we call this hallucination.
This paper is like a new "lie detector" test for robots, but instead of just checking English, the researchers built a test that works for 306 different languages, including some that are rarely seen by computers (like Icelandic).
Here is the story of how they did it, explained simply:
1. The Problem: The "Fluent Liar"
Most tests for robot honesty only check English. It's like only testing a driver's license in one city and assuming they can drive safely everywhere else. The researchers wanted to know: Do robots lie more often when speaking languages they don't know as well?
They defined "hallucination" specifically as faithfulness. This means: Did the robot stick to the story you told it?
- Good Robot: Reads a story about a cat, then says, "The cat is orange."
- Hallucinating Robot: Reads a story about a cat, then says, "The cat is actually a dragon." (It sounds plausible, but it's not in the story).
2. The Solution: Creating a "Fake News" Factory
To teach their lie detector, the researchers needed a massive pile of examples where the robot did lie. But they didn't want to wait for robots to lie naturally; that takes too long.
So, they built a synthetic factory:
- They took a huge dataset of real questions and answers (MultiWikiQA) covering 306 languages.
- They used a super-smart AI (GPT-5) to intentionally rewrite the correct answers, making them sound believable but factually wrong based on the source text.
- They did this for 306 languages, creating a giant library of "fake answers."
- They then trained a smaller, specialized AI (a "token-level classifier") to spot these lies. Think of this classifier as a spell-checker for truth. Instead of checking if a word is spelled right, it checks if a specific word in a sentence belongs to the original story or if it's a made-up invention.
They trained these "truth-spell-checkers" for 30 European languages and tested them on four: English, German, Danish, and Icelandic.
3. The Experiment: Testing the Robots
The researchers took five different AI models (ranging from a tiny one to a giant one) and asked them to answer questions in those four languages. Then, they ran the answers through their new "truth-spell-checker" to see how many lies were hidden in the text.
4. The Results: Size Matters, But Language Matters More
Here is what they found, using some simple analogies:
- The "Small Robot" Struggles: The smallest AI model (Qwen3-0.6B) was the worst liar. It was like a student who didn't study enough and just guessed. In Icelandic, 60% of its answers contained at least one lie.
- The "Big Robots" are Better: The larger, more powerful models (like the 70-billion-parameter ones) were much better at sticking to the facts. They were the "honor students" of the group.
- The "Language Gap": This is the most important finding. The robots lied much more often in languages they knew less about.
- English and German (languages the robots know very well): Very few lies.
- Danish and Icelandic (languages the robots know less well): Many more lies.
- Icelandic was the hardest. Even the best robots lied more here than in English. It's like asking a chef to cook a dish they've never seen before; they are more likely to guess the ingredients wrong.
5. A Surprising Twist
You might think, "The bigger the robot, the less it lies." But it's not that simple. Sometimes a medium-sized robot was better than a giant one, depending on the language. This suggests that how the robot was trained (what languages it studied) matters just as much as how big it is.
6. The Catch (The "Token" Issue)
The researchers also noted a technical quirk. In languages like Icelandic, the computer breaks words into many tiny pieces (tokens) to understand them. In English, it might see one word as one piece. In Icelandic, that same word might be five pieces.
- The Analogy: Imagine checking a sentence for errors. If you check it word-by-word, you might find 1 error. If you check it letter-by-letter, you might find 5 errors just because the word is longer.
- The researchers warn that their "lie count" might look higher for Icelandic partly because the words are chopped up into more pieces, giving the detector more chances to find a "lie," even if the meaning is the same.
Summary
The paper introduces a new, open-source tool that can detect when AI models are making things up in 306 languages. They found that while bigger models are generally more honest, robots are significantly more prone to hallucinating in languages they are less familiar with, like Icelandic. They have released their "fake news" datasets and their "truth detectors" for anyone to use.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.