💻 computer science

FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs

This paper introduces FREAK, a comprehensive fine-grained hallucination evaluation benchmark utilizing high-quality photorealistic images with counter-commonsense edits to reveal severe hallucination issues in state-of-the-art Multimodal Large Language Models and provide deeper insights into their reasoning processes through controlled subsets and Chain-of-Thought analysis.

Original authors: Zhihan Yin, Jianxin Liang, Yueqian Wang, Yifeng Yao, Huishuai Zhang, Dongyan Zhao

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhihan Yin, Jianxin Liang, Yueqian Wang, Yifeng Yao, Huishuai Zhang, Dongyan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot friend who can look at pictures and describe what it sees. You might think, "Great! It sees a dog, so it says 'dog'." But what if the robot is actually looking at a picture of a cat wearing a dog costume, and it confidently says, "That's a dog"?

That's what happens with Multimodal Large Language Models (MLLMs). They are incredibly smart, but they often suffer from hallucinations. They rely so much on what they "know" from their training (like "dogs usually have four legs") that they ignore what is actually in the picture (like a dog with five legs).

This paper introduces a new tool called FREAK to catch these robots in the act.

What is FREAK?

Think of FREAK as a "Spot the Difference" game designed to trick the robots.

The researchers created a special set of images that look perfectly normal at first glance, like a photo of a traffic light or a guitar. But, they have secretly tweaked tiny, specific details to break the laws of reality:

A traffic light where the red light is at the bottom and the green is at the top.
A guitar with five strings instead of six.
A clock with four hands instead of three.

These are called Counter-Commonsense (CCS) edits. They are subtle enough that a human would spot them immediately, but they are designed to confuse the robot's "common sense."

How They Made the Test (The "Baking" Analogy)

Creating these tricky images wasn't easy. The researchers didn't just draw them by hand (too slow!). Instead, they built an automated assembly line:

The Idea Chef (LLM): A text AI comes up with a crazy idea, like "A traffic light with the lights swapped."
The Baker (Image Generator): Another AI bakes a perfect, realistic photo of a normal traffic light.
The Pastry Chef (Image Editor): A third AI takes that normal photo and surgically swaps the red and green lights, making it look real but wrong.
The Taster (Human Review): Real humans check the photos to make sure the trick is clear and the image looks real.

The Results: The Robots Failed Miserably

The researchers tested the world's smartest AI models (like GPT-4, Gemini, and others) on this FREAK test.

Humans: Got about 87% right. It was easy for us because we actually looked at the picture.
AI Models: Only got about 45% right.

The Metaphor: Imagine you show a picture of a square circle to a human and a robot. The human says, "That's a square circle!" The robot, remembering that circles are round and squares are square, says, "No, that's a circle," or "That's a square," completely ignoring the weird shape in front of its eyes. The AI is so confident in its "knowledge" that it refuses to believe its own "eyes."

The "Thinking" Trap (Chain-of-Thought)

The researchers tried a popular trick called Chain-of-Thought (CoT), where they asked the robots to "think step-by-step" before answering, hoping it would help them slow down and look closer.

The Result: It made things worse.

Analogy: It's like asking a person who is bad at math to "talk through their steps" while solving a problem. Instead of helping, they start talking themselves into a corner, convincing themselves that the wrong answer is right because it sounds logical in their head.
The AI models started "reasoning" their way into a hallucination. They would look at the red light at the bottom, think, "Traffic lights usually have red on top," and then talk themselves into ignoring the red light at the bottom.

Why This Matters

This paper is a wake-up call. It shows that even the most advanced AI models are still blind to details when their internal knowledge clashes with reality. They are like a person who has memorized a map of a city but refuses to look out the window when the buildings have changed.

FREAK gives us a new way to measure exactly how "blind" these models are, helping developers build better, more honest AI that actually sees what is there, rather than just guessing what should be there.

1. Problem Statement

Multimodal Large Language Models (MLLMs) frequently suffer from hallucinations, where they generate content that is logically plausible but factually inconsistent with the visual input. While existing benchmarks (e.g., POPE, AMBER) have been useful, they face three critical limitations:

Metric Saturation: Current SOTA models perform near-perfectly on these benchmarks due to over-simplified tasks (often binary true/false judgments), rendering them unable to discriminate between model capabilities.
Data Contamination: Many benchmarks use images from open-source datasets that likely overlap with model training data, leading to memorization rather than genuine visual reasoning.
Lack of Fine-Grained Diversity: Existing AI-generated benchmarks often lack photorealism, suffer from low sample diversity, or fail to test specific localized details (e.g., object attributes, counting, spatial relationships) where models default to "commonsense" priors over visual evidence.

The core problem addressed is the inability to accurately quantify fine-grained hallucinations—where models misperceive or fabricate localized details in an image despite the overall scene appearing normal.

2. Methodology: The FREAK Benchmark

The authors propose FREAK (Fine-gRained Evaluation Against Knowledge), a comprehensive benchmark designed to test MLLMs' ability to perceive details that contradict commonsense knowledge.

A. Data Generation Pipeline

FREAK utilizes a novel "Generate-Then-Edit" pipeline to create high-quality, photorealistic Counter-Commonsense (CCS) images:

CCS Content Generation: LLMs are prompted to generate specific attribute descriptions ( $W$ ) for a target object ( $O$ ) that contradict reality (e.g., a traffic light with the red light at the bottom) while maintaining the object's overall structure.
Image Synthesis:
- Step 1 (Generation): An image generation model (Seedream 3.0) creates a normal, commonsense-compliant image ( $P$ ) based on the object and its correct attributes ( $A$ ).
- Step 2 (Editing): An image editing model (SeedEdit 3.0) modifies $P$ using the CCS description ( $W$ ) to introduce localized, counter-commonsense details. This ensures the image remains photorealistic while containing specific anomalies.
Question Construction: LLMs generate corresponding questions (both Multiple-Choice and Free-Form) based on the tuple $(O, A, W, CCS)$.
- Distractors: Multiple-choice questions include a "Commonsense Distractor" (the hallucinated answer based on $A$ ) and an "AI-generated Distractor."
Human Verification: A crucial manual verification step ensures image quality, removes semantic redundancy, and validates that the anomalies are detectable by humans but challenging for models.

B. Benchmark Statistics

Scale: 1,786 CCS images and 1,799 questions.
Categories: Six subtasks covering Detection, Counting, Attribute, Analysis, Position, and OCR.
Format: Mix of Free-Form questions (evaluated via LLM-as-a-judge) and Multiple-Choice Questions (evaluated via accuracy).

3. Key Contributions

Automated CCS Pipeline: A scalable method integrating LLMs, image generation, and image editing models to produce realistic images with localized, detail-level contradictions.
FREAK Benchmark: A novel evaluation framework featuring objective methodologies, diverse CCS content, and high-quality images that expose severe hallucination issues in SOTA models.
CoT Analysis: A systematic investigation into Chain-of-Thought (CoT) prompting, revealing that reasoning often degrades performance in fine-grained hallucination tasks.

4. Experimental Results

The authors evaluated a wide range of SOTA MLLMs (including GPT-4.1, Gemini-2.5, Qwen2.5-VL, InternVL3, and reasoning models like o3 and Kimi-VL-A3B-Thinking).

Performance Gap: There is a massive gap between human and model performance.
- Human Baseline: ~86.71% accuracy.
- SOTA Models: Average accuracy is only ~45% (ranging from 30% to 43% for most mainstream models).
Task-Specific Failures:
- Models perform worst on Counting tasks (often failing even with small numbers).
- Performance is slightly better on low-level tasks (Attribute, OCR) but degrades significantly on high-level reasoning tasks (Analysis, Position), suggesting a heavy reliance on linguistic priors.
Impact of Reasoning (CoT):
- Contrary to expectations, enabling CoT or using reasoning-specialized models degraded performance for most MLLMs.
- Hallucination Rate: The rate of models selecting commonsense distractors increased when reasoning was activated.
- Mechanism: Analysis of probability trajectories during reasoning showed that models often start with the correct answer but drift toward incorrect, hallucinated choices as the reasoning process continues, reinforcing distractors.
Normal vs. CCS: When tested on normal images, models achieved ~90% accuracy. Switching to CCS images caused a sharp drop to ~35-45%, proving the benchmark effectively isolates hallucination rather than general comprehension failure.

5. Significance and Conclusion

New Standard: FREAK establishes a new, rigorous standard for evaluating MLLMs, moving beyond saturated binary benchmarks to test fine-grained visual perception.
Revealing Limitations: The results highlight a critical flaw in current MLLMs: they struggle to override parametric knowledge (commonsense) with visual evidence, even when the visual evidence is clear.
CoT Ineffectiveness: The study provides evidence that simply adding reasoning steps (CoT) does not solve hallucination; in fact, it may exacerbate it by allowing the model's internal biases to dominate the reasoning chain.
Future Directions: The paper suggests that future improvements require better alignment between visual encoders and language models, rather than just scaling up reasoning capabilities.

In summary, FREAK demonstrates that despite rapid advancements, current MLLMs remain fundamentally unreliable in detailed visual perception, often "seeing" what they expect to see rather than what is actually there.