💻 computer science

KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation

The paper introduces KnowHal, a comprehensive benchmark that unifies the evaluation of entity, attribute, relation, and knowledge hallucinations in Multimodal Large Language Models through a novel paired-question design, revealing that knowledge-related errors and false-premise acceptance remain significant challenges for current models.

Original authors: Ruihan Li, Jiyang Tan, Kailin Jiang, Huining Li, Hengyang Lu, Yu Huang, Qian Li, Yuntao Du

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruihan Li, Jiyang Tan, Kailin Jiang, Huining Li, Hengyang Lu, Yu Huang, Qian Li, Yuntao Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to understand the world. This robot, called a Multimodal Large Language Model (or MLLM for short), is like a brilliant student who can look at a picture and read a book at the same time. It's amazing at describing what it sees, like saying, "That's a red apple on a wooden table." But sometimes, this robot gets a little too creative. It might look at a picture of a cat and confidently tell you, "This cat is flying a spaceship," even though cats can't fly and there's no spaceship in the photo. This is called a "hallucination." It's like the robot is daydreaming while it's supposed to be working.

Why does this matter? Because we want to trust these robots with important jobs, like helping doctors diagnose illnesses or guiding self-driving cars. If a robot starts making things up, the results could be dangerous. Scientists have been trying to build tests to catch these daydreams. Most of these tests check if the robot can correctly identify simple things in a picture, like "Is there a dog?" or "Is the dog brown?" But there's a tricky part: sometimes the robot knows the facts about the world (like "dogs have tails") but gets confused when the picture doesn't match, or it invents facts that aren't in the picture at all. Until now, we didn't have a single, fair test that checked both what the robot sees and what it knows about the world, all in one go.

Enter KnowHal, a new and clever test designed to catch these daydreams in a whole new way. Think of KnowHal as a "spot the difference" game, but for robot brains. The researchers built a massive collection of 1,800 picture-and-question pairs, covering 10 different areas of life, from sports to music. For every single picture, they created two types of questions: "Positive" questions that ask about things actually in the photo or true facts about the object, and "Negative" questions that are tricky traps. These traps sound very plausible but are actually lies. For example, if a picture shows a real dog, a positive question asks, "What color is the dog?" A negative trap might ask, "What color is the dog's invisible blue hat?"

The researchers tested 14 different robot brains on this new game. They found some surprising things. First, the robots were generally okay at answering the "Positive" questions, but they stumbled badly when faced with the "Negative" traps. It's like a student who knows their math facts perfectly but gets tricked by a question that asks, "If 2 plus 2 equals 5, what is 2 plus 2?" The robots often said "5" instead of correcting the lie. This shows that even the smartest robots aren't very good at saying, "Wait, that's not true," when they are being misled.

The biggest surprise was that the hardest part for almost every robot was the "Knowledge" dimension. This is where the robot has to use facts it learned from books or the internet, not just what it sees in the picture. Even the best robots got this wrong more often than they got simple visual questions wrong. It suggests that while these robots are getting better at seeing, they are still struggling to separate what they see from what they think they know.

The study also looked at how the size of the robot brain affected its performance. They found that bigger brains generally did better, but even the biggest ones still struggled with the tricky "Knowledge" questions and the "Negative" traps. This tells us that just making the robot bigger isn't a magic fix; we need to teach them how to be more careful and skeptical.

In short, the paper introduces KnowHal, a new benchmark that pairs real questions with tricky, false ones to test how well robots can tell truth from fiction. The results suggest that while our AI friends are getting smarter, they still have a long way to go before they can be fully trusted to spot a lie or admit when they don't know the answer. The authors believe this new test will help future robots become more reliable and less prone to making up stories.

Technical Summary: KnowHal

Problem Statement

Despite significant advancements in Multimodal Large Language Models (MLLMs), hallucination—the discrepancy between model-generated content and ground truth—remains a critical barrier to reliable deployment in high-stakes domains such as medical diagnosis, legal reasoning, and autonomous driving. Existing evaluation benchmarks primarily focus on perceptual-level hallucinations (Entity, Attribute, and Relation) by asking "What does the model see?" However, they largely neglect knowledge-level hallucinations, where models generate factually incorrect information about depicted entities that contradicts external world knowledge. Furthermore, current frameworks lack a unified protocol to evaluate perceptual errors, knowledge errors, and the model's susceptibility to false premises (accepting misleading cues) under shared visual contexts. This leaves the question "What does the model know?" and its robustness against misleading inputs insufficiently addressed.

Methodology: The KnowHal Benchmark

To bridge this gap, the authors propose KnowHal, a comprehensive benchmark designed to evaluate hallucinations across four dimensions: Entity, Attribute, Relation, and Knowledge.

1. Benchmark Design and Structure

Paired Question Design: The core innovation of KnowHal is the construction of paired positive and negative questions for every image-entity sample.
- Positive Questions: Query valid visual evidence or factual knowledge associated with the entity.
- Negative Questions: Introduce carefully controlled, semantically plausible but factually false premises to test robustness.
Four Negative Induction Strategies:
1. Attribute Variant: Replacing an attribute with a similar but incorrect one.
2. Object Substitution: Replacing the target entity with a semantically related but absent entity.
3. Spatial Relation: Reversing or altering the actual spatial relationship between entities.
4. Action Fabrication: Assigning nonexistent actions or states to visible entities.
Dimensions:
- Entity, Attribute, Knowledge: Formulated as open-ended Visual Question Answering (VQA) tasks.
- Relation: Formulated as Multiple-Choice Questions (MCQs) to avoid ambiguity in synonymous descriptions.

2. Data Construction Pipeline

The dataset comprises 1,800 unique entity-image samples spanning 10 domains and 50 categories, resulting in 14,400 total QA pairs. The construction follows a semi-automated pipeline:

Entity Selection: Manual definition of 10 domains and identification of entities with distinctive visual characteristics.
Image Collection: LLMs analyze entities to generate relation-aware queries. Images are retrieved via Google Image Search and filtered using CLIP to ensure semantic consistency with the queries.
QA Pair Construction:
- Positive: Generated using templates (Entity), direct VLM observation (Attribute), or Wikipedia-derived facts (Knowledge).
- Negative: Generated via specific prompts implementing the four induction strategies.
Human Verification: Three annotators perform rigorous quality control, validating factual correctness, logical consistency, and answer uniqueness. Inter-annotator agreement was high (Fleiss' $\kappa = 0.854$ ).

3. Evaluation Metrics

The authors evaluate 14 representative MLLMs (5 closed-source, 9 open-source) in a zero-shot setting.

Primary Metric: Accuracy, reported separately for positive and negative questions across all four dimensions.
Hallucination Proneness Score (HPS): A derived metric defined as $HPS = (1 - \frac{Acc_{neg}}{Acc_{pos}}) \times 100\%$ . This measures the relative performance degradation when models are exposed to misleading inputs; higher scores indicate greater susceptibility to hallucination.
Judging: Positive questions use standard VQA soft matching; negative questions are evaluated by GPT-4o-mini as an automatic judge, with human verification confirming a 94.3% agreement rate.

Key Results

Experiments on 14 MLLMs reveal several systematic patterns:

Knowledge is the Hardest Dimension: Across nearly all models, the Knowledge dimension consistently yields the lowest accuracy, significantly lagging behind Entity, Attribute, and Relation. For instance, even top-performing models show substantial drops in knowledge accuracy compared to perceptual tasks.
Robustness Gap on Negative Questions: Most models exhibit significant performance degradation on negative questions compared to positive ones. This indicates that high accuracy on factual queries does not guarantee robustness against false premises.
Closed-Source vs. Open-Source: Closed-source models generally outperform open-source counterparts, particularly on positive questions. However, the Qwen series (open-source) demonstrates competitive robustness on negative questions, suggesting targeted training can mitigate hallucination.
Scaling Effects: Increasing model size generally improves both factual accuracy and robustness against misleading inputs. However, scaling does not uniformly resolve knowledge hallucinations; the Knowledge dimension remains a bottleneck even for larger models.
Domain Variability: Stronger models maintain balanced performance across domains, while weaker models show significant variance, particularly in Art and Science & Technology domains on negative questions.

Significance and Contributions

The paper claims three primary contributions:

Unified Framework: KnowHal addresses an important gap in existing evaluation frameworks by systematically unifying perceptual-level (Entity, Attribute, Relation) and knowledge-level hallucination evaluation within a single, image-centered protocol.
Controlled Evaluation via Paired Design: By introducing paired positive and negative questions, KnowHal enables a controlled comparison of a model's ability to interpret valid evidence versus its ability to reject plausible but false assumptions. This directly addresses the "false-premise acceptance" gap in existing literature.
Diagnostic Utility: Through extensive evaluation of 14 models and detailed analysis (scaling, domain-wise, and case studies), KnowHal reveals that knowledge hallucination is a persistent, distinct challenge and that current models lack robustness against misleading inputs, providing a diagnostic tool for future model development.

The authors conclude that while current MLLMs show progress, the Knowledge dimension and robustness to false premises remain critical challenges, and KnowHal provides the necessary infrastructure to drive research toward more reliable and factually accurate multimodal systems.