Technical Summary: MM-THEBench: Do Reasoning MLLMs Think Reasonably?

Problem Statement

Recent advancements in Multimodal Large Language Models (MLLMs) have shifted the paradigm from non-thinking models to post-trained reasoning models capable of generating intermediate Chain-of-Thoughts (CoTs) to solve complex problems. While these intermediate steps offer explainability, a critical gap remains in understanding whether this "thinking" process mitigates hallucinations or introduces new ones. Existing benchmarks primarily evaluate the correctness of final outputs, neglecting the internal reasoning process. Consequently, it is unclear if correct final answers are derived from faithful reasoning or if models can arrive at coincidentally correct answers despite significant hallucinations in their intermediate steps. Furthermore, the lack of a systematic taxonomy for intermediate CoT hallucinations and the difficulty of evaluating free-form, long-form reasoning steps hinder the development of robust reasoning MLLMs.

Methodology

To address these challenges, the authors introduce MM-THEBENCH (Multimodal thinking hallucination evaluation benchmark), a comprehensive framework designed to assess hallucinations in the intermediate CoTs of reasoning MLLMs.

1. Benchmark Design and Taxonomy

MM-THEBENCH defines a two-layer hallucination taxonomy grounded in three cognitive dimensions:

• Knowledge: Subdivided into world knowledge, commonsense, and domain knowledge.
• Perception: Subdivided into recognition, OCR, spatial, count, audio, grounding, and temporal.
• Reasoning: Subdivided into deductive, inductive, spatial, arithmetic, causal, decision, and instructional.

This hierarchical structure allows for fine-grained analysis of where and how hallucinations occur within the reasoning chain.

2. Data Construction

The benchmark comprises 1,340 diverse multimodal questions spanning images and videos, sourced from eight high-quality datasets (e.g., MathVision, MM-vet-v2, MMMU-pro, HallusionBench, Omni-Spatial, CharXiv, GUI-Agent, Video-MME).

• Annotation Process: A hybrid pipeline was employed. First, a state-of-the-art reasoning MLLM (Gemini-2.5-pro) generated atomic reasoning steps. Second, human annotators verified these steps for correctness, completeness, and necessity, correcting ambiguities.
• Rubric Generation: Based on verified chains, fine-grained rubrics were constructed for each question. Each rubric item corresponds to an atomic step, tagged with a cognitive dimension and a binary satisfaction indicator.

3. Evaluation Framework

The evaluation utilizes an LLM-as-a-Judge approach (employing Qwen-3-32B) to automate the assessment of free-form CoTs through three levels:

• Answer-level: Verifies the correctness of the final output against ground truth.
• Step-level: Segments the model's reasoning into atomic steps and aligns them with annotated ground-truth steps to calculate Precision, Recall, and F1-score.
• Rubric-level: The judge model evaluates whether specific rubric items are satisfied and identifies hallucinations. This yields normalized scores for Knowledge (K), Perception (P), and Reasoning (R), as well as an H-score (Hallucination-free score).

Key Contributions

1. MM-THEBENCH Construction: The design and construction of a comprehensive benchmark featuring a fine-grained hallucination taxonomy specifically for multimodal reasoning CoTs.
2. Automated Evaluation Framework: The development of a multi-level, automated evaluation pipeline that quantifies hallucination categories and reasoning quality without requiring manual inspection of thousands of tokens.
3. Empirical Analysis: A systematic evaluation of 14 state-of-the-art reasoning MLLMs (including GPT-5, OpenAI-o3, and various open-source models) revealing consistent patterns in thinking hallucinations.

Experimental Results

The evaluation of 14 reasoning MLLMs on MM-THEBENCH yields several critical insights:

• Discrepancy between Answer and Reasoning: The correctness of intermediate CoTs lags significantly behind the accuracy of final answers. High-performing models often produce correct answers despite flawed intermediate reasoning, suggesting the CoTs do not always faithfully explain the decision process.
• Impact of Hallucination Types:
  • Perception Hallucinations: These are the most frequent type of hallucination across all models. However, they show a weaker correlation with incorrect final answers; models often self-correct or bypass perceptual errors to reach the right conclusion.
  • Reasoning Hallucinations: These are less frequent but far more detrimental. There is a strong association between reasoning hallucinations (and mixed reasoning-perception hallucinations) and incorrect final outcomes.
• Spatial Dominance: Spatial-related hallucinations dominate both the perception and reasoning subcategories, highlighting a specific weakness in current models' spatial understanding.
• Thinking Length vs. Quality: Analysis reveals a trade-off where longer reasoning chains often lead to lower precision (due to redundancy) and a positive correlation with reasoning hallucinations. Smaller models sometimes exhibit more stable reasoning with shorter chains, while larger models are more prone to "overthinking" and drifting.

Significance and Claims

The paper claims that MM-THEBENCH provides a necessary systematic method to monitor the internal thinking processes of reasoning MLLMs. By moving beyond final answer accuracy, the benchmark reveals that current state-of-the-art models frequently rely on flawed reasoning paths to reach correct conclusions, or fail due to specific types of reasoning hallucinations that are not captured by traditional benchmarks.

The authors assert that their findings call for a shift in future research toward detailed monitoring of intermediate CoTs. The proposed taxonomy and evaluation framework are intended to guide the development of more reliable reasoning MLLMs by identifying specific cognitive dimensions where hallucinations occur, thereby facilitating targeted mitigation strategies. The work emphasizes that for reasoning MLLMs to be truly trustworthy, their intermediate "thoughts" must be as robust as their final outputs.