💻 computer science

M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

This paper introduces M2-Verify, a large-scale, expert-validated multimodal benchmark spanning 16 scientific domains with over 469K instances, which reveals that current state-of-the-art models struggle with complex claim-evidence consistency and frequently hallucinate explanations despite high performance on simpler tasks.

Original authors: Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou, Wenpeng Yin, Dongwon Lee

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou, Wenpeng Yin, Dongwon Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a suspect's statement (the Claim) and a pile of evidence (a photo and a description of that photo). Your job is to figure out: Does the evidence actually prove what the suspect says, or are they lying?

For a long time, detectives only had text to work with. But in science, the "evidence" is often a complex X-ray, a weird chart, or a 3D model. If you only read the text description, you might miss the lie hidden in the picture.

This paper introduces M2-VERIFY, a massive new training ground for AI detectives to learn how to spot these lies in scientific papers.

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: The "Blindfolded" AI

Imagine an AI that is great at reading books but terrible at looking at pictures.

The Issue: Scientists write papers with text and images. Sometimes, the text says, "This drug shrinks tumors," but the image actually shows a tumor getting bigger.
The Gap: Existing tests for AI were like giving the detective a text-only clue. They didn't test if the AI could actually look at the evidence and say, "Wait a minute, that photo doesn't match the story."
The Risk: If AI tools start generating scientific papers, they might accidentally (or intentionally) create fake images that look real but contradict the text. We need a way to catch them.

2. The Solution: The "Giant Science Gym" (M2-VERIFY)

The authors built a massive gym for AI to train in. It's called M2-VERIFY.

The Size: It's huge. They collected 469,000 examples from real scientific papers (like medical journals and computer science archives).
The Variety: It covers 16 different fields, from Medicine (looking at X-rays) to Physics (looking at particle collision charts).
The Twist: They didn't just copy-paste papers. They created a "Trickster Mode."
- Normal Mode: The text and image match.
- Trickster Mode: They took a real image and subtly changed the text to lie about it. For example, they might take a photo of a healthy heart and change the caption to say, "This heart has a massive blockage."
- The Goal: The AI has to look at the photo, read the caption, and say, "Nope, that's a lie!"

3. The "Human Referees"

You can't just trust a computer to build a test for other computers. That's like asking a robot to grade a math test for robots.

The Audit: They hired 92 real human experts (doctors, scientists, engineers) to check the work.
The Process:
1. Blind Test: Experts looked at the images and claims without knowing the answer to see if humans could even agree on the truth. (Turns out, even humans find medical images tricky!)
2. Quality Control: They checked if the AI's explanations for why something was true or false made sense.
The Result: They created a "Gold Standard" dataset that is as close to perfect as possible.

4. The Results: The AI Got Stumped

They put 12 of the smartest AI models (like GPT-4o, Llama, and Qwen) into this gym to see how they performed.

The Good News: The AIs are pretty good at simple stuff. If the text says "The sky is blue" and the picture shows a blue sky, they get it right 85% of the time.
The Bad News: When things get complicated, the AIs fail hard.
- The "Anatomy Shift": If the AI has to tell the difference between two similar-looking body parts in an X-ray, its accuracy drops to 61%.
- The "Hallucination": Sometimes, the AI ignores the picture entirely. It sees a chart with 12 blocks and confidently says, "This chart has 6 blocks," because it's just guessing based on what it remembers from its training, not what it's actually seeing.
- The "Explanation" Problem: Even when the AI guesses the right answer, its explanation is often nonsense. It's like a student who gets the math right but writes a story about aliens in the solution.

5. Why This Matters

Think of M2-VERIFY as a stress test for the future of science.

As AI starts writing more scientific papers and creating more diagrams, we need to make sure it doesn't accidentally (or maliciously) spread misinformation.
This dataset shows us exactly where current AI is weak: It can read, but it can't really "see" and "reason" together yet.

In a nutshell: The authors built a giant, expert-verified obstacle course for AI to run through. They found that while the AI is fast, it often trips over the details, ignores the pictures, and makes up stories. This dataset is the map they need to fix those weaknesses and build trustworthy scientific AI.

1. Problem Statement

Scientific claim consistency checking requires verifying whether a stated claim is strictly supported by specific multimodal evidence (figures and text) within a closed document. While existing benchmarks exist for text-only verification (e.g., SciFact) or general multimodal tasks (e.g., FEVEROUS), they suffer from three critical limitations:

Scale and Diversity: Existing datasets are small and lack coverage across diverse scientific domains.
Visual Complexity: They often ignore the intricate, domain-specific nature of scientific figures (e.g., medical scans, complex architecture diagrams) which are essential for valid scientific communication.
Reasoning Gaps: Current models struggle with cross-modal reasoning, often relying on parametric memory or text-only cues rather than grounding claims in visual evidence. This is particularly risky as generative AI tools are increasingly used to create scientific figures, potentially introducing hallucinations.

2. Methodology: The M2-VERIFY Framework

The authors introduce M2-VERIFY, a large-scale dataset and evaluation framework constructed through a rigorous, four-stage pipeline validated by 92 domain experts.

A. Data Construction

Sources: Data is sourced from PubMed (for the medical subset, M2-VERIFY-Med) and arXiv (for the general science subset, M2-VERIFY-Gen), leveraging existing collections like MedICaT and SciMMIR.
Scale: The dataset contains 469,264 instances across 16 scientific domains.
Extraction:
- Claim Extraction: Uses state-of-the-art models (Google MedGemma for medical, Google Gemma for general science) to extract claims explicitly grounded in figures and captions.
- Strict Filtering: Instances where the claim can be verified using only the caption are discarded to enforce a strict dependency on visual evidence.
Perturbation (Negative Generation):
- General Science: Uses standard negation protocols.
- Medical Domain: Introduces a novel 7-category perturbation taxonomy curated by experts to create "refuted" claims that are semantically plausible but factually incorrect based on the image. Categories include:
  1. Status Swap (Normal $\leftrightarrow$ Abnormal)
  2. Numeric Change
  3. Attribute Flip
  4. Directional Flip
  5. Diagnosis Swap
  6. Location Shift (e.g., Thalamus $\leftrightarrow$ Basal Ganglia)
  7. Certainty Shift
Explanation Generation: Models generate natural language rationales (approx. 100 words) justifying the verification label, constrained to the provided evidence.

B. Quality Control & Expert Validation

A three-phase human evaluation ensures reliability:

Blind Inter-Annotator Agreement (IAA): 92 experts evaluated 200 instances blind to ground truth. Results showed substantial agreement (79.6% for Gen, 65.4% for Med), establishing a realistic human baseline.
Large-Scale Alignment: Experts verified 1,295 claims and 1,096 explanations. Agreement rates were high (84.9% for Med claims, 84.8% for Med explanation faithfulness).
Statistical Reliability: A gold-standard subset was used to audit the full dataset. High correlations (Pearson $r > 0.91$ ) between the gold set and full test set confirmed the dataset reliably reflects expert judgment.

3. Key Contributions

M2-VERIFY Dataset: The largest multimodal scientific claim verification benchmark to date (469K instances), covering 16 domains with fine-grained medical perturbations and explainable rationales.
Domain-Specific Perturbation Taxonomy: A rigorous framework for generating challenging negative samples, particularly in the medical domain, moving beyond simple negation to complex anatomical and diagnostic shifts.
Comprehensive Evaluation: A systematic evaluation of 12 leading Vision-Language Models (VLMs), including closed-source (GPT-4o/5), open-source (Qwen, LLaMA, InternVL), and domain-specific models.

4. Experimental Results

The authors evaluated 12 models on claim verification and explanation generation.

A. Claim Verification Performance

Overall Struggle: State-of-the-art models struggle to maintain robust consistency.
Domain Disparity:
- Medical: Closed-source models (GPT-4o-mini) lead with 73.0% Macro-F1.
- General Science: Open-source models (Qwen2.5-VL-7B) lead with 68.1% Macro-F1.
Complexity Analysis (Medical): Performance degrades significantly with reasoning complexity.
- Low-complexity (Status Swap): 85.8% Micro-F1.
- High-complexity (Anatomical Location Shift): Drops to 61.6% Micro-F1.
Domain Specificity: Models perform well in quantitative fields (e.g., Quantitative Finance: 72.9%) but struggle in abstract fields like Physics (58.1%) and Mathematics, where diagrammatic reasoning differs from standard plots.

B. Explanation Quality

Fluency vs. Logic: Open-source models generate fluent text (high ROUGE-L scores) but often lack logical faithfulness.
Hallucinations: Expert evaluation revealed that models frequently hallucinate facts (e.g., incorrect biological ranges) or insert linguistic artifacts (e.g., random Chinese characters in English text) even when the final label is correct.
Closed-Source Superiority: GPT-5-mini achieved the highest scores in correctness and completeness (9.43/10), significantly outperforming open-source models in semantic reasoning.

C. Modality Ablation

Necessity of Multimodality: Removing either the image or text causes significant performance drops (e.g., GPT-4o-mini drops from 73.0% to 56.4% without images).
Fusion Challenges: Some traditional fusion models (like fine-tuned MOCHEG) performed worse in multimodal settings than text-only, suggesting representation interference when integrating diverse scientific imagery.

D. Task Learnability

Supervised Fine-Tuning (SFT): Fine-tuning models (e.g., LLaMA-3.2, Qwen2.5) on M2-VERIFY resulted in substantial gains (up to +28.9% in Mathematics), proving that the dataset effectively teaches foundation models complex cross-modal scientific reasoning.

5. Significance and Future Impact

Benchmarking Standard: M2-VERIFY sets a new standard for evaluating scientific AI, highlighting that current models are not yet reliable for high-stakes scientific verification.
Safety & Integrity: By exposing hallucinations in visual reasoning, the benchmark addresses the critical risk of AI-generated scientific figures undermining scientific integrity.
Research Direction: The results indicate a need for models that can perform fine-grained spatial reasoning and integrate visual/textual streams without interference, rather than relying on parametric memory.
Open Science: The dataset and code are released to facilitate the development of trustworthy, explainable multimodal AI for scientific discovery.

In summary, M2-VERIFY demonstrates that while VLMs are improving, they currently lack the robust cross-modal reasoning required for rigorous scientific claim verification, particularly in complex medical and abstract domains. The dataset provides the necessary infrastructure to drive future advancements in trustworthy scientific AI.