MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval

Technical Summary: MEVER (Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval)

1. Problem Statement

The paper addresses the limitations of current Claim Verification (CV) systems, which primarily focus on textual evidence and lack transparency. The authors identify three critical gaps in existing research:

• Uni-modality: Most models rely solely on text, failing to reason over visual evidence (e.g., charts, images) that is often essential for verifying modern multimedia claims.
• Lack of Explainability: Existing models often provide a verdict (Support/Refute/NEI) without a textual justification, making the reasoning process "black-box" and unconvincing.
• Domain Gap: Most multi-modal datasets are in the general domain, lacking the specialized scientific reasoning required for academic or technical claim verification.

2. Methodology

The authors propose MEVER, a unified framework that integrates three core tasks: Evidence Retrieval, Multi-modal Claim Verification, and Explanation Generation.

A. Multi-Modal Evidence Retrieval (Two-Layer Graph):
To retrieve relevant evidence, the model constructs a two-layer heterogeneous graph:

• Bottom Layer (Textual): Nodes represent evidence texts.
• Top Layer (Visual): Nodes represent images.
• Cross-layer Edges: Connect texts to their associated images.
• Reasoning Mechanism: The model employs a nested architecture using Image-to-Text and Text-to-Image reasoning. It uses Graph Neural Networks (GNN) to aggregate multiple images for a single text and propagates information through Transformer (for text) and ViT (for images) steps to create unified multi-modal embeddings.

B. Multi-Modal Claim Verification (Hierarchical Fusion):
Once evidence is retrieved, MEVER performs joint reasoning through two levels of fusion:

• Token-level Fusion: Uses Multi-Head Attention (MHA) to allow fine-grained interaction between the tokens of the claim and the tokens/patches of the evidence.
• Evidence-level Hierarchical Fusion: Aggregates multiple retrieved pieces of evidence using a GNN, ensuring the model considers the global context of all supporting/refuting information.

C. Explainable Generation (Fusion-in-Decoder & Consistency):

• Multi-modal Fusion-in-Decoder (FiD): To generate explanations, the model concatenates claim/evidence text with their corresponding image embeddings, allowing the decoder to "see" the evidence while writing.
• Consistency Regularizer: To ensure the generated text matches the predicted verdict, a regularizer is introduced that minimizes the KL-divergence between the verification label and a label predicted from the explanation's own logits.

3. Key Contributions

1. MEVER Model: A novel end-to-end architecture that solves the retrieval, verification, and explanation tasks simultaneously using multi-modal graph reasoning.
2. AIChartClaim Dataset: A new scientific dataset specifically for the AI domain. It contains 1,200 claims, 300 charts, and corresponding captions/explanations, designed to test scientific reasoning in charts (line, bar, scatter, etc.).
3. Advanced Fusion Techniques: Introduction of both token-level and evidence-level hierarchical fusion to capture complex cross-modal relationships.

4. Experimental Results

The model was evaluated on several datasets, including the new AIChartClaim, ChartCheck, Mocheg, and MR2.

• Retrieval Performance: MEVER outperformed text-only baselines (BM25, RAV, JustiLM) and existing multi-modal models, proving that the two-layer graph approach captures better cross-modal relevance.
• Verification Accuracy: MEVER achieved state-of-the-art (SOTA) Macro F1 scores across multiple datasets. For example, on AIChartClaim, it significantly outperformed multi-modal baselines like Mocheg and ESCNet.
• Explanation Quality: Using ROUGE-L, METEOR, and BLEU-2 metrics, MEVER showed superior performance in generating human-like, accurate explanations. It also scored highest in "VLM-as-a-Judge" evaluations for consistency, relevance, and correctness.

5. Significance

MEVER represents a significant step toward trustworthy AI. By moving beyond text-only verification and providing transparent, multi-modal justifications, the model is better suited for high-stakes environments like scientific research and news verification. The introduction of the AIChartClaim dataset also provides the research community with a much-needed benchmark for evaluating the scientific reasoning capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs).