Technical Summary: ExDR – Explanation-driven Dynamic Retrieval Enhancement for Multimodal Fake News Detection

1. Problem Statement

The rapid dissemination of multimodal fake news (combining text and images) poses a significant societal threat. Existing detection methods face three primary challenges:

1. Reliance on Internal Knowledge: Large Vision-Language Models (LVLMs) often suffer from hallucinations and struggle with newly emerging knowledge or long-tail instances due to their reliance on training-accumulated memory.
2. Inefficiency of Static Retrieval: Traditional Retrieval-Augmented Generation (RAG) often retrieves evidence for every input regardless of necessity, leading to computational overhead and noise, particularly for straightforward or commonsense-based samples where the model already possesses sufficient knowledge.
3. Coarse Retrieval Strategies: Naive dynamic RAG applications often rely on sample-level similarity, neglecting fine-grained internal cues critical for deception. Furthermore, retrieved evidence often lacks deception-specific signals, potentially introducing misleading information that impairs detection accuracy.

2. Methodology: The ExDR Framework

The authors propose ExDR, an Explanation-driven Dynamic Retrieval framework designed specifically for Multimodal Fake News Detection (MFND). ExDR operates on a "one-shot" scheme (a single trigger decision followed by at most one retrieval step) and consists of two core modules:

A. Retrieval Triggering: "When to Retrieve"

Instead of triggering retrieval based on token uncertainty (common in QA tasks), ExDR employs a Multi-Level Confidence Estimation strategy centered on the model's generated explanation. It assesses the need for retrieval using three complementary dimensions:

1. Label-level Uncertainty ($\tau_{label}$): Measures the log-probability difference between the "Real" and "Fake" labels.
2. Token-level Support ($\tau_{tok}$): Analyzes the top-$K$ candidate tokens at the classification position, checking for semantic alignment with curated "Real" and "Fake" lexicons.
3. Sentence-level Confidence ($\tau_{sent}$): Computes the perplexity of the model-generated explanation to gauge reasoning coherence.

To determine optimal thresholds for these indicators, the authors employ a Two-Stage Hybrid Search Strategy (Global Monte Carlo sampling followed by Local Grid Fine-tuning) on a validation set to maximize detection accuracy while minimizing unnecessary retrievals.

B. Evidence Retrieval: "How" and "What to Retrieve"

Once retrieval is triggered, ExDR utilizes a specialized strategy to fetch high-quality, deception-targeted evidence:

1. Entity-enriched Multimodal Hybrid Indexing ("How"):
   • The framework extracts key entities from the model's generated explanation using Named Entity Recognition (NER).
   • It constructs a hybrid feature vector by fusing visual features (from CLIP), textual features, and entity features (derived from the extracted entities).
   • This entity-enriched index ensures retrieval is semantically aligned with potential deceptive cues rather than just general image-text similarity.
2. Contrastive Evidence Retrieval ("What"):
   • The system infers a fine-grained deception label (e.g., Entity Inconsistency, Image Fabrication) for the input sample by comparing its explanation against a corpus of labeled examples.
   • Based on this inferred label, the system retrieves contrastive evidence:
     • A Positive Sample ($x^+$): A sample sharing the same fine-grained deception label.
     • A Negative Sample ($x^-$): A sample with a different binary label (Real vs. Fake).
   • These samples are prepended to the input as few-shot examples to guide the model's final decision.

3. Key Contributions

The paper outlines three primary contributions:

1. First Application of Dynamic RAG to MFND: Unlike prior supervised methods requiring task-specific training, ExDR enables efficient detection without fine-tuning the core LVLM for the detection task itself, leveraging a dynamic retrieval paradigm.
2. Explanation-Driven Framework: The authors innovate on both retrieval triggering (using multi-level confidence derived from explanations) and evidence retrieval (using entity-enriched indexing and contrastive sampling), achieving more efficient and deception-targeted retrieval.
3. New Evaluation Metrics: The paper introduces Retrieval Identification Rate (RI) to measure the accuracy of recognizing when retrieval is needed, and Retrieval Efficiency (RE) to quantify performance gains relative to retrieval costs.

4. Experimental Results

Experiments were conducted on two benchmark datasets: AMG (in-domain) and MR2 (cross-domain, multilingual). The framework was tested across various LVLMs (Qwen2.5-VL, InternVL2.5, LLaVA-1.6).

• Retrieval Triggering: ExDR's trigger module consistently outperformed baselines (FL-RAG, FLARE, DRAGIN) in both RI and RE metrics. Notably, on fine-tuned models like InternVL2.5-26b, other methods failed to trigger retrieval entirely, whereas ExDR successfully activated it, achieving superior performance.
• Evidence Retrieval: The entity-enriched hybrid indexing significantly outperformed traditional multimodal indexing (Text+Image). For instance, on the AMG dataset with vanilla LLaVA-1.6-7B, ExDR achieved 71.5% accuracy compared to 58.1% for Text+Image@full.
• Contrastive Retrieval: Using positive-negative contrastive evidence provided further gains. On the MR2 cross-domain dataset, this approach improved accuracy from 66.6% (single retrieval) to 72.0%.
• Overall Detection Performance: ExDR achieved state-of-the-art results, outperforming strong baselines like MGCA by 2.9% in Accuracy and 1.5% in F1 score on the AMG dataset.

5. Significance and Claims

The paper claims that ExDR addresses the critical limitations of current MFND methods by balancing efficiency and accuracy. Its significance lies in:

• Adaptability: It effectively handles evolving knowledge and long-tail cases by decoupling the retriever from the model, allowing for independent updates.
• Precision: By focusing on "when" to retrieve (avoiding noise on easy samples) and "what" to retrieve (focusing on deception-specific entities and contrastive examples), it enhances the model's ability to identify inconsistencies.
• Generalization: The framework demonstrates robust performance in cross-domain scenarios (AMG to MR2), suggesting its potential for building reliable, general-purpose misinformation detection systems without the need for extensive task-specific retraining.

The authors conclude that their approach highlights the effectiveness of integrating model-generated explanations into the retrieval process to create a more robust and efficient detection system.