Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering

This paper identifies and addresses the "visual shortcuts" plaguing existing Multimodal Knowledge-Based Visual Question Answering benchmarks by introducing the RETINA dataset, which forces models to reason about related entities, and proposing the MIMIR model that leverages multi-image retrieval to overcome these limitations.

The Gist

The Problem: The "Cheat Code" in AI

Imagine you are taking a history test. The teacher shows you a picture of a Beetle and asks, "What part of the world does this animal live in?"

In the old way of doing things (the "Existing Benchmarks"), the answer key was always sitting right next to the picture. The textbook page shown to the AI had a picture of the exact same beetle and the text "North America."

The AI learned a Visual Shortcut (or a "Cheat Code"). It didn't actually read the text or understand the biology. It just learned: "If I see a picture of a beetle, I look for a document with a picture of a beetle, and I grab the answer from there." It was like a student who memorized that "Question 5 always has a picture of a dog, so the answer is always 'Dog Park'."

The researchers found that if you took away the text and only gave the AI the picture, it could still get the answer right! This proves the AI wasn't "thinking"; it was just matching patterns.

The Solution: The "RETINA" Exam

To fix this, the authors built a new, tougher exam called RETINA.

The Analogy:
Imagine the teacher still shows you the picture of the Beetle. But this time, the answer isn't in a book about beetles. The answer is hidden in a book about Potatoes.

Why? Because the beetle eats potatoes. The book about potatoes mentions the beetle, but the book about the beetle doesn't mention the potato.

• The Old Way: Show a picture of a Beetle $\rightarrow$ Find a book with a Beetle picture. (Easy/Cheat).
• The RETINA Way: Show a picture of a Beetle $\rightarrow$ Find a book about Potatoes. (Hard/Real).

This forces the AI to stop looking for a visual match and start actually reasoning. It has to understand that "Beetle" $\rightarrow$ "Eats" $\rightarrow$ "Potato" $\rightarrow$ "Read about Potato."

They used a smart AI (an LLM) to automatically create 120,000 of these tricky questions, ensuring the picture in the question never matched the main picture in the answer book.

The New Tool: MIMIR (The "Multi-Image Detective")

The researchers realized that existing AI models were terrible at this new "RETINA" exam because they were trained to only look at one picture per document (the main subject).

If the AI is looking for a book about Potatoes, but the question shows a Beetle, a standard AI says, "No match! The pictures are different!" and gives up.

So, they built a new AI model called MIMIR (Multi-Image MultImodal Retriever).

The Analogy:
Think of a standard AI as a librarian who only looks at the cover photo of a book to decide if it's relevant.

• Librarian (Old AI): Sees a Beetle. Looks at the "Potato" book cover. Sees a Potato. Says, "Wrong book!"

Think of MIMIR as a librarian who opens the book and looks at every single photo inside before deciding.

• MIMIR (New AI): Sees a Beetle. Opens the "Potato" book. It sees the cover has a Potato, but it also flips through the pages and sees a picture of a Beetle eating the potato inside. It says, "Aha! This book has a picture of the beetle too! This is the right book!"

By attaching pictures of related things (like the beetle, the potato, the soil, the farmer) to the "Potato" document, MIMIR can find the connection even when the main cover photo doesn't match.

Why This Matters

1. Real-World Skills: In the real world, information is messy. If you ask a doctor about a rash, they might need to look at a book about the virus causing it, not just a book about the skin. The old AI was too lazy to do this; the new one is forced to be smarter.
2. Better Search Engines: This helps build search engines that don't just match keywords or images but actually understand the relationships between things.
3. Honest Evaluation: The paper proves that many previous AI tests were "broken" because they let the AI cheat. RETINA is a fair test that shows who is actually smart and who is just guessing.

Summary

• The Issue: Old AI models were cheating by matching pictures instead of reading.
• The Fix (RETINA): A new dataset where the picture in the question and the picture in the answer book are different, forcing the AI to think.
• The Hero (MIMIR): A new AI model that looks at all the pictures inside a document (not just the cover) to find the right answer, even when the visual clues are tricky.

The paper is essentially saying: "Stop letting our AI take shortcuts. Let's give it a real puzzle, and build a smarter detective to solve it."

Technical Summary

1. Problem Statement

The paper identifies a critical flaw in existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks (such as InfoSeek and EVQA): the prevalence of "visual shortcuts."

• The Shortcut Mechanism: In current benchmarks, the query image almost always depicts the primary subject entity of the target document (the Ground Truth or GT document). For example, if the question asks about a beetle, the query image is a picture of that specific beetle, and the answer is found in the document titled with that beetle's name.
• The Consequence: Models can exploit this correlation to retrieve the correct document using visual cues alone, without actually reading the text or performing complex multimodal reasoning. Preliminary experiments showed that models trained without text inputs could achieve comparable retrieval performance to those using full multimodal inputs, proving that existing benchmarks do not truly test knowledge integration.
• Real-World Mismatch: In real-world scenarios, a user might query an image of a related entity (e.g., a plant) to find information about a different entity (e.g., the beetle that eats the plant). Current benchmarks fail to capture this "non-main entity" retrieval challenge.

2. Methodology

The authors propose a two-pronged solution: a new benchmark (RETINA) to expose the flaw, and a new model architecture (MIMIR) to solve it.

A. RETINA Benchmark Construction

RETINA (Relational Entity Text-Image kNowledge Augmented) is a new benchmark designed to eliminate visual shortcuts.

• Pipeline: It uses an LLM-driven pipeline to automatically generate 120k training samples and a 2k human-curated test set.
• Process:
  1. Knowledge Graph Construction: For a target Wikipedia document (Main Entity), the LLM extracts a one-hop neighborhood graph of related entities (e.g., for "Lema daturaphila," it extracts "Potato" and "North America").
  2. Subgraph Selection: The system selects a Query Entity (a related entity, not the main entity) and a Qualifying Entity (to ensure the answer is unique).
  3. QA Generation: The LLM generates a question where the answer is the Main Entity, but the query image is the Query Entity.
  4. Paraphrasing: Questions are paraphrased to prevent retrieval via simple text matching (BM25).
• Result: The query image (e.g., a potato) and the GT document (e.g., about the beetle) are visually distinct, forcing the model to rely on textual knowledge and relational reasoning rather than visual similarity.

B. MIMIR Model (Multi-Image MultImodal Retriever)

Existing retrievers (like MuKA) augment a document with only a single image (the main entity). MIMIR extends this by enriching document embeddings with multiple images of related entities.

• Document Augmentation: For a given document, MIMIR identifies related entities mentioned in the text and retrieves their corresponding images from the Knowledge Base (KB).
• Architecture:
  • Multi-Image Encoding: Instead of a single image, the document encoder processes the main entity image plus $R$ related entity images.
  • Multimodal Feature Extraction: It uses a transformer-based cross-attention mechanism where patch-level visual features interact with textual features.
  • Entity Token Embedding (ETE): A novel learnable vector ($\theta_{ETE}$) is added to the text tokens corresponding to specific named entities. This helps the cross-attention mechanism distinguish which text tokens are semantically relevant to which specific image, improving the alignment of multimodal features.
• Scoring: Uses a late-interaction scoring mechanism (similar to ColBERT) where query tokens interact with all document tokens (text + visual features from all images) to compute relevance.

3. Key Contributions

1. Identification of Visual Shortcuts: The paper rigorously demonstrates that current MKB-VQA benchmarks are solvable via visual shortcuts, rendering them insufficient for evaluating true multimodal reasoning.
2. RETINA Benchmark: A large-scale, automatically generated benchmark (120k train / 2k test) that specifically targets scenarios where the query image is a related entity, not the main subject.
3. MIMIR Architecture: A novel retriever that augments document representations with multiple related entity images and introduces Entity Token Embeddings to better align text and diverse visual contexts.
4. Comprehensive Evaluation: The study shows that while state-of-the-art models fail on RETINA, MIMIR significantly outperforms them, proving the effectiveness of multi-image augmentation.

4. Experimental Results

The models were evaluated on InfoSeek, EVQA (existing benchmarks with shortcuts), and RETINA (no shortcuts).

• Performance on Existing Benchmarks: On InfoSeek and EVQA, standard multimodal models (MuKA, UniIR) perform well, often outperforming text-only models.
• Performance on RETINA:
  • Failure of Baselines: Existing models (MuKA, PreFLMR) suffer a drastic performance drop on RETINA (e.g., MuKA's Recall@5 drops from ~65% on EVQA to ~20% on RETINA). They frequently retrieve "distractor" documents whose main entity matches the query image visually, confirming their reliance on shortcuts.
  • MIMIR Success: MIMIR achieves nearly double the Recall@5 of the best baseline on RETINA (43.2% vs. 20.8% for Seen; 36.9% vs. 8.6% for Unseen).
• Ablation Studies:
  • Adding Multiple Images (MI) significantly boosts performance on RETINA.
  • Adding Multimodal Features (MMF) (cross-attention between text and image patches) further improves results.
  • Adding Entity Token Embeddings (ETE) provides the final significant gain, showing the importance of explicitly linking text tokens to specific images.
• Qualitative Analysis: MIMIR successfully retrieves documents where the query image (e.g., a specific tree) matches a related entity image within the GT document, whereas baselines fail and retrieve visually similar but semantically incorrect documents.

5. Significance

• Benchmark Integrity: The paper highlights a major gap in current AI evaluation, urging the community to move beyond "visual shortcut" datasets to ensure models are actually reasoning and not just pattern matching.
• Real-World Applicability: By simulating real-world queries where users ask about related entities (e.g., "What eats this plant?"), RETINA and MIMIR provide a more realistic framework for Visual Information Seeking.
• Architectural Innovation: The concept of augmenting document embeddings with multiple related images and using Entity Token Embeddings offers a new direction for multimodal retrieval systems, moving beyond the "one document, one image" paradigm.

In conclusion, the paper successfully breaks the visual shortcuts in MKB-VQA by introducing a rigorous benchmark (RETINA) and a robust solution (MIMIR), demonstrating that true multimodal reasoning requires models to integrate diverse visual and textual contexts rather than relying on superficial image-document correlations.