M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) excel at surface-level perception and commonsense reasoning but struggle with fine-grained entity understanding and complex multi-hop reasoning in real-world scenarios. Existing Visual Question Answering (VQA) benchmarks (e.g., OK-VQA, A-OKVQA) primarily focus on:

• Coarse-grained categories: Basic objects rather than specific named entities (e.g., "a dog" vs. "a specific breed of dog").
• Single-entity reasoning: Questions usually revolve around one primary entity.
• Simple inference: Most questions require only one or two reasoning steps.

These limitations fail to challenge MLLMs on tasks requiring the integration of information from multiple visual and textual sources, sequential logic chains, and the retrieval of specific external knowledge.

2. Methodology: The M3-VQA Benchmark

The authors introduce M3-VQA, a novel dataset and evaluation framework designed to test three core capabilities simultaneously: Multimodal understanding, Multi-Entity interaction, and Multi-Hop reasoning.

A. Dataset Construction

• Sources: Images are sourced from fine-grained datasets (e.g., CelebTo for celebrities, FGVD for vehicles, LogosInTheWild) and manually collected via web search to ensure diverse, multi-entity scenes.
• Entity Linking: Entities are mapped to Wikidata IDs and linked to corresponding Wikipedia pages, creating a traceable knowledge base.
• Question Generation:
  • Parallel Multi-Hop: Questions require independent reasoning about multiple entities (visual or textual) followed by synthesis (e.g., comparing the origins of two different animals in an image).
  • Sequential Multi-Hop: Questions use "bridging entities" where the answer to one sub-question becomes the input for the next (e.g., Identify a stadium $\rightarrow$ Determine the sport played there $\rightarrow$ Find the origin country of that sport).
• Scale & Complexity: The dataset contains 13,125 (Image, Question, Answer) triplets.
  • Entities: Average of 2.8 entities per question (ranging up to 4+).
  • Hops: Average of 3.1 reasoning hops (ranging up to 4+).
  • Knowledge Base: A curated multimodal knowledge base of ~2 million Wikipedia pages with gold evidence annotations (specific sentences/sections supporting the answer).

B. Evaluation Protocols

The authors evaluate 16 leading MLLMs (including GPT-4o, Qwen2.5-VL, InternVL2.5, LLaVA-OneVision) under three distinct settings:

1. Original (No Evidence): Model relies solely on internal knowledge and the image.
2. Oracle (Gold Evidence): Model is provided with the exact ground-truth evidence (sentence/section) required to answer.
3. KB (Retrieval-Augmented): Model must retrieve relevant evidence from the external knowledge base using two strategies:
   • Heuristic Retrieval: Single-shot text/image search using the full query.
   • Agentic Retrieval: A multi-agent framework (Planner, Executor, Solver) that decomposes complex questions, segments images, and iteratively retrieves information.

C. Metrics

• Intersection over Union (IoU): Used to handle questions with multiple valid answers or aliases, providing a more robust accuracy measure than exact match.
• Traceability: The ability to link the final answer back to specific evidence steps.

3. Key Contributions

1. Novel Benchmark: M3-VQA is the first VQA benchmark to simultaneously demand fine-grained multi-entity recognition, parallel and sequential multi-hop reasoning, and retrieval from a large-scale multimodal knowledge base.
2. Comprehensive Evaluation Framework: It introduces a rigorous testing protocol comparing models without knowledge, with perfect knowledge (Oracle), and with retrieval mechanisms, isolating where models fail (knowledge acquisition vs. reasoning).
3. Agentic Retrieval Framework: The paper proposes and validates an Agentic Retrieval approach that outperforms traditional heuristic methods by decomposing complex queries and performing iterative planning.
4. Open Resources: The dataset, code, and knowledge base are made publicly available to foster further research.

4. Experimental Results

The evaluation of 16 models reveals significant gaps in current MLLM capabilities:

• Poor Performance Without External Knowledge:
  • Even the best model (Qwen2.5-VL-72B) achieves only 32.6% accuracy in the "Original" setting.
  • Pure language models perform worse than vision-language models, confirming that image descriptions alone are insufficient for fine-grained entity details.
• Critical Role of Evidence:
  • Providing gold evidence (Oracle setting) boosts performance significantly (e.g., Qwen2.5-VL-72B reaches 58.4%).
  • However, even with perfect evidence, models do not reach 100%, indicating that reasoning capabilities (integrating multi-hop facts) remain a bottleneck, not just knowledge retrieval.
• Retrieval Strategies:
  • Agentic Retrieval significantly outperforms Heuristic Retrieval. For Qwen2.5-VL-72B, Agentic Retrieval achieved 38.9% vs. Heuristic's 36.6%.
  • Heuristic methods suffer from "overloaded queries" when dealing with multiple entities, while Agentic methods successfully decompose the problem.
• Complexity Impact:
  • Accuracy drops monotonically as the number of reasoning hops and entities increases.
  • Interestingly, in the "Original" setting, models sometimes perform better on higher-hop questions because the question text itself contains the necessary entity names (general knowledge), whereas fine-grained visual reasoning fails without evidence.

5. Significance and Future Directions

• New Standard for Reasoning: M3-VQA sets a higher bar for evaluating MLLMs, moving beyond simple perception to complex, structured reasoning required in real-world applications (e.g., medical diagnosis, legal document analysis, scientific research).
• Highlighting the "Reasoning Gap": The results demonstrate that current models are heavily reliant on external information and struggle to perform multi-step logical integration even when that information is provided.
• Advancing Retrieval-Augmented Generation (RAG): The success of Agentic Retrieval suggests that future MLLMs must incorporate planning and iterative reasoning rather than simple one-shot retrieval to handle complex multimodal tasks.
• Limitations: The dataset is currently English-only and relies on Wikipedia. Future work should expand to other languages and specialized domains (e.g., biomedical, legal).

In conclusion, M3-VQA exposes the current limitations of MLLMs in fine-grained understanding and complex reasoning, providing a critical pathway for developing more robust, knowledge-integrated, and reasoning-capable multimodal agents.