💻 computer science

mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQA

This paper proposes mKG-RAG, a novel retrieval-augmented generation framework that leverages multimodal knowledge graphs and a dual-stage retrieval strategy to overcome the limitations of unstructured document-based methods, achieving state-of-the-art performance in knowledge-intensive Visual Question Answering.

Original authors: Xu Yuan, Liangbo Ning, Qingqing Ye, Wenqi Fan, Qing Li

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xu Yuan, Liangbo Ning, Qingqing Ye, Wenqi Fan, Qing Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart but Forgetful" Robot

Imagine you have a super-smart robot (called a Multimodal Large Language Model, or MLLM) that is great at looking at pictures and answering questions. It knows a lot about the world. However, it has a major flaw: it has a bad memory for specific facts.

If you ask it, "Who designed this specific museum?" or "When was this stadium last renovated?", the robot might:

Hallucinate: Make up a plausible-sounding but completely wrong answer.
Refuse: Say, "I don't know," even though the answer exists somewhere.

This happens because the robot only "remembers" what it was trained on. It doesn't have access to a library of up-to-date, specific facts about every building, person, or event in the world.

The Old Solution: The "Noisy Library"

To fix this, researchers tried giving the robot a "Retrieval-Augmented Generation" (RAG) system. Think of this as giving the robot a library to look up answers before it speaks.

However, the old way of doing this was like handing the robot a stack of messy, unorganized newspapers.

The robot has to read through thousands of words to find the one sentence that matters.
It often gets distracted by irrelevant noise (ads, unrelated stories).
It misses the connections between facts. For example, it might see "Stadium" and "Renovation" in two different paragraphs but fail to realize they are part of the same story.

The New Solution: mKG-RAG (The "Smart Map")

The authors propose a new system called mKG-RAG. Instead of giving the robot a messy stack of newspapers, they give it a structured, visual map (a Multimodal Knowledge Graph).

Here is how it works, step-by-step:

1. Turning Chaos into a Map (Graph Construction)

Imagine taking a Wikipedia page that has text and photos.

Old Way: Just read the text.
mKG-RAG Way: The system uses a smart AI to read the text and look at the photos simultaneously. It extracts the "skeleton" of the information.
- It identifies Entities (e.g., "The Stadium," "The Architect").
- It identifies Relations (e.g., "The Architect designed the Stadium").
- Crucially, it links the text description of the stadium to the actual photo of the stadium.
- Result: Instead of a wall of text, you get a clean, organized map where every fact is connected to the picture that proves it.

2. The Two-Step Search (Dual-Stage Retrieval)

When you ask a question, the system doesn't just dump the whole map in front of the robot. It uses a smart, two-step search strategy:

Step 1: The Broad Net (Document Retrieval)
First, the system quickly scans the entire library to find the few documents that are likely to contain the answer. It's like a librarian saying, "Okay, the answer is probably in the 'Sports' section, not the 'Cooking' section."
Step 2: The Precision Net (Graph Retrieval)
Once the relevant documents are found, the system doesn't just read them linearly. It zooms in on the Map created in Step 1. It looks for specific nodes (facts) and edges (connections) that match your question.
- Analogy: Instead of reading the whole book, it highlights the exact paragraph and the specific photo that answers your question, ignoring the rest.

3. The "Question-Aware" Detective (QM-Retriever)

The paper introduces a special tool called the QM-Retriever.

The Problem: Standard search engines are bad at matching a question (e.g., "Who built this?") with a statement (e.g., "John built this."). They often look for exact word matches.
The Fix: The QM-Retriever is a detective trained to understand the intent of the question. It learns to translate your question into a "declarative" format (a statement) so it can find the perfect match in the knowledge graph, even if the words are slightly different. It looks at both the text and the image to find the right evidence.

Why This Matters (The Results)

The authors tested this system on two difficult quizzes (E-VQA and InfoSeek) that require deep, specific knowledge.

The Result: mKG-RAG significantly outperformed all previous methods.
The Analogy: If the old methods were like a student guessing answers from a messy textbook, mKG-RAG is like a student with a perfectly organized, cross-referenced encyclopedia and a highlighter that knows exactly what to read.

Summary

mKG-RAG is a new way to help AI answer hard questions about the real world. Instead of drowning the AI in messy text, it:

Builds a structured map linking text and images.
Searches efficiently by narrowing down the library first, then finding exact connections on the map.
Understands the question better than standard search tools.

This allows the AI to stop guessing and start providing accurate, fact-based answers backed by visual evidence.

1. Problem Statement

Knowledge-intensive Visual Question Answering (VQA) requires models to answer questions that rely on external, encyclopedic, or long-tail facts not present in the training data or the immediate visual input. While Multimodal Large Language Models (MLLMs) excel at commonsense reasoning, they struggle with specific factual queries, often hallucinating or refusing to answer.

Current Retrieval-Augmented Generation (RAG) approaches attempt to solve this by retrieving external documents. However, existing methods suffer from three critical limitations:

Unstructured Noise: They rely on unstructured text chunks or paragraphs, which introduce irrelevant context and "noise," making it difficult for MLLMs to isolate key facts.
Lack of Structural Relations: They overlook the logical connections between knowledge elements, limiting the model's ability to perform multi-hop reasoning.
Modality Gap: Most RAG systems use unimodal retrievers (text-only or vision-only) or fail to effectively align visual and textual evidence, leading to suboptimal retrieval in multimodal scenarios.

2. Methodology: mKG-RAG

The authors propose mKG-RAG, a framework that integrates Multimodal Knowledge Graphs (KGs) into the RAG pipeline. The framework consists of two main stages:

A. Multimodal Knowledge Graph Construction

Instead of using pre-existing KGs (which lack specific/long-tail knowledge), mKG-RAG constructs dynamic, high-quality KGs from unstructured multimodal documents (e.g., Wikipedia pages with text and images).

Textual Graph Extraction: Uses an MLLM to extract entities and relations from text segments, forming a textual subgraph ( $G_t$ ).
Visual Graph Extraction: Uses Scene Graph Generation (SGG) techniques to detect objects and their spatial/semantic relations within images, forming a visual subgraph ( $G_v$ ).
Multimodal Graph Aggregation: A core innovation where an MLLM acts as a Vision-Text Matcher. It aligns entities and relations from $G_t$ $G_{t}$ and $G_v$ $G_{v}$ based on semantic consistency.
- Mechanism: The MLLM receives the image, the textual graph (as natural language), and the visual graph (as object IDs, categories, and bounding boxes). It outputs matched pairs, effectively attaching visual attributes (bounding boxes, object categories) to textual entities.
- Result: A unified multimodal KG where nodes and edges possess both textual descriptions and visual grounding.

B. Dual-Stage Multimodal Retrieval

To handle the computational cost of searching large graphs, mKG-RAG employs a two-stage retrieval strategy:

Coarse-grained Document Retrieval:
- Uses a Query-aware Multimodal Retriever (QM-Retriever) to perform vector search over the entire document corpus.
- The QM-Retriever is trained to align multimodal queries (Image + Question) with evidence (Image + Text). It includes a "Question Converter" that transforms interrogative questions into declarative statements in the latent space to better match evidence text.
- This stage narrows the search space to the top- $K$ candidate documents.
Multi-granularity Graph Retrieval:
- Constructs a query-specific subgraph by merging the offline KGs of the candidate documents.
- Entity/Relation Retrieval: Retrieves specific nodes and edges based on embedding similarity to the query.
- Subgraph Expansion: Performs $l$ -hop neighbor traversal (Breadth-First Search) on the retrieved subgraph to capture contextual connections, but only includes neighbors that exceed a similarity threshold to prevent noise.

C. Answer Generation

The final context fed to the MLLM includes the original image, the question, the retrieved structured graph elements (providing a knowledge outline), and the associated textual chunks (providing details).

3. Key Contributions

Multimodal KG-Augmented RAG: The first framework to specifically integrate constructed multimodal KGs into RAG for knowledge-intensive VQA, capturing structural relations and cross-modal cues.
Novel Construction Pipeline: A method to distill unstructured multimodal documents into structured KGs using MLLM-driven extraction and vision-text matching, ensuring semantic consistency across modalities.
Dual-Stage Retrieval & QM-Retriever: A two-stage retrieval paradigm (Document $\to$ Graph) and a specialized Query-aware Multimodal Retriever trained with contrastive learning and question reformulation objectives to bridge the modality gap.
State-of-the-Art Performance: The approach significantly outperforms existing baselines on standard benchmarks.

4. Experimental Results

The method was evaluated on two benchmarks: E-VQA (5.8K samples) and InfoSeek (73K validation samples).

Performance:
- E-VQA: Achieved 36.3% accuracy (All set), surpassing the previous best (ReflectiVA*) by a significant margin.
- InfoSeek: Achieved 40.5% accuracy (All set), setting a new state-of-the-art.
- Improvement: Compared to zero-shot MLLMs, mKG-RAG improved accuracy by over 20% on E-VQA and 31% on InfoSeek.
Retrieval Efficiency: The QM-Retriever achieved average Recall improvements of 9.9% (E-VQA) and 7.0% (InfoSeek) over the best unimodal/cross-modal baselines (like CLIP).
Ablation Studies:
- Removing the QM-Retriever (using CLIP instead) dropped accuracy by ~4-5%.
- Replacing graph retrieval with chunk-based retrieval dropped accuracy by ~8%.
- Subgraph expansion contributed an additional ~1.3% gain.
Generalization: The framework showed consistent performance gains (average +9.4%) across various MLLM architectures (Phi3V, LLaMA-3.2, Qwen2.5-VL, etc.), regardless of parameter size.
Efficiency: Despite the complex pipeline, the heavy KG construction is offline. Online inference adds only a 1.14x computational cost compared to zero-shot generation while delivering a 1.37x accuracy improvement over naive RAG.

5. Significance

This paper addresses a critical bottleneck in multimodal AI: the inability of current models to reason over structured, external knowledge when visual and textual cues are intertwined. By shifting from unstructured text retrieval to structured multimodal graph retrieval, mKG-RAG:

Reduces Hallucination: Provides precise, structured evidence that guides the MLLM.
Enhances Reasoning: The graph structure explicitly models relationships, aiding in multi-hop reasoning (e.g., connecting an object in an image to a specific fact in a document).
Bridges Modalities: The QM-Retriever and vision-text matching ensure that visual and textual evidence are retrieved and aligned coherently, rather than treating them as separate silos.

The work demonstrates that integrating structured knowledge representations (KGs) with generative models is a superior paradigm for knowledge-intensive tasks compared to traditional RAG, paving the way for more reliable and factual multimodal AI systems.