Technical Summary: MKG-RAG-Bench

Problem Statement

Retrieval-augmented generation (RAG) has improved the factual grounding of large language models (LLMs), yet existing approaches predominantly rely on unstructured textual corpora. While knowledge graph (KG)-based RAG (KG-RAG) addresses issues of noise and fragmentation by utilizing structured data, current methods are largely confined to textual knowledge graphs. This limitation restricts applicability in real-world scenarios where critical information is conveyed through images, charts, and tables.

The specific challenge addressed in this work is Multimodal KG-RAG (MKG-RAG), which integrates structured knowledge across multiple modalities. However, the field lacks a dedicated benchmark to evaluate this setting. Existing benchmarks for multimodal RAG (M-RAG) retrieve from unstructured corpora, while KG-RAG benchmarks focus on textual graphs. Neither captures the unique challenges of MKG-RAG, specifically:

1. Heterogeneous Retrieval: Queries and target triplets may involve different modality combinations (e.g., text-only queries requiring visual evidence, or multimodal queries requiring textual relations).
2. Cross-Modal Alignment: Relevance often requires reasoning across modalities and graph topology rather than simple surface-level matching.
3. Benchmark Utility: Existing multimodal knowledge graphs often contain low-utility or generic triplets that lead to ambiguous queries, obscuring the true contribution of retrieval mechanisms.

Preliminary analysis in the paper demonstrates that naively combining existing multimodal knowledge graphs (e.g., MedMKG) with downstream tasks results in performance degradation compared to RAG-free baselines, primarily due to the retrieval of task-irrelevant knowledge and the absence of task-critical knowledge.

Methodology

The authors propose MKG-RAG-Bench, a cross-domain benchmark designed to explicitly evaluate retrieval and generation in MKG-RAG. The benchmark is constructed using a principled, LLM-based curation pipeline involving three stages:

1. LLM-based Utility Filtering: An LLM assesses triplets from source knowledge graphs (MarKG for general domains and MedMKG for medical domains) to filter out low-utility data. Triplets that yield generic questions or do not require external evidence are removed to ensure queries are well-grounded and retrieval is necessary.
2. Controlled Completeness Simulation: To create queries with exact supervision, the pipeline simulates incompleteness by masking components of retained triplets.
   • Relation Masking: The relation is hidden, requiring inference from head and tail entities.
   • Tail Masking: The tail entity is hidden, requiring inference from the head and relation.
   • The head entity is preserved as a semantic anchor to facilitate multimodal grounding.
3. Hybrid Question Synthesis: An LLM (GPT-5) synthesizes natural language questions from the masked triplets in two modes:
   • Text-only: For text-only knowledge, generating questions that explicitly target the missing component.
   • Image-grounded: For triplets with visual entities, the image is provided to the LLM, and the entity is referenced as [Image] to enforce visual grounding and prevent trivial lexical matching.

The resulting dataset includes aligned query-triplet pairs for retrieval evaluation and question-answer pairs for generation evaluation. The benchmark covers five specific evaluation settings (S1–S5) varying by query modality (text-only vs. multimodal) and candidate triplet scope (text-only vs. multimodal vs. all).

The evaluation framework tests four categories of retrievers in a training-free setting:

• Text-only retrievers (treating triplets as plain text).
• Fusion-based multimodal retrievers (shared representation space).
• Captioning-based retrievers (converting images to text).
• Reranking-based retrievers (multi-stage filtering).

Key Contributions

1. Identification of a Research Gap: The work highlights that retrieval in MKG-RAG is underexplored and insufficiently evaluated in current multimodal RAG research.
2. First Cross-Domain MKG-RAG Benchmark: The introduction of MKG-RAG-Bench, comprising two multimodal knowledge graphs and corresponding QA datasets, enabling systematic evaluation of both retrieval quality and downstream generation.
3. Principled Curation Pipeline: A novel pipeline using LLMs to address challenges in heterogeneous retrieval, multimodal alignment, and benchmark utility by filtering low-utility knowledge and generating structurally grounded queries.
4. Unified Evaluation Framework: Comprehensive experiments across diverse multimodal settings demonstrating the critical role of effective retrieval in end-to-end MKG-RAG performance.

Results

Extensive experiments were conducted on the MKG-RAG-Bench-G (general) and MKG-RAG-Bench-M (medical) datasets using representative retriever architectures. Key findings include:

• Retrieval Difficulty: Effective multimodal retrieval remains a significant challenge. In the medical domain (MedMKG), multimodal retrievers (Fusion, Reranking) significantly outperformed text-only and captioning-based retrievers in multimodal settings, but performance varied drastically based on modality alignment.
• Modality Impact:
  • In the general domain, Fusion-based retrievers achieved the highest NDCG@K and Recall@K (e.g., ~62% Recall@100 for multimodal queries with multimodal triplets), outperforming text-only and captioning approaches.
  • In the medical domain, Reranking-based retrievers showed superior performance in complex multimodal settings (e.g., ~51% Recall@100 for multimodal queries with all triplets), suggesting that multi-stage pipelines are effective for handling noisy or complex medical data.
• Retrieval Determines Generation: The paper establishes that retrieval quality strongly determines generation outcomes. Poor retrieval leads to the injection of noise or the absence of critical facts, degrading the final answer quality.
• Baseline Comparison: In preliminary studies using existing graphs without curation, RAG-free models often outperformed retrieval-augmented models, underscoring the necessity of the curated, high-utility dataset provided by MKG-RAG-Bench.

Significance

The paper claims that MKG-RAG-Bench provides a principled foundation for diagnosing current limitations in multimodal knowledge graph systems. By isolating retrieval as a first-class evaluation target, the benchmark allows researchers to:

• Distinguish improvements arising from effective retrieval versus those from generative priors.
• Systematically cover diverse modality configurations to reflect realistic usage.
• Advance the development of more powerful retrievers specifically designed for structured, multimodal knowledge.

The work asserts that without such a benchmark, the true capabilities and limitations of MKG-RAG systems remain obscured by misalignment between knowledge sources, retrieval objectives, and evaluation metrics.