💻 computer science

DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation

DualG-MRAG addresses the limitations of existing multimodal RAG systems in complex reasoning tasks by introducing a decoupled dual-tier framework that separates global structural reasoning (Macro-reasoning) from fine-grained evidence matching (Micro-matching) to reduce retrieval noise and enhance QA accuracy through graph-based message passing and dynamic programming decoding.

Original authors: Jiacheng Tao, Qingyun Sun, Haonan Yuan, Ziwei Zhang, Jianxin Li

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiacheng Tao, Qingyun Sun, Haonan Yuan, Ziwei Zhang, Jianxin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, multi-layered mystery. You have a giant library filled with books, photos, charts, and diagrams. A standard "smart assistant" (an AI) tries to help you, but it often gets confused. It might look at a picture of a cat and a sentence about a dog, mix them up, and confidently tell you a story that isn't true. This is because the AI struggles to connect the dots between different types of clues, especially when the answer requires jumping from one document to another, like a detective following a trail of breadcrumbs across a whole city. This field of study is called Multimodal Retrieval-Augmented Generation (MM-RAG). "Multimodal" just means the AI can see and read different things (like images and text). "Retrieval-Augmented Generation" means the AI doesn't just guess from its memory; it goes out, grabs the right facts from a library, and then writes its answer based on those facts. The big problem researchers are trying to solve is how to make the AI grab the right clues without getting overwhelmed by noise or missing the hidden connections between them.

Enter DualG-MRAG, a new framework proposed by researchers at Beihang University that acts like a brilliant detective with a very specific strategy. Instead of trying to read every single page of every book and look at every pixel of every photo at once (which creates a chaotic mess), DualG-MRAG splits the job into two distinct teams: a "Macro" team and a "Micro" team.

Think of the Macro team as the city planners. They don't care about the tiny details of a single house; they look at the big map. They draw the roads, the neighborhoods, and the major connections between different parts of the city. In the AI's world, this is the Macro Graph. It connects big ideas and documents together, helping the AI understand the "big picture" story and how one document might lead to another. This prevents the AI from getting lost in the weeds.

Then there is the Micro team, which acts like the forensic experts. Once the Macro team points them to a specific neighborhood, the Micro team zooms in. They look at the fine details: the specific texture of a fabric in a photo, the exact number in a table, or a tiny label on a chart. This is the Micro Graph. Their job is to verify the local evidence without getting distracted by the rest of the city.

The magic of DualG-MRAG is that it keeps these two teams separate but working together. In the past, AI systems tried to mix everything into one giant, messy graph. This was like trying to find a specific grain of sand on a beach while also trying to map the entire coastline at the same time—it was too noisy and confusing. DualG-MRAG says, "No, let's map the coastline first (Macro), and then go look for the sand (Micro)."

To make this even smarter, the system uses a special "query-driven" engine. Imagine you ask, "Where is the red car parked?" instead of the AI blindly checking every car in the city, the engine only sends messages along the roads that lead to red cars. It uses a Graph Neural Network (GNN) to pass these messages dynamically, only following the paths that matter for your specific question.

Finally, once the clues are found, the system doesn't just dump a pile of documents in front of the AI. Instead, it builds a clear, step-by-step "reasoning path." It's like the AI is handed a treasure map that says, "Start at the library, go to the photo of the car, then look at the parking ticket in the next document." This explicit path helps the AI generate a much more accurate answer.

The researchers tested this new detective system on some very tough puzzles involving complex questions that required jumping between images, tables, and text. They found that DualG-MRAG was significantly better at finding the right clues and answering correctly than previous methods. For example, on a test called MMQA, it improved the accuracy of finding the right answer by a noticeable margin compared to the best existing systems. It also managed to do this without being too slow, keeping the response time under a second in many cases. The study suggests that by separating the "big picture" thinking from the "fine detail" checking, we can build AI that is both smarter and more reliable when dealing with the messy, mixed-up world of real-world information.

Technical Summary: DualG-MRAG

Problem Statement

Multimodal Retrieval-Augmented Generation (MM-RAG) has demonstrated potential in leveraging heterogeneous data sources (text, images, tables) for Large Language Models (LLMs). However, existing methods face significant limitations when handling complex, multi-hop reasoning tasks. Current approaches primarily rely on instance-level matching using static embeddings, which fail to capture explicit relationships across modalities and documents.

The paper identifies three core challenges in extending graph-enhanced methods to multimodal scenarios:

Granularity vs. Noise Trade-off: Naively integrating fine-grained visual features into a global graph leads to rapid expansion and retrieval noise, while coarse-grained representations discard critical local evidence.
Static vs. Dynamic Mismatch: Existing graph retrievers often rely on static topologies and query-agnostic propagation rules, failing to adapt to the specific intent of a user query in complex multimodal contexts.
Lack of Explicit Structural Fusion: Conventional MM-RAG treats retrieved documents as a flattened list, forcing the generative model to implicitly reconstruct cross-document relationships, which increases cognitive load and hinders multi-hop reasoning.

Methodology: DualG-MRAG

The authors propose DualG-MRAG, a dual-tier framework that decouples Macro-Reasoning (global topology) from Micro-Matching (local verification). The system operates through three query-driven phases:

1. Dual-Tier Multimodal Graph Construction

Instead of a single flattened graph, the framework constructs two distinct layers:

Macro Reasoning Graph ( $T_M$ ): A high-level structural backbone representing global entities and relations. Visual information is incorporated via concise factual captions generated by a frozen Vision-Language Model (VLM), merged with text, and processed via Open Information Extraction (OpenIE). Equivalence edges are added based on semantic entity resolution to enhance connectivity.
Micro Matching Graph ( $T_m$ ): Designed to capture fine-grained dependencies within specific images and tables. Micro-facts are defined as 4-tuples $(u, r, v, d)$ , where $u$ and $v$ are textual anchors, $r$ encodes rich visual details (spatial relationships, attributes), and $d$ is a pointer to the source document (raw image/table). This abstraction preserves visual fidelity without exploding the global graph size.

2. Query-Driven GNN Retriever

Retrieval is formulated as a dynamic message-passing process over the Macro Graph, utilizing a Graph Neural Network (GNN) based on the NBFNet architecture.

Cross-Tier Alignment & Activation: The system first performs subgraph matching on the Micro Graph to identify relevant micro-facts. These are aligned with global entities to initialize a query-specific binary mask.
Dynamic State Initialization: Unlike static GNNs, the hidden states are initialized dynamically. Query-relevant nodes receive the semantic embedding of the query, while others are zeroed out, effectively pruning the search space.
Relational Message Passing: The GNN performs $L$ layers of message passing where edge messages integrate source node states, relation projections, and target node states. This allows the model to learn distinct traversal logic at varying reasoning depths.
Relevance Scoring: Final node scores are projected back to the document space. Crucially, visual scores are modulated by the Micro-matching score to suppress unverified visual noise, ensuring that documents with structurally verified visual evidence are ranked higher.

3. Explicit Path Decoding

To guide the downstream MLLM, the framework extracts explicit reasoning paths directly from the GNN's forward pass using a layer-wise dynamic programming algorithm.

Local Flow Decomposition: The contribution of each edge during message passing is quantified as a Local Flow Probability.
Path Extraction: The algorithm identifies the most probable reasoning chains (paths) on the computational Directed Acyclic Graph (DAG) formed by the $L$ hops.
Serialization: These paths are serialized into a structured evidence graph. Abstract macro-entities are inversely mapped to their original multimodal sources (text, image regions, tables) and verbalized as step-by-step reasoning chains in the generation prompt.

Key Contributions

Dual-Tier Architecture: A novel framework that decouples macro-topological routing from micro-feature verification, mitigating visual retrieval noise while preserving connectivity for multi-hop reasoning.
Query-Driven GNN Mechanism: The first application of a query-guided message passing mechanism in MM-RAG, enabling dynamic alignment of heterogeneous evidence and explicit extraction of logical reasoning chains.
Explicit Path Injection: A mechanism that transforms isolated retrieved instances into transparent, verifiable reasoning paths, reducing the implicit reasoning burden on the generative model.

Experimental Results

The framework was evaluated on MMQA, WebQA, and ScienceQA benchmarks against state-of-the-art baselines, including vector-based RAG (e.g., VisRAG, VLM2Vec) and graph-enhanced RAG (e.g., HM-RAG, MMGraphRAG).

QA Performance: DualG-MRAG outperformed all baselines across all metrics. On MMQA with a Qwen3-VL-4B backbone, it achieved an Exact Match (EM) of 44.20%, a 7% absolute improvement over the strongest baseline. On ScienceQA, it achieved an average accuracy of 90.99%, surpassing human performance in specific sub-categories and outperforming GPT-4 with Chain-of-Thought.
Retrieval Effectiveness: The model demonstrated superior recall, achieving 61.9% R@5 on MMQA, significantly outperforming the graph-based baseline MMGraphRAG (42.1%).
Efficiency: Despite the complexity of graph reasoning, DualG-MRAG maintained a sub-second average query latency (~0.44s), offering a competitive trade-off between retrieval efficiency and reasoning depth compared to slower graph-based methods.
Ablation Studies: Removing the Macro Graph caused a drastic drop in recall (from 61.9% to 21.8% R@5 on MMQA), confirming the necessity of global topology. Removing the Micro Graph reduced performance on WebQA, highlighting its role in filtering local visual noise.

Significance

The paper claims that DualG-MRAG addresses the fundamental dilemma in multimodal retrieval: balancing the need for fine-grained visual evidence with the risk of retrieval noise. By decoupling these tasks, the framework enables deep, multi-hop reasoning without sacrificing retrieval precision. Furthermore, by explicitly extracting and injecting reasoning paths, the method reduces the cognitive load on downstream MLLMs, allowing them to generate more accurate answers from heterogeneous, complex evidence sources. The work establishes a new paradigm for structured, graph-enhanced multimodal reasoning that is both effective and computationally efficient.