Technical Summary: Efficient Multimodal Planning Agent for Visual Question-Answering

Problem Statement

Visual Question-Answering (VQA) is a complex multimodal task requiring the integration of visual and textual information. While Multimodal Retrieval-Augmented Generation (mRAG) has improved VQA performance by incorporating external evidence from images and text, current implementations suffer from significant inefficiencies. Existing systems typically rely on rigid, multi-stage pipelines (e.g., image grounding, image retrieval, query rewriting, text retrieval) that execute sequentially regardless of the specific query's needs. These static workflows often introduce redundant computations, excessive input lengths, and unnecessary tool calls, particularly for queries that could be resolved by the model's intrinsic knowledge or require only a subset of retrieval types. Furthermore, dependencies between stages (e.g., query rewriting relying on prior image retrieval) create bottlenecks that hinder scalability.

Methodology

The paper proposes a Multimodal Planning Agent designed to dynamically optimize mRAG pipelines by intelligently determining the necessity of each retrieval step for a given VQA query. The core approach involves training a Multi-Modal Large Language Model (MLLM) to act as an agent that decomposes the standard mRAG workflow into four distinct strategies based on the input query:

1. No mRAG (Path 1): The agent determines the model's internal knowledge is sufficient; no external retrieval is performed.
2. Image Retrieval Only (Path 2): The agent identifies that additional visual context ($k_i$) is needed but textual context is not.
3. Text Retrieval Only (Path 3): The agent determines that textual context ($k_t$) is required, often necessitating a "gold query" ($q_g$) rewrite to facilitate accurate text search.
4. Dual Retrieval (Path 4): The agent decides that both image and text contexts are necessary.

Data Construction and Training

To train this agent, the authors developed an automated data annotation pipeline:

• Query Decomposition: For each VQA pair $(q, a)$, the system generates an "image query" ($q_i$) to identify image entities and a "gold query" ($q_g$) that resolves ambiguous references (e.g., replacing "this" with the specific entity name) to enable effective text retrieval.
• Category Labeling: The system simulates four scenarios using a strong MLLM to determine the correct retrieval strategy ($c_1$ to $c_4$) by checking which combination of inputs yields the correct answer.
• Fine-Tuning: The MLLM is fine-tuned (using LoRA or Full Fine-Tuning) to predict the correct category $c$ given the input query $q$ and a prompt $T$. The objective is to minimize the cross-entropy loss for predicting the category.

Inference Process

During inference, the trained agent analyzes the input query and selects one of the four paths.

• If Text Retrieval is selected, the agent first rewrites the query into a gold query ($q_g$) using the image (and potentially image search results) to remove ambiguities before performing text search.
• If Image Retrieval is selected, the system retrieves visual context directly.
• If No Retrieval is selected, the model answers directly, bypassing all tool calls.
  This dynamic selection allows the system to skip redundant steps, thereby reducing latency and computational cost.

Key Contributions

1. Dynamic Pipeline Optimization: The paper introduces a planning agent that moves away from rigid, multi-stage mRAG pipelines. Instead, it dynamically decomposes the workflow, selecting only the necessary components (image search, text search, both, or neither) for each specific query.
2. Efficiency-Performance Trade-off: The method successfully balances efficiency and effectiveness. It significantly reduces search time and costly tool calls while maintaining or improving VQA task performance compared to static baselines.
3. Generalizability: The authors demonstrate that a fine-tuned 7B-scale agent can effectively guide diverse MLLMs (including closed-source models like GPT-4o and Qwen-VL-Max) without requiring further fine-tuning of the base task models.

Experimental Results

The method was evaluated across six diverse VQA datasets (Life VQA, Private VQA, Dyn-VQA, NoCaps, Visual7W, and a Mix dataset) and compared against strong baselines, including a Deep Research agent (WebWatcher) and a prompt-based method (OmniSearch).

• Performance: The proposed agent outperformed all baselines on average across the six datasets. Notably, it achieved performance comparable to or better than the "complete" mRAG setting (which always uses both image and text retrieval) while using significantly fewer resources.
• Efficiency Gains:
  • Compared to OmniSearch, the method reduced search time by over 60% on average.
  • Compared to WebWatcher, the method was 3x to 4.5x faster in terms of tool-call latency.
  • The agent successfully identified scenarios where retrieval was unnecessary (e.g., ~60% of queries in NoCaps and Visual7W), avoiding redundant operations.
• Cross-Model Applicability: When applied to other MLLMs (DeepSeek-VL, Qwen-VL-Max, GPT-4o), the agent consistently improved performance over both "No mRAG" and "Prompt-based" baselines, demonstrating its utility as a plug-and-play planning module.

Significance and Claims

The paper claims that its primary significance lies in addressing the inherent inefficiency of static pipeline architectures in mRAG contexts. By proving that adaptability does not require sacrificing task performance, the research offers a path toward more scalable and efficient multimodal agent systems. The authors emphasize that their approach mitigates the "over-retrieval" problem common in current RAG systems, where valuable computational resources are expended on steps that provide marginal benefit. The work suggests that intelligent, adaptive planning is a critical component for deploying multimodal agents in real-world scenarios where latency and cost are constraints.