🤖 AI

Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering

The paper introduces SKIP, a unified inference architecture that employs sparse cross-modal routing and an adaptive compute budget to significantly reduce computational costs and latency in knowledge-intensive multimodal question answering while maintaining or exceeding the accuracy of dense baselines.

Original authors: Noor Islam S. Mohammad, Uluğ Bayazıt

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Noor Islam S. Mohammad, Uluğ Bayazıt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery. You have a giant photo album (the image), a list of suspects and clues from a massive library (external knowledge), and a specific question you need answered. In the world of artificial intelligence, this is called "Knowledge-Intensive Multimodal Question Answering." It's a fancy way of saying: "Can a computer look at a picture, read a question, and then go dig through a huge library to find the right answer?"

Currently, the smartest computer programs trying to do this are like over-enthusiastic detectives who refuse to skip a single page. No matter how simple the question is, they read every single word in the library and stare at every single pixel in the photo. They do this even if the answer only depends on one tiny detail, like the color of a boat's flag. This approach works, but it's incredibly slow and burns through a massive amount of computer power, making it hard to run on regular devices like phones or laptops. The big question scientists are asking is: Can we teach these AI detectives to be smarter about what they look at, so they can solve mysteries faster without getting the wrong answer?

Enter SKIP, a new system created by researchers Noor Islam S. Mohammad and Uluğ Bayazıt that tries to solve this problem by teaching AI to be a bit more lazy—but in a very clever way. Instead of treating every question like a life-or-death emergency requiring a full-blown investigation, SKIP acts like a seasoned detective who knows when to take a shortcut.

Here is how SKIP works, using a few everyday metaphors:

1. The "Smart Filter" (Question-Guided Visual Saliency)
Imagine you are looking at a photo of a busy street scene to answer the question, "What brand of soda is in the vending machine?" A normal AI would stare at every person, every car, and every tree. SKIP, however, looks at the question first and says, "I only care about the vending machine." It instantly ignores 90% of the photo, focusing only on the tiny patch where the soda machine is. This is called "pruning." The researchers found that by cutting out the irrelevant parts of the image before even starting the search, they could save a huge amount of time without losing any accuracy.

2. The "Targeted Librarian" (Region-Conditional Sparse Retrieval)
Once the AI knows what part of the image matters, it needs to go to the library. Usually, an AI would shout its question to the whole library at once. SKIP is different. If the photo has a few distinct interesting things (like a logo on a shirt and a sign on a building), SKIP sends out separate, tiny queries for each one. It's like sending three different interns to find three different facts, rather than making one intern run around the whole building shouting a vague question. This ensures the AI finds the specific, tiny details that a "one-size-fits-all" search often misses.

3. The "Selective Mixer" (Bipartite Sparse Cross-Attention)
Now the AI has the picture details and the library facts. It needs to mix them together to form an answer. Normally, the computer tries to connect every piece of picture data with every piece of library data. That's like trying to shake hands with everyone in a stadium at once. SKIP is smarter; it only shakes hands with the pairs that actually make sense. If a piece of the photo is about a "dog" and a piece of the library text is about "baking," SKIP ignores that connection entirely. This saves a massive amount of energy.

4. The "Difficulty Judge" (Adaptive Budget Controller)
SKIP has a little manager that looks at the question and asks, "Is this hard or easy?" If the question is simple, like "How many wheels does the car have?", the manager says, "No need to go to the library!" and the AI answers immediately. If the question is complex, like "Who invented the engine on this specific car model?", the manager says, "Okay, use the full team and check the whole library." This means the computer doesn't waste energy on easy questions.

5. The "Speed-Runner" (Speculative Knowledge Verification)
Finally, SKIP has a tiny, fast assistant that tries to guess the answer instantly. If the assistant is very confident, SKIP just accepts the guess and skips the long, slow process entirely. If the assistant is unsure, then the full, slow team steps in. This is like a student quickly checking their math homework; if they are sure they got it right, they don't need to re-calculate everything from scratch.

What Did They Find?
The researchers tested SKIP on five different difficult quizzes involving pictures and knowledge. They found that SKIP could answer questions just as accurately as the giant, slow systems, but it used 3.4 to 6.8 times less computer power (measured in FLOPs) and was 2.7 times faster.

In fact, on the hardest tests where the answer depended on finding tiny details in a picture, SKIP actually got more questions right than the big systems. This suggests that by ignoring the noise and focusing only on what matters, the AI actually performs better. The paper also includes a mathematical proof suggesting that the amount of information needed to answer a question grows much slower than the size of the picture, which explains why cutting out 90% of the image works so well.

What It Doesn't Do
It's important to note that SKIP isn't magic. The researchers admit it still struggles with questions that require very complex, multi-step reasoning (like connecting three different facts together) or questions about extremely rare things that don't appear often in the library. Also, if a question doesn't need any library research at all, SKIP doesn't get as much of a speed boost, because the main savings come from skipping the library search and the heavy mixing of data.

The Bottom Line
SKIP shows us that we don't need to brute-force our way through every problem. By teaching AI to be selective—ignoring the boring parts of the picture, asking specific questions to the library, and knowing when to take a shortcut—we can build systems that are faster, cheaper to run, and just as smart as the giants of today. It's a reminder that sometimes, knowing what not to look at is the most important skill of all.

Technical Summary: Salient Knowledge Pathways (SKIP)

Problem Statement
Knowledge-intensive Multimodal Question Answering (KI-MMQA) requires models to answer visual questions that cannot be resolved by image content alone, necessitating the retrieval of external world knowledge. Current state-of-the-art systems face a "dense pricing" problem: they uniformly incur the full computational costs of three expensive primitives for every query, regardless of the query's actual complexity or information needs. These costs include:

Long Visual Token Sequences: Modern vision encoders emit hundreds of patch tokens (e.g., 576), most of which are irrelevant to a specific question.
Dense Retrieval: Searching large external corpora (millions of passages) using global image-question queries.
Full Cross-Modal Fusion: Attending over the union of all visual tokens, question tokens, and all retrieved knowledge chunks, resulting in quadratic attention costs.

This uniform cost structure makes KI-MMQA systems prohibitively expensive for on-device deployment and interactive applications, scaling as $O(V \cdot T \cdot K \cdot d)$ , where $V$ is visual tokens, $T$ is question length, $K$ is retrieved chunks, and $d$ is model width. The authors observe that the informational requirements of queries are highly uneven; some require precise visual localization and minimal retrieval, while others require extensive retrieval with minimal visual grounding.

Methodology: SKIP Architecture
The authors introduce SKIP (Salient Knowledge-Injected Pathways), a unified inference architecture that routes computation along sparse, question-conditional pathways. Rather than treating sparsity as an isolated optimization, SKIP applies it compositionally across the visual encoding, retrieval, and fusion stages. The system comprises five learned components:

Question-Guided Visual Saliency (QVS): A cross-attention module that scores visual tokens based on their relevance to the question before retrieval occurs. By pruning irrelevant tokens early, QVS ensures that subsequent retrieval queries are derived only from question-relevant regions, preventing background noise from diluting the retrieval signal.
Region-Conditional Sparse Retrieval (RCSR): Instead of a single global query, RCSR clusters retained visual tokens into salient regions and issues separate retrieval queries for each region. This approach surfaces knowledge tied to small, discriminative image areas (e.g., logos, text, fine-grained entities) that global retrieval often misses.
Bipartite Sparse Cross-Attention (BSCA): An operator that restricts attention to (visual token, knowledge chunk) pairs exceeding a learned compatibility threshold. This reduces the dominant fusion FLOP term by an order of magnitude by skipping structurally irrelevant cross-modal edges.
Difficulty-Adaptive Budget Controller (DBC): A lightweight MLP that predicts per-query compute budgets (number of visual tokens $V'$ and retrieved chunks $K'$ ) based on global image-question features. It concentrates capacity on difficult queries and reduces it for easy ones without manual difficulty labels.
Speculative Knowledge Verification (SKV): A small "drafter" model that speculatively answers easy queries (e.g., counting, spatial questions) to determine if full retrieval is necessary. If the drafter is confident, the pipeline short-circuits; otherwise, the full pipeline runs. This routes 34–41% of queries through a fast path.

Theoretical Contribution
The paper derives an information-bottleneck bound (Theorem 4.3) under piecewise-Lipschitz mutual information assumptions. It proves that the optimal visual sparsity rate scales as $O(1/\sqrt{V})$ (specifically $V' \ge C \cdot \sqrt{V} \cdot \log(1/\varepsilon)$ ) to retain accuracy within $\varepsilon$ of the dense baseline. This provides a theoretical justification for the empirical observation that a small fraction of visual tokens suffices for high-accuracy KI-MMQA.

Experimental Results
SKIP was evaluated on five KI-MMQA benchmarks: OK-VQA, A-OKVQA, InfoSeek, Encyclopedic-VQA, and ViQuAE.

Accuracy: SKIP matches or exceeds the accuracy of strong dense baselines (e.g., RA-CM3) across all datasets. Notably, it improves over the strongest baseline by +2.7 to +4.3 accuracy points, with the largest gains on entity-centric and fine-grained tasks where knowledge access is critical.
Efficiency: SKIP achieves these results using 3.4–6.8× fewer FLOPs and 2.7× less end-to-end latency compared to dense baselines.
Compute-Accuracy Frontier: On the OK-VQA benchmark, SKIP Pareto-dominates all baselines, sustaining >59% accuracy at 5× less compute than dense systems. At matched FLOP budgets, SKIP significantly outperforms retrieval-free models (e.g., +17.3 points over BLIP-2 on OK-VQA).
Ablations: Removing any single component degrades performance. QVS is identified as the most critical component for accuracy; replacing it with random pruning drops accuracy by 9.1 points. BSCA is the primary driver of efficiency, reducing FLOPs by an order of magnitude with negligible accuracy loss.

Significance and Claims
The authors claim that SKIP is the first system to jointly optimize visual sparsity and retrieval sparsity under a unified, question-conditional routing policy. The paper argues that knowledge-intensive multimodal inference is inherently sparser than current systems assume. By treating sparsity as a first-class design principle that spans modalities and retrieval, SKIP unlocks substantial computational headroom. The authors emphasize that their approach does not merely trade accuracy for speed; rather, by reducing the cost of retrieval and fusion, the system can afford to retrieve from larger effective corpora and fuse richer contexts, thereby improving accuracy over dense baselines. The theoretical bound and empirical results align, suggesting that the low-dimensional structure of KI-MMQA queries can be effectively exploited for efficient inference.

Technical Summary: Salient Knowledge Pathways (SKIP)

More like this