💬 NLP

UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

UniHEAR is a unified, lightweight framework that overcomes the limitations of single-source retrieval and source-blind reranking in Knowledge-Based Visual Question Answering by employing a coarse retrieval descriptor, retrieval-guided attentive modality gating, and entropy-weighted source fusion to achieve state-of-the-art performance on E-VQA and InfoSeek benchmarks.

Original authors: Ganzhong Luo, Yang Ren, Hanyong Wang, Shuyu Zheng, Menglong Yang

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ganzhong Luo, Yang Ren, Hanyong Wang, Shuyu Zheng, Menglong Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky riddle, but the answer isn't hidden in the picture in front of you; it's hidden in a massive library of books, websites, and photos that you can't see. This is the world of Knowledge-Based Visual Question Answering (KB-VQA). It's a branch of artificial intelligence where a computer looks at an image, reads a question about it, and then has to go dig up the right facts from the outside world to give a smart answer. Think of it like a detective who can see the crime scene but needs to check the police database to know who the suspect really is.

For a long time, these AI detectives had a major problem: they were terrible at finding the right clues. They usually relied on just one way to search. Sometimes they only looked for pictures that looked like the one in the question (like finding a twin), and other times they only looked for text that sounded like the question (like finding a synonym). But what if the answer is in a picture that doesn't look exactly like the original, or in a text that doesn't use the exact same words? The AI would miss it. Furthermore, even when they found a list of possible clues, they were often "blind" to where those clues came from, treating a blurry guess the same as a solid hit. This paper introduces a new system called UniHEAR that fixes these mistakes by letting the AI search in two different ways at once and then using a clever trick to decide which clues are the best.

The Detective's New Toolkit: UniHEAR

Meet UniHEAR, a new, lightweight AI framework designed to be the ultimate detective for visual riddles. The authors, a team from Sichuan University, realized that existing AI systems were stuck in two big traps.

Trap #1: The One-Eyed Search
Imagine you are looking for a lost dog. If you only ask people who saw a dog that looks exactly like yours, you might miss the person who saw a dog that barks exactly like yours. Existing AI systems often did this. They would either search for visual matches (Image-to-Image) or text matches (Image-to-Text), but rarely both. This created a "Single-Source Retrieval Bottleneck." If the true answer was only in the text search, the visual search would fail, and vice versa. The result? The AI missed the ground-truth facts it needed to answer the question.

Trap #2: The Clueless Sorter
Once the AI found a list of potential clues, it had to sort them to find the best one. The old way was like a librarian who treats every book on the shelf exactly the same, regardless of whether it came from the "History" section or the "Fiction" section. The paper calls this "Retrieval-Source-Blind Reranking." The AI ignored the fact that some clues were strong visual matches while others were strong text matches, leading it to waste time on redundant information and rank the wrong answers at the top.

How UniHEAR Solves the Mystery

UniHEAR changes the game by acting like a detective who uses two different search engines simultaneously and then uses a smart filter to pick the winner.

Step 1: The Double Search
Instead of picking just one way to search, UniHEAR sends the question out to two different "libraries" at the same time:

The Visual Library: It looks for images that look like the one in the question.
The Text Library: It looks for text summaries that match the meaning of the question.
It combines the results from both into one giant "candidate pool." This ensures that if the answer is hidden in a picture or a paragraph, it gets found.

Step 2: The "ID Card" for Every Clue
Here is the clever part. For every single clue (or "entity") found in that giant pool, UniHEAR creates a special Coarse Retrieval Descriptor. Think of this as an ID card for the clue. This card doesn't just say "I found this"; it records how it was found. It notes:

How high up was it on the list?
How confident was the search engine?
How "surprised" was the search engine (entropy) to find it?
This ID card tells the system, "Hey, this clue came from the text search and was ranked #1, while that one came from the image search and was ranked #5."

Step 3: The Smart Filter (Retrieval-Guided Gating)
Now comes the sorting. Old systems treated all clues the same. UniHEAR uses a new module called Retrieval-Guided Attentive Modality Gating. Imagine a bouncer at a club who looks at the ID card of every person trying to get in.

If a clue came from the Visual Search, the bouncer knows the visual part is already "proven," so it doesn't need to focus as much on the visual features again. It shifts its attention to the text.
If a clue came from the Text Search, it shifts attention to the visual details.
This prevents the AI from getting stuck in a loop of looking at the same thing twice. It uses the "ID card" to decide exactly how much weight to give to the picture versus the words.

Step 4: The Final Mix
Finally, UniHEAR adds a "training-free" bonus. It takes the original scores from the search engines and mixes them in with the new smart sorting score, using a method called Entropy-Weighted Source Fusion. This ensures that if a search engine was very confident (low entropy), its clues get a little extra boost.

The Results: Faster and Smarter

The authors tested UniHEAR on two major datasets, E-VQA and InfoSeek, which are like the "final exams" for these AI detectives.

Better at Finding Clues: On the E-VQA test, UniHEAR improved the "Recall@1" score (the ability to find the very first correct answer) by 6.7 points over the strongest previous method. On InfoSeek, it improved by 1.2 points. This means it found the right answer much more often than before.
Better at Answering Questions: When it came to actually answering the questions, UniHEAR achieved state-of-the-art results. For example, on the E-VQA dataset, it scored 53.2%, beating other top methods that used much larger, heavier models.
Lightweight: Despite doing more work (searching two sources and using complex filters), UniHEAR is surprisingly light. It uses a model with only 197 million parameters, whereas its competitors often use models with 1.2 billion to 1.7 billion parameters. It's like getting the power of a supercomputer in a backpack.

What This Means

The paper explicitly argues against the idea that we need to rely on a single type of search or that we can ignore where a clue came from. The authors show that ignoring the "source" of the information leads to mistakes. By proving that combining visual and text searches and then using a smart, source-aware filter works better, UniHEAR suggests a new way forward for AI. It shows that you don't need a giant, heavy brain to be smart; you just need to know how to look in the right places and listen to the right clues.

In short, UniHEAR is a lighter, faster, and more accurate detective that knows how to use both its eyes and its reading glasses to solve visual mysteries, proving that sometimes, the best way to find the truth is to look at it from two different angles at once.

Technical Summary: UniHEAR

Problem Statement

Knowledge-Based Visual Question Answering (KB-VQA) requires models to retrieve external entity knowledge (e.g., from Wikipedia) to answer questions grounded in visual input. Existing retrieval-augmented systems face two critical limitations:

Single-Source Retrieval Bottleneck: Current systems typically rely on a single retrieval modality, either Image-to-Image (I2I) or Image-to-Text (I2T). These modalities diverge significantly; I2I prioritizes morphological similarity (color, shape), while I2T emphasizes semantic consistency. Consequently, under limited retrieval budgets, ground-truth entities are often captured by only one source, rendering single-source retrieval inherently incomplete and suppressing the upper bound of VQA accuracy.
Retrieval-Source-Blind Reranking: Existing rerankers often assign uniform modality attention across all candidates, ignoring the retrieval origins and candidate-level priors established during the coarse retrieval stage. This leads to redundant reliance on modalities that have already been exploited by large-scale coarse retrievers, reducing the discriminability of the reranking process and causing misranking of ground-truth entities.

Methodology: UniHEAR

The authors propose UniHEAR (Unified Heterogeneous-Source Attentive Retrieval), a lightweight framework designed to bridge coarse retrieval and fine-grained reranking through a unified architecture. The framework operates in three stages:

1. Heterogeneous-Source Coarse-Grained Retrieval

The system simultaneously matches the query image against two complementary knowledge base sources: entity images (I2I) and textual summaries (I2T).

Candidate Pool: The union of top- $k$ candidates from both sources forms a unified candidate pool, ensuring complementary coverage.
Coarse Retrieval Descriptor (CRD): Each candidate entity is characterized by a descriptor $D(e)$ $D (e)$ that encodes retrieval evidence as prior guidance. This descriptor includes:
- Normalized rank ( $r_s$ ) within each source.
- Raw similarity scores ( $sim_s$ ) and z-normalized similarity ( $z_s$ ).
- Similarity gap to the top-ranked entity ( $\delta_s$ ).
- Pool-level entropy ( $H_s$ ) to measure the concentration of the retrieval distribution and source reliability.
  This descriptor bridges the gap between large-scale coarse retrievers and the lightweight reranker.

2. Retrieval-Guided Multimodal Entity Reranking

This stage employs a lightweight reranker built upon the VISTA architecture, enhanced by two novel modules:

Retrieval-Guided Attentive Modality Gating (RAMG): Instead of treating all candidates equally, RAMG conditions modality attention weights on the Coarse Retrieval Descriptor. It uses affine modulation parameters ( $\gamma, \beta$ ) derived from the descriptor to adjust attention logits. This allows the model to suppress reliance on modalities already exploited during coarse retrieval (e.g., reducing visual attention for candidates already confirmed via I2I) and aggregate modality-specific representations ( $C_{it}, C_t, C_i$ ) into a unified fused representation.
Entropy-Weighted Source Fusion (EWSF): This module re-incorporates coarse retrieval priors as a training-free complement to the learned reranking score. It weights source-specific scores by normalized rank and pool entropy, then calibrates them via a pool-mean ratio before fusing them with the learned reranking score.

3. Multi-Entity Section-Augmented Generation

For the final answer generation, the framework aggregates sections across the top- $k_e$ reranked entities.

Section Selection: It combines fine-grained section-level IT2T similarity with entity-level retrieval scores.
Rank-Proportional Allocation: Sections are allocated proportionally to the entity's rank (e.g., the top-ranked entity receives more sections than lower-ranked ones) to concentrate the context budget on the most confident entities.
Generation: A Multimodal Large Language Model (MLLM) generates the final answer conditioned on the input image, question, and the aggregated retrieved sections.

Training Strategy

UniHEAR employs a hybrid training strategy to unify entity-level and section-level retrieval within a single model:

Contrastive Loss ( $L_{con}$ ): Optimizes the fused representations to distinguish positive pairs from hard negatives.
Auxiliary Modality-Preserving Loss ( $L_{aux}$ ): An auxiliary loss preserves cross-modal matching capabilities (IT2T, I2T, I2I) required for downstream section-level retrieval. It includes a binary mask to filter out unreliable positive supervision (e.g., when ground-truth image labels are missing or URLs are inaccessible), preventing modality collapse.

Key Contributions

Unified Framework: Proposes UniHEAR, a lightweight framework that unifies heterogeneous-source entity retrieval and reranking, constructing a Coarse Retrieval Descriptor to encode multi-source evidence.
Novel Gating Mechanism: Introduces Retrieval-Guided Attentive Modality Gating (RAMG), which conditions modality attention on retrieval descriptors to mitigate redundant modality reliance.
Training-Free Fusion: Implements Entropy-Weighted Source Fusion (EWSF) to re-incorporate calibrated coarse retrieval priors without additional training overhead.
Unified Training Objective: Designs a training strategy combining contrastive learning with an auxiliary modality-preserving loss to enable seamless entity-to-section retrieval within a single model.

Experimental Results

Extensive experiments were conducted on the E-VQA and InfoSeek datasets.

Retrieval Performance: UniHEAR achieves state-of-the-art retrieval performance. On E-VQA, it improves Recall@1 by 6.7 points over the strongest baseline (OMGM) and by 33.9 points over MLLM-based baselines (ReflectiVA). On InfoSeek, it improves Recall@1 by 1.2 points over OMGM.
VQA Performance: Without generation fine-tuning, UniHEAR with Qwen2.5-VL-7B achieves 53.2% accuracy on E-VQA, surpassing all fine-tuned retrieval-augmented baselines under the same generator. With a fine-tuned generator, it achieves 50.4% on E-VQA and 44.1% on InfoSeek, outperforming all baselines.
Efficiency: Despite unifying heterogeneous sources and handling a larger candidate pool, UniHEAR maintains a lightweight architecture with only 197M parameters (compared to 1.2B–1.7B for BLIP-2-based competitors). The RAMG module adds only 0.59M parameters (0.3% overhead), demonstrating that retrieval-guided gating enhances performance with negligible parameter cost.

Significance

The paper claims that UniHEAR establishes heterogeneous-source retrieval with retrieval-guided reranking as a promising paradigm for future KB-VQA research. By explicitly addressing the Single-Source Retrieval Bottleneck and Retrieval-Source-Blind Reranking, the framework demonstrates that leveraging complementary signals from multiple sources, guided by coarse retrieval priors, can significantly improve both retrieval accuracy and downstream VQA performance while maintaining high efficiency. The results suggest that lightweight, unified models can outperform larger, single-source or multi-stage systems by effectively utilizing the rich evidence already surfaced during coarse retrieval.