💬 NLP

Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

This paper reveals that multimodal Knowledge-Based Visual Question Answering systems suffer from a "Lost at the End" phenomenon where placing the correct retrieved passage at the beginning of the context significantly outperforms placing it at the end, a bias driven by the reader model's prompt slot 0 rather than retrieval quality, thereby challenging the adequacy of recall@k as a performance metric.

Original authors: Jieyuan Liu, Jianyang Gu, Shijie Chen, Jefferson Chen, Zhen Wang

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jieyuan Liu, Jianyang Gu, Shijie Chen, Jefferson Chen, Zhen Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Goldilocks" Problem

Imagine you are a detective (the AI) trying to solve a mystery (answer a question) about a specific object in a photo. You have a library of 50 books (retrieved text passages) that might contain the answer.

In the world of pure text AI, researchers previously found a "U-shaped" problem: If you put the right book at the very beginning or the very end of the stack, the detective finds the answer. But if you hide that book in the middle of the stack, the detective ignores it completely. This is called the "Lost in the Middle" effect.

This paper asks: Does this same problem happen when the detective is looking at a photo and reading the books?

The Experiment: The "Gold Position" Test

The researchers set up a controlled experiment to find out. They took a single question and a single photo, and they gave the AI the exact same 10 books every time. The only thing they changed was where the "Gold Book" (the one with the correct answer) was placed in the list:

First: The Gold Book is at the very top.
Middle: The Gold Book is in the middle of the stack.
Last: The Gold Book is at the very bottom.

They did this with three different AI "detectives" (large vision-language models) to see how they performed.

The Discovery: "Lost at the End"

The results were surprising. The "U-shape" disappeared. Instead, the AI developed a strong "Primacy Bias" (a preference for the beginning).

The Analogy: Imagine a teacher reading a list of students' names to pick a class representative. If the teacher hears the right name first, they pick it. If they hear it last, they often forget it entirely.
The Result: When the Gold Book was at the start, the AI got the answer right about 60–65% of the time. When the Gold Book was at the end, the accuracy dropped to about 35–45%.
The "Lost at the End" Effect: The AI didn't just ignore the middle; it actively ignored the end. The "Gold Book" at the end was effectively invisible. The AI was 2.2 to 4.5 times more likely to get the answer right if the correct information was at the very top of the list compared to the bottom.

Why Is This Happening? (The Investigation)

The researchers wanted to know why the AI ignored the end. They tested three theories:

Is it the photo? They moved the photo to the end of the prompt. Result: No change. The photo's position wasn't the main culprit.
Is it the quality of the book? They shuffled the books so the "best" book (highest relevance score) wasn't always at the top. Result: The AI still picked the book at the top of the list, even if it was a worse book. It followed the position, not the quality.
Is it just text? They tested the AI with only text (no photo). Result: The AI still preferred the first book, but the gap was smaller. Conclusion: Adding the photo amplified the bias. The photo made the AI even more stubborn about only looking at the start.

The Failed Fixes

The researchers tried to fix this by changing how the books were retrieved (the "search engine" part of the system):

They tried to make the list more diverse.
They tried to force the best book to the top.
Result: None of these retrieval-side fixes worked. As long as the AI (the reader) was "frozen" (not retrained), it would still ignore the end of the list.

The Main Takeaway

The paper concludes that for these specific AI systems, Recall@K (a common metric that asks, "Did the right answer appear somewhere in the top 50 books?") is the wrong way to measure success.

The Analogy: It's like a librarian saying, "I found the right book in the library!" but the reader is only allowed to look at the first book on the shelf. If the right book is on the last shelf, the reader fails, even though the librarian did their job.

The Solution: To fix this, we cannot just improve the search engine (retrieval). We have to retrain the reader (the AI) to pay attention to the whole list, not just the beginning. Until then, if you want an AI to answer a question correctly, you must put the most important information at the very top of its prompt.

Technical Summary: Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

Problem Statement
Knowledge-based Visual Question Answering (KB-VQA) systems rely on retrieving passages from a large knowledge base (e.g., Wikipedia-scale) to condition a vision-language reader (VLM) to answer questions about an image. While pure-text long-context Large Language Models (LLMs) exhibit a well-documented "lost-in-the-middle" effect (Liu et al., 2024)—where information at the start and end of a context is utilized while the middle is ignored—it remains unknown whether this phenomenon transfers to deployed multimodal KB-VQA systems. Existing multimodal position-bias studies have not characterized this setting at deployment scale, often using small numbers of distractors or general image-text comprehension benchmarks rather than specific KB-VQA stacks. Consequently, it is unclear if the U-shaped bias holds, if a different shape emerges, where the bias originates (image proximity vs. text position), and whether retrieval-side optimizations (e.g., diversification, reranking) can mitigate the issue for frozen readers.

Methodology
The authors design a controlled "gold-position protocol" to isolate reader-side position dependence in multimodal KB-VQA.

Protocol Design: For each question, three prompts are constructed that are bit-identical except for the position of the single "gold" (relevant) passage. The gold passage is placed at the first slot (index 0), middle slot (index $\lfloor(k-1)/2\rfloor$ ), or last slot (index $k-1$ ). All other $k-1$ slots are filled with identical distractors.
Scale and Setup: The study operates at a deployment scale using top-50 candidates from a strong multi-vector retriever (PreFLMR ViT-G) and varying $k$ up to 20.
Models and Benchmarks: Experiments are conducted on three open-source 7B/8B instruction-tuned VLMs (Qwen2.5-VL-7B, InternVL3-8B, Qwen3-VL-8B) across two KB-VQA benchmarks (INFOSEEK and E-VQA).
Statistical Rigor: The study employs a within-prompt paired design with paired-bootstrap inference (10,000 resamples) to ensure exact paired comparisons. A pre-registered five-way classifier is used to categorize effect sizes (e.g., Generalization, Amplified) to prevent post-hoc harvesting.
Ablations: The authors perform targeted ablations to determine the locus of the effect:
1. Text-only control: Removing the image to measure baseline text-mode bias.
2. Image-position ablation: Moving the image token block to the end of the prompt to test for image-token proximity effects.
3. Distractor-shuffle ablation: Randomizing the order of distractors relative to their retrieval rank to decouple prompt slot position from retrieval similarity.
4. Retrieval-side fixes: Testing Maximum Marginal Relevance (MMR) diversification, oracle reranking, and rank-based reordering on a frozen reader.

Key Results

Primacy Bias, Not U-Shape: Contrary to the U-shaped "lost-in-the-middle" effect in text-only LLMs, multimodal KB-VQA exhibits a strong primacy bias. The "gold-at-first" configuration outperforms "gold-at-last" by 16 to 26 percentage points across all six reader-benchmark combinations. The performance curve is a monotonic decay rather than a U-shape; the middle and last positions often collapse together below the first position.
Multimodal Amplification: A text-only ablation reveals that primacy bias exists in the underlying text readers but is amplified 2.2 to 4.5 times in the multimodal setting. The multimodal context reshapes the bias, making the gap between the first and last positions significantly larger than in text-only modes.
Locus is Prompt Slot 0: The bias is anchored to prompt slot 0 of the instruction-tuned reader.
- Moving the image to the end of the prompt did not reduce the first-vs-last gap, ruling out image-token proximity as the primary driver.
- A distractor-shuffle experiment showed that when the most similar distractor (rank-1) was moved away from slot 0, the reader's error rate did not follow the similarity rank but remained high when the distractor was at slot 0. This confirms the reader prioritizes the first slot regardless of retrieval similarity.
Retrieval-Side Fixes Fail: Training-free retrieval-side interventions, including MMR diversification, oracle reranking (forcing gold to the top), and rank-based reordering, failed to close the gap on a frozen reader. The first-vs-last performance gap remained intact within confidence intervals.

Significance and Claims
The paper claims that Recall@k is the wrong metric for deployed KB-VQA systems relying on frozen readers. Because the reader systematically ignores or underutilizes relevant information unless it appears at the very beginning of the prompt, optimizing retrieval for recall (finding the gold passage in the top- $k$ ) does not guarantee system reliability if the gold passage is not at the optimal slot.

The authors conclude that closing the performance gap requires reader-side intervention (e.g., position-aware fine-tuning, attention calibration, or data rebalancing) rather than further optimization of the retrieval component. They release their gold-position protocol as a controlled instrument for the community to evaluate future reader-side or training-time interventions.

Limitations Acknowledged by the Authors

Scope: Findings are specific to open-source 7B/8B VLMs, short contexts ( $k \le 20$ ), and English Wikipedia-derived knowledge bases. Proprietary models or larger scales may behave differently.
Mechanism: While the locus is identified as prompt slot 0, the underlying mechanism (e.g., specific attention patterns or instruction-tuning data distributions) is not fully characterized.
Confounding: The text-only ablation compares different datasets (NQ-Open vs. M2KR), meaning the reported "amplification" conflates modality and corpus effects, though the authors argue the amplification is a property of the multimodal KB-VQA setting.
Heuristics: The failure bucket diagnostic relies on string matching, which may conflate surface-form and semantic errors.