Technical Summary: ReToken

Problem Statement

Vision-Language Models (VLMs) face significant challenges when processing long visual contexts, such as image collections or hour-long videos. Two primary issues arise:

1. Performance Degradation: As the number of distractor images or frames increases, model performance drops because the relevant information is buried within a large pool of irrelevant data.
2. Computational Infeasibility: Processing all visual tokens simultaneously often exceeds GPU memory constraints, making full-context inference impossible for long videos.

Current solutions often rely on attention-based retrieval, where the model's internal attention scores (query-key interactions) are used to select relevant frames. However, the authors identify that in pretrained VLMs, these attention signals are unreliable for visual retrieval. The correlation between attention scores and query relevance is weak, leading to poor recall (e.g., only 5.1% Recall@1 across layers on the QAEgo4DTest-MC benchmark). Furthermore, standard attention mechanisms are trained for next-token prediction, not for selecting specific visual inputs, and averaging question tokens to form a query introduces noise that further degrades retrieval performance.

Methodology: ReToken

The paper proposes ReToken, a lightweight, learnable retrieval mechanism designed to operate within the value space of the transformer rather than the conventional query-key space.

Core Insights

• Value Space Superiority: The authors observe that while query-key scores are noisy, the value projections of tokens carry the actual content propagated through attention. Pooling value vectors provides a more discriminative signal for text-based visual retrieval.
• Targeted Retrieval: Instead of averaging all question tokens (which introduces noise), ReToken uses a single, learnable embedding appended to the question prompt. This token is explicitly trained to act as a retrieval target.

Architecture and Training

• Learnable Token: A single learnable embedding ($X_r$) is appended to the input question tokens.
• Projection: A lightweight projection matrix ($W_r$) maps the final layer output of the ReToken to a retrieval query vector ($Z_r$).
• Scoring Mechanism: For each frame $f$, the model computes the mean value vector ($\bar{v}_f^{(N)}$) from the final layer of the VLM. The retrieval score is the cosine similarity between the projected ReToken and the frame's mean value vector.
• Loss Function: The system is supervised using a class-balanced binary cross-entropy loss against ground-truth relevance labels. The loss encourages high scores for relevant frames and low scores for distractors.
• Training Regime: By default, the VLM backbone is frozen. Only the ReToken embedding and the projection matrix are updated. A partial fine-tuning variant (tuning the first few layers) is also explored but found to be less effective for video transfer.

Inference Pipeline

ReToken employs a two-pass retrieve-then-answer pipeline:

1. Cache: The video is encoded once, and the per-layer Key-Value (KV) caches are stored in a persistent cache.
2. Retrieve (Pass 1): Given a question, the ReToken is appended, and the model runs a forward pass over the cached visual context. The final layer computes retrieval scores to select the top-$K$ relevant frames.
3. Answer (Pass 2): A second forward pass generates the answer, attending only to the KV cache of the selected top-$K$ frames. This ensures the generation process matches the training distribution (where retrieval was determined at the final layer).

Key Contributions

1. Diagnosis of Attention Limitations: The paper demonstrates that attention-based retrieval (query-key) is ineffective in pretrained VLMs, whereas the value space provides a significantly stronger signal for retrieving visual information.
2. ReToken Mechanism: Introduction of a single learnable token that enables pretrained VLMs to identify relevant visual information more effectively without requiring full model retraining.
3. Zero-Shot Transfer: Demonstration that a ReToken trained exclusively on multi-image Question Answering (QA) data generalizes successfully to long-video understanding benchmarks without further video-specific training.

Experimental Results

The method was evaluated on image and video benchmarks, showing consistent improvements over baselines:

• Visual Haystacks (Image Retrieval):
  • On the Visual Haystacks benchmark, ReToken improved Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (over 20% relative gain) in the presence of 50 distractor images.
  • It achieved an 88.5% Recall@1 on the 2-image task and 64.7% on the 50-image task, significantly outperforming attention-based retrievers (ReKV) and external embedding models (SigLIP2).
• Long Video Understanding:
  • QAEgo4DTest-MC: With a tight retrieval budget ($K=1$), ReToken achieved a 6.8-point improvement over uniform sampling and outperformed attention-based retrieval.
  • LVBench: ReToken transferred zero-shot to long videos (average length >1 hour), yielding an 8.0-point gain in accuracy for Qwen3VL-8B.
  • Video-MME: Significant gains were observed on medium and long video splits, while short videos (under 2 minutes) showed no gain as they fell below the retrieval input budget.
• Efficiency:
  • Both training and long-video inference fit on a single H100 GPU.
  • The retrieval pass adds only ~0.4 seconds of latency per question, while the expensive video encoding is performed only once per video.

Significance and Claims

The paper claims that ReToken offers a practical path toward scalable long-context multimodal reasoning. By identifying that the value space carries a stronger text-aligned signal than the query-key space, the authors provide a simple, lightweight solution that bypasses the need for complex external retrievers or full model fine-tuning.

The work highlights that:

• Retrieval in the value space is more robust to noisy queries than in the query-key space.
• A single learnable token can effectively replace complex retrieval pipelines.
• Skills learned on short-context multi-image QA can transfer to long-video understanding, suggesting that the fundamental capability to align text with visual content is preserved even in frozen models.

The authors acknowledge limitations, noting that ReToken currently struggles with questions requiring evidence dispersed across the entire video (e.g., summarization) and that its performance on video tasks could potentially be improved by training on video data to better capture temporal structures. However, the current zero-shot transfer from images to videos is presented as a significant step forward for efficient long-context VLMs.