Technical Summary: Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) frequently struggle with fine-grained visual reasoning tasks. While these models inherit strong language reasoning capabilities, their visual reasoning is often hindered by visual redundancy. In standard MLLM architectures, images are encoded into dense sequences of visual tokens. However, for specific questions, only a small subset of these tokens constitutes relevant evidence. The remaining tokens—representing background regions, unrelated objects, or visually similar distractors—dilute the critical cues.

Existing approaches attempt to address this by extending reasoning traces (e.g., Chain-of-Thought) or adding more visual/latent context (e.g., image crops, intermediate latent tokens). The authors argue that these additive methods fail to solve the core issue: irrelevant visual tokens remain in the context, competing with key evidence and making the decoding process harder. The paper posits that when evidence is already present, the primary challenge is visual selectivity (filtering distractors) rather than visual availability (finding new evidence).

2. Methodology: Latent Noise Mask (LENS)

The authors propose LENS (LatEnt Noise maSk), a framework designed to purify visual evidence before the decoding phase without altering the model backbone or token sequence structure. The method operates in two coupled stages:

A. Evidence Probing

LENS introduces a lightweight, temporary Lens Evidence Token (LET), denoted as <mask>, which is appended to the joint input of visual tokens ($V$) and text tokens ($T$).

• Mechanism: The hidden state of this temporary token is decoded by a simple Multi-Layer Perceptron (MLP) head ($f_\theta$) to produce a relevance score vector $\mathbf{a} \in [0, 1]^N$, where $N$ is the number of visual tokens.
• Conditioning: These scores are question-conditioned, meaning the same image region may receive a high score for one question and a low score for another.
• Supervision: The LET is trained using object-level annotations (bounding boxes) rather than expensive chain-of-thought rationales. A token is labeled positive if its corresponding image patch overlaps with a question-relevant bounding box. The training objective minimizes Binary Cross-Entropy (BCE) loss between the predicted scores and these binary labels.

B. Latent Suppression

Once the relevance scores are generated, the <mask> token is removed, and the scores guide a soft suppression mechanism in the latent space.

• Noise Generation: A lightweight noise generator ($G_\phi$) predicts a mean ($\mu_i$) and scale ($\sigma_i$) for a token-specific Gaussian noise distribution for each visual token $v_i$.
• Adaptive Perturbation: The actual noise $r_i$ is sampled via reparameterization: $r_i = \mu_i + \sigma_i \odot \epsilon_i$, where $\epsilon_i \sim \mathcal{N}(0, I)$.
• Gating: A suppression gate $g_i$ is computed based on the LET score $a_i$ and a threshold $\tau$: $g_i = \frac{1}{\tau} \text{ReLU}(\tau - a_i)$.
  • If $a_i \geq \tau$ (high relevance), $g_i = 0$, and the token remains unchanged.
  • If $a_i < \tau$ (low relevance), $g_i > 0$, and adaptive noise is injected.
• Purified Sequence: The final visual sequence is $\tilde{V} = \{v_i + g_i r_i\}$. This preserves the original token layout (crucial for compatibility with existing backbones) but destabilizes the representation of distractor tokens, effectively "masking" them in the latent space.

C. Optimization

The framework is trained in two phases:

1. Supervised Fine-Tuning (SFT): The model is trained to answer questions using the purified sequence $\tilde{V}$. The total loss combines the answer generation loss and the LET supervision loss ($\mathcal{L}_{SFT} = \mathcal{L}_{ans} + \beta \mathcal{L}_{LET}$).
2. Reinforcement Fine-Tuning (RFT): To refine the evidence selection policy, the authors model the LET scores as a Bernoulli policy. They optimize for a set-level reward (F1 score) based on the overlap between selected tokens and ground-truth relevant boxes, regularized by KL divergence from the SFT checkpoint.

3. Key Contributions

1. Problem Re-framing: The paper identifies that dense visual tokens contain question-irrelevant distractors that hinder fine-grained reasoning. It argues that suppressing these distractors is more effective than simply adding more reasoning context.
2. Lightweight Framework: LENS introduces a question-conditioned purification mechanism that requires only one temporary learnable control token and a lightweight noise generator. It does not require modifying the backbone architecture, cropping images, or generating intermediate visual outputs.
3. Soft Latent Intervention: Unlike token pruning or hard region selection, LENS uses adaptive latent noise to softly suppress low-relevance tokens. This maintains the structural integrity of the token sequence while reducing the influence of distractors.

4. Experimental Results

The authors evaluated LENS on 10 benchmarks covering Visual Question Answering (VQA) and Grounding tasks, using base models from the Qwen3-VL, InternVL3.5-VL, and Qwen3.5 families.

• Performance Gains:

  • VQA: LENS improved the base MLLM by 2.4–6.4 points across most VQA datasets (e.g., CUB, GQA, OpenImages). The LENS-GRPO variant achieved an average improvement of 6.65 points over the strongest non-LENS baseline.
  • Grounding: Improvements were even more pronounced, with gains of 4.1–6.4 points on grounding tasks (e.g., COCO, Objects365). On VisDrone, LENS-GRPO improved performance by 18.81 points over SFT.
  • Generalization: The method proved transferable across different model scales (2B to 9B) and architectures, consistently improving performance without task-specific architectural changes.

• Ablation Studies:

  • Removing the latent suppression (using only Evidence Probing) yielded limited gains, confirming that identifying evidence is insufficient without actively suppressing distractors.
  • Replacing the learned soft noise with random masking, zero-padding, or attention-based masking resulted in significant performance drops, validating the necessity of learned, adaptive, soft suppression.
  • Reinforcement tuning (GRPO) further enhanced results, suggesting that optimizing the evidence selection policy at a set level is beneficial.

5. Significance and Claims

The paper claims that cleaner question-relevant visual evidence is a more direct path to improving multimodal reasoning than extending the reasoning trace. By demonstrating that suppressing irrelevant tokens yields superior results compared to adding more context, the authors suggest a shift in paradigm for MLLM reasoning: from visual addition to visual purification.

The significance lies in the efficiency and generality of the approach. LENS achieves state-of-the-art improvements in fine-grained reasoning and spatial localization with minimal overhead, proving that the bottleneck in current MLLMs is often the "noise" in the visual input rather than a lack of reasoning capacity. The visualizations confirm that LENS successfully focuses attention on key regions (attributes, objects, relations) while suppressing backgrounds and distractors, leading to more reliable fine-grained vision-language reasoning.