🤖 machine learning

Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models

This paper introduces Visual-Redundancy-Controlled Decoding (VRCD), a training-free inference method for diffusion-based multimodal large language models that mitigates the limitations of independent confidence-based token selection by prioritizing visually complementary positions to reduce redundancy and significantly improve accuracy on multimodal benchmarks.

Original authors: Yulin Yuan, Hongshuo Zhao, Xiangming Meng

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yulin Yuan, Hongshuo Zhao, Xiangming Meng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Team of Artists Painting a Picture Together

Imagine you have a team of artists trying to paint a picture of a scene based on a description. In the old way of doing things (called "autoregressive"), they would paint one tiny brushstroke at a time, waiting for the previous one to dry before starting the next. This is slow.

The new method described in this paper (called Diffusion-based Multimodal Large Language Models, or dMLLMs) is like a team of artists working in parallel. Instead of painting one stroke, they look at the whole canvas, guess what many different parts of the picture should look like simultaneously, and then decide which of those guesses are good enough to "lock in" as the final paint.

The Problem: Everyone Looking at the Same Spot

The paper identifies a specific problem with how these teams currently decide which guesses to lock in.

Usually, the team leader looks at each artist's guess and says, "Okay, Artist A is 90% sure about this tree, and Artist B is 90% sure about that tree. Let's lock both of them in!"

The flaw: The paper argues that Artist A and Artist B might both be looking at the exact same tree in the reference photo. They are both confident, but they are providing redundant information. They are both staring at the same visual detail.

Because they locked in two guesses about the same tree, the team has now "used up" their visual attention on that one spot. They have less visual information left to help them figure out the other parts of the picture (like the sky or the grass) in the next round of painting.

The authors call this "Visual Redundancy." It's like having a committee where everyone votes on the same thing, leaving no one to vote on the other important issues.

The Solution: The "Visual Redundancy Controller" (VRCD)

To fix this, the authors created a new rule for the team leader called VRCD (Visual-Redundancy-Controlled Decoding).

Instead of just asking, "Who is the most confident?", VRCD asks: "Who is confident, AND who is looking at a different part of the picture?"

Here is how VRCD works, step-by-step:

The Candidate List: First, it gathers the top artists who are most confident in their guesses (just like before).
The "Gaze" Check: It looks at where each artist is looking in the reference photo. It uses a special tool (called "token-to-image attention") to see if Artist A and Artist B are staring at the same leaf or the same cloud.
The Penalty: If two artists are staring at the same spot, VRCD gives them a "redundancy penalty." It says, "You are both right, but you are repeating each other."
The Selection: VRCD then picks the group of artists who are confident and are looking at the most diverse, complementary parts of the image.

The Result: A Better Team Dynamic

By forcing the team to pick artists looking at different parts of the image, the paper found that:

Less Confusion: The team doesn't waste time re-examining the same object.
Better Context: Because they locked in a diverse set of details (a tree, a cloud, and a car), the remaining artists have a much richer "context" to help them guess the rest of the picture in the next round.
Speed: The team doesn't slow down much. It's a very lightweight check, like a quick glance, rather than a full re-evaluation.

The Evidence

The authors tested this on several difficult puzzles (benchmarks) involving images and text, such as:

M3CoT: Complex reasoning questions with multiple steps.
MMBench: General vision and language understanding.

They found that using VRCD made the models significantly more accurate (up to nearly 19% better on some complex tasks) compared to the standard method. The models made fewer mistakes because they weren't "double-dipping" on the same visual clues.

Summary Analogy

Imagine you are assembling a puzzle with a group of friends.

The Old Way: You ask everyone, "What piece fits here?" and you grab the first three pieces that anyone says fit, even if they are all pieces for the sky. You end up with three sky pieces and no pieces for the ground.
The VRCD Way: You ask, "What fits here?" and you grab the pieces that fit, but you make sure you don't grab three sky pieces. You grab one sky piece, one ground piece, and one tree piece. Now, when you try to fill in the rest of the puzzle, you have a much better idea of what the whole picture looks like.

This paper simply teaches the AI how to be a better puzzle assembler by ensuring it picks a diverse set of visual clues at every step.

Technical Summary: Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models

1. Problem Statement

Diffusion-based Multimodal Large Language Models (dMLLMs) generate text by iteratively decoding multiple masked positions in parallel, conditioned on visual input. Unlike autoregressive models that extend a prefix token-by-token, dMLLMs must decide, at each decoding step, which subset of masked positions to "commit" (unmask) to serve as context for subsequent steps.

Existing decoding strategies, primarily confidence-based decoding, select the top- $K$ positions with the highest predicted token probabilities. While efficient, this approach treats each position independently. The authors identify a critical limitation in multimodal settings: visual redundancy. High-confidence tokens selected in the same step may rely on overlapping visual grounding (i.e., attending to the same image regions or objects). When such tokens are committed together, they provide redundant visual context rather than complementary cues, potentially degrading the model's ability to resolve remaining masked positions that require diverse visual information. This redundancy is distinct from semantic repetition in the generated text or redundancy within the input visual tokens; it specifically refers to the overlap in visual grounding among textual tokens unmasked simultaneously.

2. Methodology

2.1 Visual Redundancy Index (VRI)

To quantify the extent of overlapping visual grounding among committed tokens, the authors introduce the Visual Redundancy Index (VRI).

Definition: Given a set of committed positions $S_t$ at step $t$ , VRI measures the overlap in token-to-image attention.
Mechanism: For each image token $u$ , the total attention weight assigned by committed tokens is calculated. The maximum single contribution is subtracted to isolate the "redundant" weight assigned by additional tokens.
Formula:
$\text{VRI}(S_t) = \frac{\sum_{u=1}^{N} (\sum_{i \in S_t} v_i(u) - \max_{i \in S_t} v_i(u))}{m - 1}$
where $v_i(u)$ is the normalized attention of text token $i$ to image token $u$ , and $m = |S_t|$ . A lower VRI indicates less visual redundancy.

2.2 Visual-Redundancy-Controlled Decoding (VRCD)

The authors propose VRCD, a training-free, inference-time decoding method designed to mitigate visual redundancy without altering the underlying model weights. VRCD operates as a lightweight reranking mechanism:

Candidate Window Construction: Instead of evaluating all masked positions, VRCD constructs a candidate window $U_t$ containing the top- $M_t$ positions based on initial confidence scores, where $M_t$ is a function of the commit size $K_t$ and a scaling factor $\lambda$ .
Visual Saliency Extraction: For each candidate, the method extracts normalized token-to-image attention from the final decoder layer. To reduce noise, a uniform component ( $1/N$ ) is subtracted, and only positive residual attention weights are retained and renormalized.
Pairwise Overlap Calculation: The visual overlap between every pair of candidates $(i, j)$ is computed using the intersection of their saliency distributions (e.g., $\sum \sqrt{q_i(u)q_j(u)}$ ).
Redundancy Score Computation: Overlaps are converted to percentile ranks within the current window. A token-level redundancy score $r_i$ is calculated as a confidence-weighted sum of its overlaps with other likely candidates. This ensures that tokens overlapping with high-confidence neighbors are penalized more heavily.
Reweighting: The original confidence score $c_i$ is reweighted to form a redundancy-controlled score $s_i$ :
$s_i = c_i (r_i + 1)^{-\alpha}$
where $\alpha$ controls the strength of the redundancy penalty.
Selection: The top- $K_t$ positions with the highest $s_i$ are committed. This process prioritizes high-confidence tokens that offer visually complementary grounding.

3. Key Contributions

Identification of Visual Redundancy: The paper identifies and formalizes the phenomenon where individually confident tokens in parallel decoding steps rely on overlapping visual grounding, a factor previously overlooked in dMLLM decoding strategies.
VRI Metric: Introduction of the Visual Redundancy Index (VRI) to systematically quantify and analyze visual grounding overlap across decoding steps.
VRCD Algorithm: Proposal of a lightweight, training-free reranking method that utilizes token-to-image attention to prioritize visually complementary positions, thereby reducing redundancy while maintaining the efficiency of confidence-based decoding.

4. Experimental Results

The method was evaluated on five multimodal benchmarks: M3CoT (multi-step reasoning), MMBench (general vision-language), SQA-IMG (science QA), DocVQA, and InfoVQA. The backbone model used was LaViDa (based on LLaDA-8B).

Performance Gains: VRCD consistently outperformed confidence-based decoding and other baselines (Margin, Entropy, InfoGain) across various decoding lengths ( $L=192, 384$ $L = 192, 384$ ) and forward ratios (FR).
- On M3CoT, VRCD achieved relative accuracy gains of up to 18.8% over confidence-based decoding.
- On MMBench, gains reached 6.9%.
- Improvements were most significant in tasks requiring long-horizon multimodal reasoning (M3CoT, MMBench, SQA-IMG) compared to short-answer tasks (DocVQA, InfoVQA).
Redundancy Reduction: VRCD significantly lowered the mean VRI across all decoding steps compared to baselines, confirming its ability to select tokens with less overlapping visual grounding.
Predictive Certainty: The method reduced the entropy of remaining masked positions, indicating higher predictive certainty for future steps.
Efficiency: VRCD introduced minimal runtime overhead (approx. 1.5% increase in runtime), retaining near-baseline throughput (tokens/s), whereas other advanced methods like InfoGain incurred significantly higher computational costs.
Ablation Studies:
- The hyperparameter $\alpha$ (redundancy penalty strength) showed optimal performance at $\alpha=1.5$ for M3CoT and $\alpha=2.0$ for MMBench; a fixed moderate setting was used across benchmarks.
- Confidence-weighted aggregation of redundancy scores outperformed uniform averaging.
- Visual Saliency Extraction (VSE) was shown to be crucial for effective overlap estimation.

5. Significance and Claims

The paper claims that visual redundancy is a critical, yet previously unaddressed, signal for parallel decoding in visually grounded dMLLMs. By shifting the decoding decision from "which individual token is most confident" to "which set of tokens provides complementary visual grounding," VRCD improves the quality of the generated context for subsequent steps.

The authors emphasize that VRCD is a plug-in, training-free method that enhances decoding quality without requiring model retraining or significant computational overhead. The work suggests that managing the diversity of visual grounding during parallel commitment is essential for maximizing the efficiency and accuracy of diffusion-based multimodal generation. Future work is suggested to explore adaptive decoding schedules that dynamically adjust commit sizes based on visual redundancy signals.