💬 NLP

VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

The paper proposes VoidPadding, a method that decouples padding and termination roles in Masked Diffusion Language Models by introducing a dedicated [VOID] token for padding, which resolves [EOS] overflow issues and significantly improves performance and decoding efficiency on instruction-tuned models.

Original authors: Chunyu Liu, Zhengyang Fan, Kaisen Yang, Alex Lamb

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chunyu Liu, Zhengyang Fan, Kaisen Yang, Alex Lamb

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write stories. In the old way (Autoregressive models), the robot writes one word at a time, left to right, like a human typing. But in this new method called Masked Diffusion, the robot starts with a blank page full of question marks and tries to guess all the words at once, refining its guesses over and over until the story makes sense.

The problem the paper identifies is that when we teach this robot, we have to give it practice examples. To make the practice efficient, we take short stories and pad them with "dummy words" so they all fit on the same size page.

The Problem: The Confused "Stop" Sign

In the past, researchers used a special token called [EOS] (End of Sentence) as that "dummy word."

Its Job 1: To tell the robot, "Stop writing here, the story is done."
Its Job 2: To fill up empty space on the page so the robot can practice.

The paper argues that giving the [EOS] token these two jobs is like asking a traffic light to be both a Stop sign and a construction barrier.

When the robot is practicing, it learns that [EOS] means "fill this space." But when it's actually writing a real story, it sees [EOS] and gets confused. It thinks, "Oh, I need to fill this space with more [EOS]s!" This causes a glitch called [EOS] overflow. The robot writes a tiny bit of text and then just spam-fills the rest of the page with "End of Sentence" markers, cutting the story short.

This gets worse when the robot tries to write long stories in big chunks (large blocks). The "Stop" sign gets so crowded with "dummy" signs that the robot can't tell where the real story ends.

The Solution: VoidPadding

The authors propose a simple fix called VoidPadding. They introduce a new, invisible token called [VOID].

Think of it like this:

[VOID] is the new "dummy word." It fills the empty space on the practice page, but it has no meaning. It's like a ghost that takes up space but doesn't speak.
[EOS] is now only the "Stop" sign. It is reserved strictly for telling the robot when the story is actually finished.

By separating these roles, the robot learns clearly:

[VOID] = "This is just empty space, ignore it."
[EOS] = "The story is done, stop now."

How It Works in Practice

The paper describes two clever tricks that happen during the "inference" (when the robot is actually doing the work):

The "Ghost" Ban: When the robot is generating a story, it is strictly forbidden from writing [VOID]. If it tries to guess [VOID], the system says "No, that's just a ghost." This prevents the robot from accidentally filling its story with invisible ghosts.
The Early Stop: Because [EOS] is now a pure "Stop" sign, the robot can listen to it. As soon as it sees a clear [EOS] signal, it stops writing immediately. It doesn't waste time filling out the rest of the page with garbage. This makes the robot much faster.
The Canvas Expander: Sometimes the robot starts with a page that is too small for the story it wants to tell. The paper introduces a trick called VoidExpansion. The robot looks at the "ghost" signals ([VOID]) it learned during training. If the page is too short, the [VOID] signals will look "uncomfortable" (like a ghost trying to squeeze into a tiny box). The robot senses this, expands the page to a bigger size, and then finishes the story.

The Results

The authors tested this on two different robot models (LLaDA and Dream) using math and coding puzzles.

Better Quality: When the robots tried to write long answers, the old method often failed and just spat out "End of Sentence" over and over. The new method (VoidPadding) solved this, keeping the answers long and correct.
Faster: Because the robots stopped exactly when they were done (instead of filling the whole page), they finished tasks 55% faster on average.
Robust: The new method worked well whether the robot was writing short answers or very long ones, and whether it was working in small chunks or big chunks.

Summary

The paper shows that by giving the "Stop" sign and the "Empty Space" filler two different names, we can stop the robot from getting confused. This makes the robot write better, stop at the right time, and work much faster. It's a small change in how we teach the robot that leads to a big improvement in how it performs.

Technical Summary: VoidPadding

Problem: [EOS] Overflow in Masked Diffusion Language Models

Masked Diffusion Language Models (MDLMs) generate text by denoising a pre-allocated, fully masked response canvas in parallel, utilizing bidirectional attention. This architecture contrasts with autoregressive models (ARLMs), which generate tokens sequentially. A critical challenge arises during instruction tuning (IT) for MDLMs: to facilitate efficient batching, prompt-response pairs are padded to a uniform sequence length. Following conventions inherited from ARLM training, existing MDLMs typically use repeated [EOS] (End-of-Sequence) tokens as padding.

This practice creates a training-inference role mismatch that leads to [EOS] overflow, particularly under large-block decoding:

Dual Role of [EOS]: In training, [EOS] serves a dual function: it acts as a semantic terminator for the actual response and as a padding token for the remainder of the sequence. Consequently, the model learns to associate high [EOS] probabilities with the tail of the sequence, effectively treating [EOS] as a "length signal" indicating where the response should end.
Inference Failure: During inference, MDLMs operate on a fixed canvas. If the model commits to high-confidence [EOS] tokens at the tail of the canvas (based on the learned length signal), these committed tokens become part of the bidirectional context for subsequent denoising steps. This causes the model to propagate [EOS] probabilities leftward, leading to premature termination and repeated [EOS] tokens, even when the semantic content is incomplete. This phenomenon, termed [EOS] overflow, degrades generation quality and limits the effectiveness of large-block decoding strategies.

Methodology: VoidPadding

The authors propose VoidPadding, a training-inference co-design that decouples the semantic termination role from the padding role.

1. Training Design: Introducing `[VOID]`

Instead of padding with [EOS], the training data is formatted as [prompt] [response] [EOS] [VOID]*.

[EOS]: Reserved exclusively as the semantic terminator. It appears exactly once per example, immediately following the response.
[VOID]: Introduced as a new token type used solely for padding the remainder of the sequence.
Training Objective: The model is trained to predict the original token at masked positions. Crucially, the single [EOS] token is always masked during training, forcing the model to learn [EOS] strictly as a stable semantic terminator rather than a padding artifact. The [VOID] tokens are also masked, teaching the model to associate [VOID] with the "unused" portion of the canvas.

2. Inference Design: Decoupled Stopping and Expansion

VoidPadding introduces two complementary mechanisms for inference:

[EOS]Termination (Early Stopping):
- Standard MDLM stopping rules (VanillaStopping) require decoding the entire canvas or until all masks are filled, often leading to redundant generation after the semantic endpoint.
- VoidPadding implements [EOS]Termination: decoding stops as soon as all positions to the left of the leftmost visible [EOS] are unmasked. The model discards the remaining canvas (including any [VOID] padding).
- To prevent [VOID] from appearing in the output, its logits are set to $-\infty$ during token selection, ensuring it remains a hidden signal.
VoidExpansion (Adaptive Canvas Allocation):
- MDLMs require a pre-allocated canvas. If the initial canvas is too short, generation quality drops.
- VoidExpansion leverages the learned [VOID] length signal. Before decoding, the model evaluates the tail of the current canvas. If the tail positions show a strong preference for non-[VOID] tokens (indicating the canvas is insufficient), the canvas is expanded by one block size ( $B$ ).
- This process repeats until the tail preference shifts to [VOID] (indicating sufficient length) or a maximum length is reached. This allows the model to adaptively allocate canvas space without relying on fragile [EOS] signals.

Key Contributions

Mechanism Analysis: The paper systematically identifies the root cause of [EOS] overflow as the dual use of [EOS] for both padding and termination during instruction tuning, which creates a learned length signal that conflicts with the bidirectional context of inference.
VoidPadding Framework: The proposal of a lightweight training-inference co-design that introduces [VOID] for padding and reserves [EOS] for termination. This enables reliable early stopping ([EOS]Termination) and adaptive canvas expansion (VoidExpansion).
Empirical Validation: Extensive experiments demonstrate that VoidPadding prevents [EOS] overflow, significantly improves generation quality across mathematical reasoning and code generation benchmarks, and reduces decoding computational cost (NFE) by enabling early termination.

Results

Experiments were conducted on LLaDA-8B-Instruct and Dream-7B-Instruct across four benchmarks: GSM8K, MATH500, HumanEval, and MBPP.

Robustness to Block Size:
- On Dream-7B-Instruct, VoidPadding improved the block-size-averaged four-task mean score by +17.84 points over the original model and +6.95 points over the previous state-of-the-art method, RainbowPadding.
- On LLaDA-8B-Instruct, the improvement was +7.06 points over the original model and +3.59 points over RainbowPadding.
- Crucially, while original models suffered severe performance degradation as block size increased (e.g., dropping from ~~76% to ~28% on GSM8K for LLaDA at block size 512), VoidPadding maintained stable performance (~~78%).
Decoding Efficiency:
- By utilizing [EOS]Termination, VoidPadding significantly reduced the Number of Forward Evaluations (NFE). On Dream-7B-Instruct, NFE was reduced by an average of 55.7%, with specific reductions of 69.3% on GSM8K and 84.6% on MBPP compared to the standard 512-step requirement.
Variable-Length Generation:
- The VoidExpansion strategy allowed the model to handle short initial canvases (e.g., $L_0=64$ ) effectively. On LLaDA, the full VoidPadding + VoidExpansion method achieved a mean score of 51.34, outperforming inference-only baselines like DAEDAL (50.77) and $\rho$ -[EOS] (45.50), while also running 2.12 $\times$ faster in wall-clock time.
- Under large-block stress tests ( $L_0=256, B=256$ ), inference-only methods relying on the original [EOS] signal collapsed (DAEDAL dropped to 23.46 mean), whereas VoidExpansion maintained a robust 50.04 mean score.

Significance and Claims

The paper claims that [EOS] overflow is not a failure of model capacity but a structural failure arising from the interaction between learned length priors, bidirectional context, and standard sampling rules. By decoupling padding from termination, VoidPadding resolves this mismatch.

The authors position VoidPadding as a necessary adaptation for MDLMs, arguing that conventions inherited from ARLMs (such as using [EOS] for padding) are "largely benign" under causal masking but become "fragile" under bidirectional denoising. The method offers a practical path to unlocking the full potential of large-block decoding in diffusion models, improving both the quality of complex reasoning outputs and the efficiency of the generation process. The paper concludes that future MDLM designs must explicitly consider token commitment timing, canvas allocation, and stopping decisions under bidirectional conditions, rather than relying on autoregressive assumptions.