💬 NLP

Just on Time: Token-Level Early Stopping for Diffusion Language Models

This paper introduces a training-free, token-level early stopping method for diffusion language models that dynamically identifies and finalizes stable tokens based on lightweight prediction signals, significantly reducing computational costs while maintaining generation quality across diverse tasks.

Original authors: Zakhar Kohut, Severyn Shykula, Mykola Vysotskyi, Serhii Dmytryshyn, Dmytro Khamula, Michal Zakrzewski, Damian Rynczak, Jacek Małecki, Taras Rumezhak, Volodymyr Karpiv

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zakhar Kohut, Severyn Shykula, Mykola Vysotskyi, Serhii Dmytryshyn, Dmytro Khamula, Michal Zakrzewski, Damian Rynczak, Jacek Małecki, Taras Rumezhak, Volodymyr Karpiv

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant jigsaw puzzle, but instead of looking at the picture on the box, you start with a box full of blank, gray squares. To solve it, you have to guess what color each square should be, then look at the whole picture, realize some guesses were wrong, and repaint them. You repeat this "guess-and-fix" cycle hundreds of times until the picture finally looks right. This is how a new kind of AI, called a Diffusion Language Model, writes text. Unlike older AI that writes one word at a time like a typist, these models start with a blank page and refine the whole sentence at once, like a painter slowly adding details to a canvas.

The problem is that this painting process is incredibly slow. Even though the AI often figures out the easy parts of the sentence (like "The" or "cat") after just a few brushstrokes, it keeps painting over them anyway, just to be safe. It's like a chef who tastes a soup, realizes it's perfect, but then keeps stirring and tasting it for another hour before serving it. This wastes a massive amount of computer power and time. The question scientists are asking is: How can we tell the AI, "Hey, you've got this part right, stop working on it and move on to the hard stuff"?

This is where a new method called JoT (Just on Time) comes in. Think of JoT as a smart supervisor standing over the AI painter. Instead of telling the whole team to stop painting at the same time, JoT watches each individual word on the canvas. As soon as the AI is super confident that a specific word is correct—say, it's 99% sure the word is "cat"—JoT shouts, "Freeze that word! Don't touch it anymore!" and moves the AI's attention to the next blurry, uncertain word.

The researchers behind JoT found that this "token-level early stopping" works wonders. They tested it on two powerful AI models, Dream-7B and LLaDA-8B, across four different challenges: solving math problems, answering trivia, finishing sentences, and writing code. The results were striking. On a math reasoning test called GSM8K, JoT made the AI 5.5 times faster while only dropping the score by a tiny bit (about 2.3 points). On a coding challenge called HumanEval, the speedup was even more dramatic, reaching nearly 20 times faster (19.6×), with the AI still getting almost all the answers right.

The paper argues against a "one-size-fits-all" approach where the AI stops all words at the same time. Instead, JoT uses a clever trick: it lowers the confidence bar for words that are next to words the AI has already finished. If the AI has already nailed the beginning of a sentence, it can trust the next word a little sooner. This allows the AI to focus its energy only on the parts of the sentence that are truly confusing, skipping the redundant work on the easy parts.

While the method is incredibly fast, the authors are careful to note that it's not magic. They measured that the AI still makes a few mistakes, mostly when it gets stuck on a tricky math step and locks in the wrong number too early. However, these errors are rare, and the trade-off is heavily in favor of speed. The paper suggests that by letting each word "exit" the process at its own perfect moment, we can make these powerful, slow AI models much more practical for everyday use without needing to retrain them or change their core design. It's a simple, training-free fix that lets the AI work smarter, not harder.

Technical Summary: Just on Time (JoT) for Diffusion Language Models

Problem Statement

Diffusion Language Models (DLMs) generate text by iteratively refining a fully masked sequence into a clean output. While this approach enables parallel token prediction and bidirectional context integration, it suffers from decoding inefficiency. Standard DLMs require a fixed number of refinement steps ( $T$ ) for the entire sequence, even though many tokens often reach stable, high-confidence predictions well before the final step. This leads to unnecessary computational overhead, as the model continues to refine tokens that have already converged. Existing early-stopping methods often apply global criteria or rely on training-specific adaptations, failing to dynamically address the varying convergence rates of individual tokens within a single generation.

Methodology: JoT (Just on Time)

The authors propose JoT, a training-free, per-token early stopping mechanism designed to identify convergence independently at each position within the sequence. JoT allows different tokens to "exit" the iterative refinement process at different steps, concentrating computation only where it is needed.

Core Components

Confidence Metric:
JoT quantifies prediction certainty at each masked position $i$ using the ratio of the top two predicted probabilities ( $p_1$ and $p_2$ ):
$r_i = \frac{p_1}{p_2 + \epsilon}$
This metric is derived from unscaled logits, making it invariant to sampling temperature. A high ratio indicates a decisive preference for the top candidate, while a ratio near 1 suggests uncertainty.
Spatial Modulation:
Recognizing that tokens adjacent to already-resolved (unmasked) context benefit from richer local information and lower effective noise, JoT introduces a spatially adaptive threshold.
- A geometric kernel computes a weight $w_i$ based on the proximity of position $i$ to unmasked tokens within a window $D$ .
- This weight is normalized to a softening factor $\phi_i \in [0, 1]$ .
- The acceptance threshold $\tau_i$ for position $i$ is interpolated between a maximum threshold $\tau_{max}$ and a minimum $\tau_{min}$ :
  $\tau_i = \tau_{max} - (\tau_{max} - \tau_{min}) \cdot \phi_i$
- Tokens near resolved context receive lower thresholds (easier to finalize), while tokens in the center of masked regions retain stricter thresholds.
Early-Exit Decision:
At each step, if a masked token's confidence ratio $r_i$ exceeds its adaptive threshold $\tau_i$ , the token is finalized (selected via argmax or sampling) and removed from the set of masked positions. The process continues until all tokens are finalized or the maximum step count is reached.

Theoretical Analysis

The paper provides a formal analysis (Proposition 1) regarding the probability of "leading-pair reversals"—instances where a committed token is later overtaken by a runner-up as more context is revealed. Under the assumption that the model's prediction matches the true conditional distribution (calibration), the probability of such a reversal is bounded by $2/(1+\tau)$ . The authors note that residual errors at high thresholds may stem from calibration failures rather than the stopping mechanism itself.

Key Contributions

JoT Framework: A training-free, per-token early stopping method that adapts acceptance thresholds based on spatial proximity to resolved context.
Comprehensive Evaluation: Experiments conducted on two state-of-the-art models (Dream-7B and LLaDA-8B) across four diverse benchmarks: GSM8K (math), MMLU (knowledge), HellaSwag (commonsense), and HumanEval (code).
Ablation Studies: Detailed analysis isolating the effects of threshold selection and spatial modulation, providing practical guidance for hyperparameter configuration.

Experimental Results

The authors evaluated JoT against full decoding and other early-stopping baselines (Prophet, KLASS, Fast-dLLM).

Speed-Quality Trade-off:
- Dream-7B: JoT achieved up to 5.54× speedup on GSM8K and 19.60× speedup on HumanEval. Accuracy drops were modest (e.g., -2.3 points on GSM8K, -0.6 on HumanEval), with performance remaining within ~3 percentage points of full decoding on most benchmarks.
- LLaDA-8B: JoT maintained accuracy within 3.1 points of full decoding across all benchmarks while providing competitive speedups (e.g., 3.75× on GSM8K).
Comparison: JoT generally outperformed Prophet (which suffered larger accuracy drops) and offered better speed-accuracy Pareto frontiers than KLASS and Fast-dLLM on several tasks.
Composability: JoT was successfully integrated with Fast-dLLM's DualCache (which reduces per-step latency), demonstrating that step reduction and per-step optimization are orthogonal and can be combined.
Failure Modes: Analysis of 500 GSM8K samples revealed that errors primarily occurred when the model prematurely committed to a token at a critical reasoning branch (e.g., choosing "× 2" instead of "/ 2"). These errors were not novel failure modes but occurred at different rates compared to full decoding.

Significance and Claims

The paper claims that JoT offers a compelling solution to the decoding inefficiency of diffusion language models without requiring retraining or task-specific fine-tuning. By dynamically finalizing tokens as soon as they exhibit sufficient confidence, JoT substantially reduces the total number of diffusion steps required.

The authors position JoT as a method that achieves the best speed-quality trade-off among tested early-stopping methods on Dream-7B and remains competitive on LLaDA-8B. They emphasize that the method is particularly effective because it allows the model to skip redundant refinement of "obvious" context tokens while preserving the computational effort required for complex reasoning phases. The paper concludes that while optimal thresholds may vary by task, the method's robustness across hyperparameters makes it practical for deployment across diverse applications.