Technical Summary: Focus on the Core (FoCore)

1. Problem Statement

Diffusion Large Language Models (DLMs) offer a distinct advantage over autoregressive (AR) models by enabling global context modeling through an iterative denoising paradigm. However, current decoding strategies often fail to leverage this capability, exhibiting a "locally-biased" preference where tokens closer to already-denoised positions receive disproportionately higher confidence scores. This local bias neglects the heterogeneous information density inherent in natural language.

The paper argues that natural language contexts contain a mix of:

• Low-Information-Density (LD) tokens: Redundant background padding (e.g., structural words like "for," "she").
• High-Information-Density (HD) tokens: Core reasoning units or logical anchors (e.g., entities, numbers, specific constraints) that steer the generation trajectory.

Current strategies often overlook these HD tokens, leading to degraded generation quality, particularly in complex reasoning tasks like math and code generation. Furthermore, standard decoding processes do not fully exploit the observation that HD tokens tend to converge (stabilize) earlier than surrounding tokens, missing opportunities for inference acceleration.

2. Methodology

The paper proposes FoCore (Focus on the Core), a training-free decoding strategy, and its efficient variant FoCore_A. The methodology is built upon three key empirical findings regarding HD tokens:

1. Explicit conditioning on HD tokens substantially improves output quality.
2. HD tokens exhibit an early-decoding tendency, converging to their final states earlier than surrounding tokens.
3. HD tokens are highly sensitive to global context masking, suffering significant confidence drops when their context is obscured, unlike robust LD tokens.

A. Identification of HD Tokens

To avoid domain-specific heuristics (which fail in code generation), the authors propose an automated identification mechanism based on Distribution Stability:

• Metric: They utilize a Top-K truncated Jensen-Shannon (JS) divergence to measure the instability of a token's predicted probability distribution between consecutive decoding steps ($t$ and $t-1$).
• Logic: HD tokens induce significant distributional shifts across iterations, whereas LD tokens maintain stable distributions.
• Refinement: An Exponential Moving Average (EMA) is applied to the step-wise divergence scores to track historical instability, filtering out transient fluctuations and identifying tokens with high cumulative instability ($S^{(i)}_t$).

B. The FoCore Mechanism (Self-Contrast)

FoCore utilizes identified HD tokens in a self-contrast manner to guide generation without retraining:

1. Negative Sample Construction: At each iteration, the top-$K$ most unstable (converged) HD tokens are temporarily remasked (replaced with [MASK]) to create an "unconditional" input sequence ($\tilde{X}_t$).
2. Contrastive Guidance: The model performs two forward passes: one on the original sequence ($X_t$) and one on the remasked sequence ($\tilde{X}_t$).
3. Logit Adjustment: The final logits are computed by contrasting the conditional and unconditional outputs:
   $$\hat{L}_t = L_{cond} + \omega \cdot (L_{cond} - L_{uncond})$$
   This forces the model to strictly anchor its attention on the core HD tokens by penalizing deviations from the trajectory where these critical anchors are present.

C. FoCore_A (Acceleration Variant)

Leveraging the finding that HD tokens converge early, FoCore_A introduces a parallel decoding mechanism:

• Monitoring: It continuously monitors the mean instability score of undecoded positions.
• Early Exit: Once the mean instability drops below a threshold ($\tau$), indicating that the core HD tokens have sufficiently converged, the system triggers parallel decoding.
• Execution: It decodes $m$ additional high-confidence tokens (typically LD tokens in the local context window) in a single step, significantly reducing the total number of forward passes and decoding steps.

3. Key Contributions

1. Empirical Analysis of HD Tokens: The paper systematically investigates the behavioral properties of HD tokens, revealing their dual benefits: they are critical for semantic guidance (quality) and exhibit early convergence (efficiency).
2. FoCore Decoding Strategy: A training-free, self-contrastive method that dynamically remasks converged HD tokens as negative samples to enhance generation quality.
3. FoCore_A Acceleration: An efficient variant that exploits the early-decoding tendency of HD tokens to enable parallel decoding, substantially reducing latency.
4. Comprehensive Evaluation: Extensive experiments across math, code, and logical reasoning benchmarks demonstrate consistent improvements over state-of-the-art baselines.

4. Experimental Results

The methods were evaluated on LLaDA 8B and Dream 7B backbones across six benchmarks: GSM8K, MATH500, HumanEval, MBPP, SVAMP, and Countdown.

• Quality Improvements:

  • On HumanEval, FoCore improved pass@1 from 39.02 (Standard CFG) to 42.68.
  • On MBPP, FoCore achieved scores of 38.40/40.40/40.00 across generation lengths, outperforming competitors like Prophet and KLASS.
  • On MATH500, FoCore reached 43.80, surpassing the best baseline (Confidence) at 42.60.
  • The method showed monotonic scaling with generation length (128 to 512), whereas many baselines exhibited performance drops at longer lengths.

• Efficiency Gains (FoCore_A):

  • Decoding Steps: Reduced by 2.07× on HumanEval compared to standard CFG.
  • Latency: Per-sample latency on HumanEval dropped from 20.76s to 8.64s (a 58.4% reduction).
  • Speedup: Achieved speedups ranging from 1.45× to 2.29× across benchmarks while maintaining or improving task accuracy compared to other efficient methods like Prophet and KLASS.

5. Significance and Claims

The paper claims that Focus on the Core addresses a fundamental bottleneck in DLM decoding: the failure to distinguish between redundant background information and critical reasoning anchors.

• Theoretical Insight: The work provides a theoretical framework linking conditional entropy to distributional instability, proving that HD tokens act as information bottlenecks that dominate the decoding trajectory. By compressing the uncertainty of these tokens via self-contrast, FoCore stabilizes the generation path.
• Practical Impact: The proposed framework offers a training-free solution that simultaneously enhances reasoning quality and inference speed. It demonstrates that DLMs can be made more efficient not just by architectural changes, but by smarter decoding strategies that respect the heterogeneous nature of information density in text.
• Future Direction: The authors suggest that incorporating this "focus-on-core" awareness into the parameter-level training (e.g., via RL-based post-training) is a promising future direction, though their current work focuses strictly on inference-time decoding.