💬 NLP

Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models

This paper proposes a reinforcement learning framework that treats the classifier-free guidance scale in diffusion language models as a dynamic, learnable control action rather than a fixed hyperparameter, demonstrating that adaptive guidance trajectories significantly improve the trade-off between controllability and generation quality across various NLP tasks.

Original authors: Fan Zhou, Tim Van de Cruys

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fan Zhou, Tim Van de Cruys

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a play where the actors (the AI) are trying to memorize a script line by line. They start with a blank page full of question marks, and with every step, they fill in a few more words until the whole sentence is complete.

In the world of AI text generation, there's a popular tool called Classifier-Free Guidance (CFG). Think of CFG as a director's megaphone. When the director shouts "Focus on the keywords!" or "Make it sound sad!", the megaphone amplifies those instructions so the actors pay attention to them.

The Problem: The "One-Size-Fits-All" Megaphone
Until now, directors have treated this megaphone like a fixed volume knob. They decide, "Okay, we'll set the volume to 7 out of 10 for the entire play," and they never touch it again.

The authors of this paper argue that this is a bad idea. Just like a real play, the needs change as the story progresses:

Early in the play: You need a loud megaphone to make sure the actors get the basic structure right (e.g., "Start with a ship," "Include these 10 words").
Later in the play: If you keep the megaphone at full volume, the actors might get too stressed, start shouting, and ruin the flow of the dialogue. You need to turn the volume down to let the story breathe and sound natural.

Using the same loudness level from start to finish forces a trade-off: either the story follows the rules but sounds robotic, or it sounds natural but ignores the rules.

The Solution: A Smart, Learning Director
The paper proposes a new approach: Don't treat the megaphone volume as a fixed setting; treat it as a skill to be learned.

Instead of a human manually deciding the volume, the authors trained an AI "assistant director" using a method called Reinforcement Learning (think of it as training a dog with treats).

The Game: The assistant director watches the play unfold step-by-step.
The Action: At every stage, the assistant decides whether to shout loudly (high guidance) or whisper softly (low guidance).
The Reward: At the very end of the play, the assistant gets a "treat" (a score) based on two things: Did the actors follow the rules? And did the play sound good?
The Learning: Over many rehearsals, the assistant learns a specific "script" for how to adjust the volume. It learns that for a "keyword" play, it should shout early and whisper late. For a "sentiment" play (making text sad or happy), it might need to shout right at the beginning to set the mood, then fade out.

What They Found
The researchers tested this on three different types of "plays":

Keyword Generation: Writing a sentence that must include 10 specific words.
Length Control: Rewriting a sentence to be shorter (40-80% of the original).
Sentiment Transfer: Changing a happy sentence to a sad one (or vice versa) without losing the meaning.

The Results:

The Old Way (Fixed Volume): The AI either followed the rules poorly or sounded terrible.
The New Way (Learning Director): The AI found a "sweet spot" for every task. It learned to be strict when the structure was being built and gentle when the details were being polished.
- For keywords and length, the AI learned to start loud and get quieter (a "hump" shape).
- For sentiment, it learned to start very loud to lock in the emotion, then get quieter immediately.

The Bottom Line
The paper proves that the "volume knob" for AI text generation shouldn't be a static number you pick before you start. It should be a dynamic tool that changes as the text is being written. By teaching the AI to learn when to be strict and when to be flexible, they achieved text that is both obedient to instructions and natural to read.

In short: They stopped treating the AI's control settings like a thermostat (set it once and forget it) and started treating it like a conductor (adjusting the orchestra's volume in real-time to make the music sound perfect).

Technical Summary: Guidance is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models

Problem Statement

Classifier-Free Guidance (CFG) is the standard mechanism for controlling diffusion-based generative models, yet its guidance scale ( $\gamma$ ) is typically treated as a fixed hyperparameter throughout the entire generation process. This static design creates a sub-optimal tradeoff between controllability (adherence to conditions) and generation quality (fluency). The authors argue that the optimal degree of guidance varies across different tasks and, crucially, across different stages of the diffusion process. In Natural Language Processing (NLP), where discrete diffusion language models (dLLMs) are emerging, applying a single, globally fixed scale introduces a structural mismatch between the stage-wise requirements of generation and the control mechanism.

Existing attempts to address this have relied on heuristic, pre-defined schedules (e.g., exponential curves, Beta distributions) that are independent of the evolving generation state or task-level objectives. These methods lack a systematic mechanism for incorporating task-level feedback during generation, assuming that a single hand-crafted schedule can generalize across diverse inputs and controllability requirements.

Methodology

The paper proposes recasting CFG scale selection as a sequential decision-making problem solved via Reinforcement Learning (RL). Instead of manually tuning a schedule, the authors learn a policy that dynamically selects the guidance scale at each generation step based on the current diffusion state.

Core Framework

Policy Formulation: The guidance selection is modeled as a Markov Decision Process (MDP). A policy $\pi_\phi(a|s)$ maps the current sampling state $s$ (including step ratio, mask ratio, task-specific progress, and previous action) to a guidance action $a$ (the CFG scale).
Sparse Terminal Rewards: Since downstream task objectives (e.g., keyword inclusion, sentiment accuracy) are only observable upon completion and are non-differentiable, the authors employ a sparse reward design. Rewards are assigned only at the final diffusion step ( $k=0$ ), defined as a weighted combination of controllability metrics and generation quality (measured via perplexity).
Action Repetition & Temporal Abstraction: To mitigate the high variance associated with sparse rewards over long horizons, the policy selects a single guidance action that is held constant for $n$ consecutive diffusion steps. This reduces the effective decision horizon and encourages smooth trajectories.
Discrete Action Space: To ensure tractability and stability, the action space is restricted to a discrete set of 13 CFG scales ranging from 0 to 3.0.
Optimization: The policy is optimized using Proximal Policy Optimization (PPO), which is well-suited for stochastic dynamics and delayed rewards. The underlying generative model is a discrete diffusion language model (specifically LLaDA), which remains frozen; only the lightweight RL policy network is trained.

Trajectory Aggregation

Post-training, the policy induces a distribution over guidance trajectories rather than a single deterministic schedule. To derive a task-generalized strategy for inference, the authors propose aggregating sampled trajectories. They compare a simple mean trajectory against a frequency-weighted aggregation, which emphasizes guidance patterns consistently selected by the policy, thereby filtering out exploratory noise.

Key Contributions

Reformulation of Guidance: The paper shifts the paradigm of CFG from a static hyperparameter to a dynamic, learnable control process.
RL-Based Adaptive Control: It introduces a principled framework using PPO to learn task-specific guidance trajectories that adapt to the evolving generation state without modifying the pretrained model parameters.
Task-Dependent Insights: The study reveals that optimal guidance behaviors are fundamentally task-dependent. For structural constraints (keywords, length), the learned policy adopts a "hump-shaped" trajectory (increasing then decreasing), whereas for stylistic control (sentiment), it learns a monotonically decreasing trajectory.
Empirical Validation: The approach is validated on three distinct NLP tasks: keyword-conditioned generation, length-controlled rewriting, and sentiment style transfer.

Experimental Results

Experiments were conducted on discrete diffusion language models (LLaDA) using 30 and 60 diffusion sampling steps. The proposed method (RL-Mean and RL-Freq) was compared against fixed CFG and various heuristic schedules (linear, cosine, Beta).

Performance Gains: The RL-learned schedules consistently outperformed both fixed and heuristic baselines across all tasks.
- Keyword Generation: Achieved higher keyword coverage (74.6% vs. 71.4% for fixed CFG) with significantly lower perplexity (56.2 vs. 61.3).
- Length Control: Demonstrated a Pareto improvement, achieving 92.8% accuracy (vs. 76.0% for fixed CFG) and the lowest perplexity (205.6 vs. 301.9).
- Sentiment Transfer: Improved accuracy in the challenging negative-to-positive direction (40.6% vs. 32.8%) while maintaining high content preservation and lower perplexity.
Breaking the Trade-off: While heuristic schedules exhibited a clear trade-off (e.g., increasing schedules improved controllability but degraded fluency), the RL approach simultaneously improved both metrics by learning when to apply strong versus weak guidance.
Qualitative Analysis: Case studies showed that adaptive guidance produced more coherent sentences with fewer repetitions and better semantic preservation compared to static or heuristic schedules.

Significance and Claims

The paper claims that treating guidance as a dynamic control process rather than a static design choice is essential for optimizing the controllability–quality tradeoff in diffusion language models. The learned trajectories provide interpretable insights into how guidance should evolve: structural tasks benefit from strong guidance during the coarse structure-forming phase, while stylistic tasks require early strong guidance to establish polarity, followed by a reduction to ensure fluency.

The authors maintain a modest stance regarding limitations. They acknowledge that while adaptive scheduling significantly improves performance within the discrete diffusion paradigm, a gap remains compared to autoregressive large language models due to the latter's stronger language modeling priors. The guidance mechanism can mitigate but not fully compensate for inherent limitations of current diffusion models, such as token repetition under strong conditioning. The work does not claim to solve all issues in controllable generation but offers a flexible, task-adaptive framework that outperforms current static and heuristic strategies.