Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

1. Problem Statement

In modern autoregressive models (LLMs and VLMs), the generation length is a fundamental unit of computation that directly impacts inference cost (compute, memory, latency) and reasoning performance. While existing methods attempt to control or predict generation length, they suffer from significant limitations:

• Coarse-Grained Control: Most approaches operate at the sequence level (e.g., sequence-level penalties during training or prompt-based instructions at inference) rather than modeling the dynamics of generation at the token level.
• Lack of Fine-Grained Signals: There is a missing mechanism to estimate the remaining generation horizon at every decoding step.
• Annotation Bottlenecks: Traditional reward modeling for length often requires human labeling or complex reward models, which do not scale well.
• Trade-off Rigidity: Hard constraints (e.g., "stop at 200 tokens") often degrade performance, while current methods lack a smooth mechanism to trade off between reasoning quality and token budget.

2. Methodology: The Length Value Model (LenVM)

The authors propose LenVM, a token-level framework that models remaining generation length as a value estimation problem within a standard Reinforcement Learning (RL) value-function framework.

Core Formulation

Instead of predicting raw token counts directly, LenVM formulates length modeling as a discounted return problem:

• Reward Structure: A constant negative reward is assigned to every generated token: $r_t = -(1 - \gamma)$, where $\gamma \in (0, 1)$ is a discount factor. The terminal state (EOS) has a reward of 0.
• Discounted Return: The target value $G_t$ at step $t$ for a trajectory of total length $L$ is defined as the sum of future discounted rewards:
  $$G_t = \sum_{i=0}^{L-t-1} \gamma^i r_{t+i} = -(1 - \gamma^{L-t})$$
• Properties:
  • Bounded: $G_t \in (-1, 0)$, avoiding the unbounded dynamic range of raw lengths.
  • Monotone: Shorter remaining horizons yield values closer to 0; longer horizons yield values closer to -1.
  • Bellman-Consistent: Satisfies the recursion $G_t = r_t + \gamma G_{t+1}$.

Architecture and Training

• Head: LenVM attaches a scalar value head (a 2-layer MLP with SiLU activation) to the final-layer hidden state of a base LLM or VLM.
• Output: The head outputs a value $V_\theta(s_t) = -\sigma(z_t)$, ensuring the output remains in the range $(-1, 0)$.
• Training Objective: The model is trained via annotation-free, dense, token-level regression.
  • Data Generation: For a given prompt, multiple completions are sampled from a fixed rollout policy.
  • Supervision: The realized return $G_t$ is computed deterministically from the sampled completion length.
  • Loss: Token-averaged Mean Squared Error (MSE) between the predicted value and the realized return.
  • Scalability: The supervision signal is dense (every token is a target), unbiased (Monte Carlo sample under fixed policy), and scales naturally with model size, prompt count, and number of sampled completions.

3. Key Contributions

1. Token-Level Length Modeling: LenVM is the first framework to model remaining generation length as a continuous, token-level value signal, enabling fine-grained control and prediction.
2. Annotation-Free Scalability: The training signal is derived entirely from sampled completions without human labeling or external reward models, making it highly scalable across model sizes and datasets.
3. Inference-Time Control: LenVM provides a mechanism to control generation length without modifying the base model weights, supporting both hard constraints and continuous trade-offs.
4. Interpretability: The token-level values reveal "length tokens" that signal shifts in reasoning regimes (e.g., tokens like "wait" or "ah" indicating a shift to longer reasoning, vs. "therefore" indicating closure).

4. Experimental Results

The authors evaluated LenVM on LLMs (Qwen2.5, Qwen3) and VLMs across math, code, and instruction-following domains.

• Length-Controlled Generation (LIFEBench):
  • LenVM significantly outperforms frontier closed-source models (GPT-4o, Claude, Gemini) in exact length matching.
  • On a 7B model, the Length Score improved from 30.9 to 64.8 on the "Equal To" constraint, while reducing length deviation from 71% to 44%.
• Performance–Efficiency Trade-off:
  • Using exponential tilting based on LenVM's predicted values, the model can continuously balance accuracy and token budget.
  • On GSM8K with a 200-token budget, LenVM maintained 63% accuracy, whereas a hard token-budget baseline achieved only 6%. This demonstrates that the base model contains shorter successful reasoning paths that LenVM can uncover.
• Generation Length Prediction:
  • LenVM can predict the expected generation horizon from the prompt boundary (before any tokens are generated).
  • At 32B scale, it achieves a Mean Relative Error (MRE) of 9.8% for math, 14.9% for code, and 17.1% for instruction following.
• Scalability:
  • Validation loss decreases consistently as model size, number of training prompts, and number of sampled completions per prompt increase.
• Case Study (Length Tokens):
  • Analysis of temporal-difference residuals reveals specific tokens that act as markers for length shifts (e.g., "ah", "but" for extending reasoning; "therefore", "perfect" for terminating).

5. Significance and Future Implications

• General Framework: LenVM establishes that generation length can be effectively modeled as a token-level value signal, providing a principled alternative to coarse-grained penalties.
• RL Foundation: The paper highlights that LenVM serves as a natural, length-specific value baseline for PPO-style RL. It can be used either as a direct objective component (to optimize efficiency) or as a potential-based reward shaping signal (to improve credit assignment without changing the task objective).
• Practical Utility: The ability to predict length and control generation dynamically offers immediate benefits for inference scheduling, memory management, and cost reduction in production LLM systems.
• Interpretability: By visualizing how specific tokens shift the predicted horizon, LenVM offers a new lens into the internal dynamics of reasoning models.

In summary, LenVM transforms the problem of length control from a sequence-level constraint into a dense, scalable, token-level value estimation task, enabling precise control, efficient inference, and deeper interpretability of autoregressive generation.