LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

1. Problem Statement

Despite the advanced capabilities of Large Language Models (LLMs) in complex instruction following, precise control over output length remains a persistent bottleneck.

• The Core Issue: LLMs struggle to adhere to specific length constraints (e.g., "write exactly 100 words"). This is attributed to the token-based nature of tokenization and intrinsic architectural limitations that prevent models from forming a distinct internal concept of "length."
• The Cognition-Action Gap: Existing methods fail because they treat length as an external constraint rather than an internal semantic feature.
  • External Markers: Methods using special tokens or rigid prompts often fail to generalize to long-form generation and add inference overhead.
  • Policy Optimization: Reinforcement Learning (RL) approaches often entangle length constraints with semantic content. Because policy updates are driven by dense token-level dependencies, the model struggles to decouple "how much to write" from "what to write," leading to low sample efficiency and imprecise control.
• Hypothesis: The fundamental limitation is a lack of intrinsic length cognition. Without an internal representation of length, training models to follow length constraints is an ill-posed optimization problem.

2. Methodology: LARFT Framework

The authors propose LARFT (Length-Aware Reinforcement Fine-Tuning), a unified framework designed to align the model's internal "cognition" of length with its external "action" of generation. The framework consists of three core components:

A. Length-Oriented Reinforcement Learning (RL)

• Algorithm: Utilizes Group Relative Policy Optimization (GRPO).
• Reward Formulation: Instead of sparse rewards, LARFT employs a verifiable length reward.
  • The reward is calculated based on the normalized absolute deviation ($\delta$) between the generated word count and the target constraint $c$.
  • $R_{len} = \max(0, 1 - \delta)$.
  • This provides a dense, bounded signal ($[0, 1]$) to guide the policy toward the target length.

B. Hindsight Length Awareness

To address the entanglement of semantics and length in standard RL, LARFT introduces a mechanism inspired by Hindsight Experience Replay:

• Concept: Transform "failed" generation trajectories (those that didn't meet the length constraint) into valuable supervision data for length cognition.
• Process:
  1. Sampling: Collect on-policy trajectories during the RL rollout phase.
  2. Relabeling: For a generated output $\tau$, construct a new input prompt $x_{aware}$ by concatenating $\tau$ with an awareness instruction (e.g., "Count the words in the text above").
  3. Supervision: The model is trained to predict the actual word count of its own generation ($L(\tau)$) given $x_{aware}$.
• Benefit: This decouples length signals from semantic content, forcing the model to learn an internal representation of length by analyzing its own outputs, effectively turning generation failures into successful "counting" examples.

C. Unified Optimization Mechanism

LARFT employs a joint loss function that dynamically balances the two objectives:
$$\mathcal{L}_{unified} = \mathcal{L}_{GRPO} + \lambda_t \cdot \mathcal{L}_{aware}$$

• $\mathcal{L}_{GRPO}$: The standard policy optimization loss for instruction following.
• $\mathcal{L}_{aware}$: The supervised loss for predicting the length of the generated text.
• Dynamic Scheduling: A Cosine Annealing schedule is applied to the weight $\lambda_t$.
  • Early Training: High $\lambda_t$ prioritizes establishing length cognition (learning to count/understand length).
  • Late Training: $\lambda_t$ decays to 0, shifting focus to precise generation control (executing the length constraint).

3. Key Contributions

1. Identification of the Bottleneck: The paper identifies the lack of intrinsic length cognition as the primary cause of failure in length instruction following, distinguishing between "knowing" (cognition) and "doing" (action).
2. LARFT Framework: A novel training paradigm that integrates Length-Oriented RL with Hindsight Length Awareness to close the cognition-action gap.
3. State-of-the-Art Performance: Demonstrates significant improvements across multiple base models and benchmarks without compromising general capabilities.
4. Interpretability: Provides mechanistic evidence (via linear probing) that LARFT successfully internalizes length planning in deep layers and length awareness in shallow layers.

4. Experimental Results

The authors evaluated LARFT on four base models (Qwen2.5-3B/7B, Llama-3.2-3B, Llama-3.1-8B) across three length benchmarks and four general capability benchmarks.

• Length Instruction Following:
  • LIFEBench: LARFT achieved the lowest Length Deviation (LD) and highest Length Score (LS) across all models.
  • Improvement: Average improvement of +20.92 points on length benchmarks compared to base models.
  • Comparison: Outperformed the strongest RL baseline by +4.59 points and surpassed SFT-based methods (which often fail on short constraints) significantly.
• General Capabilities:
  • Negligible Degradation: LARFT showed only a -1.45 point average drop on general benchmarks (MMLU, GSM8K, IFEval, GPQA).
  • Quality: In some cases, generation quality (ROUGE-L) slightly improved (+2.22 points), indicating that length control does not harm semantic coherence.
• Ablation Studies:
  • Cognition-Action Gap: Models trained only on length awareness (without RL) showed no significant improvement in generation, confirming that cognition alone is insufficient.
  • Unified Optimization: Removing the dynamic scheduling (Cosine Annealing) caused a sharp performance drop, proving the necessity of the curriculum learning approach (cognition first, then action).

5. Significance and Conclusion

• Theoretical Insight: The paper challenges the notion that length control is purely a decoding or reward-shaping problem. It argues that models must first develop an internal "concept" of length before they can reliably execute it.
• Practical Impact: LARFT offers a robust solution for applications requiring strict length constraints, such as creative writing, report generation, and summarization, without requiring model retraining with external tokens or sacrificing general intelligence.
• Future Directions: The framework suggests a path for extending length cognition to other structured generation tasks, such as logical reasoning steps or stylistic constraints.

In summary, LARFT successfully bridges the gap between a model's understanding of length and its ability to generate text of that length, achieving precise control through a synergistic combination of reinforcement learning and self-supervised length awareness.