Technical Summary: Length-Unbiased Sequence Policy Optimization (LUSPO)

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs). A prevailing observation in RLVR training is that increased response length often correlates with improved reasoning, as models learn to decompose complex problems into detailed steps. However, the paper identifies a critical flaw in mainstream RLVR algorithms, specifically Group Relative Policy Optimization (GRPO) and Group Sequence Policy Optimization (GSPO): they exhibit an inherent response length bias that can lead to "response length collapse," where models generate increasingly brief responses, thereby undermining training efficacy and reasoning depth.

The authors analyze the sources of this bias:

• In GRPO: The objective function averages token-level importance ratios within a trajectory. This normalization causes tokens in shorter responses to contribute more heavily to the gradient update than those in longer responses. Consequently, for correct answers, the model is incentivized to be brief; for incorrect answers, it is penalized less for being long, creating an unstable dynamic.
• In GSPO: While GSPO addresses GRPO's instability in Mixture-of-Experts (MoE) models by using sequence-level importance weighting, it fails to resolve the length bias. The paper argues that GSPO's sequence-level clipping mechanism, combined with the "Clip-Higher" operation, disproportionately truncates negative (incorrect) samples. This results in a gradient update dominated by positive samples, which, due to the underlying length bias, further incentivizes the generation of shorter outputs. Empirical evidence shows that while GRPO tends to increase response length, GSPO often causes a collapse in response length during training, particularly in dense and VL models.

2. Methodology: Length-Unbiased Sequence Policy Optimization (LUSPO)

To rectify the length bias inherent in GSPO, the authors propose Length-Unbiased Sequence Policy Optimization (LUSPO). The core innovation is a principled modification to the GSPO objective function.

Theoretical Basis

The authors provide a theoretical analysis demonstrating that the standard GSPO loss function is biased against longer sequences. To neutralize this, LUSPO scales the loss of each sequence by its own length ($|y_i|$).

The Algorithm

The LUSPO optimization objective is defined as:
$$J_{LUSPO}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\theta_{old}}(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( s_i(\theta) \hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i \right) \cdot |y_i| \right]$$

Where:

• $s_i(\theta)$ is the sequence-level importance ratio (same as GSPO).
• $\hat{A}_i$ is the group-based advantage estimate.
• $|y_i|$ is the length of the response.

By multiplying the clipped advantage term by the sequence length $|y_i|$, the algorithm ensures that longer sequences are not unfairly penalized during the gradient update process. Theoretical gradient analysis confirms that this modification eliminates the length-dependent bias present in GSPO, ensuring that the magnitude of the gradient update is proportional to the sequence length rather than inversely proportional to it.

3. Key Contributions

The paper outlines three primary contributions:

1. Analysis of Length Bias: A detailed theoretical and empirical analysis of the objective functions of GRPO and GSPO, clarifying the underlying mechanisms that cause inherent response length bias and subsequent length collapse.
2. LUSPO Algorithm: The proposal of a novel optimization strategy that neutralizes GSPO's length bias through a straightforward modification (length scaling), rendering the loss function unbiased with respect to response length.
3. Comprehensive Validation: Extensive experiments across diverse architectures (dense and MoE) and modalities (text-only and vision-language) demonstrating that LUSPO achieves superior performance and stability compared to existing methods.

4. Experimental Results

The authors conducted experiments on Qwen2.5-7B-Base (dense), Qwen3-30B-A3B-Instruct (MoE), and Qwen2.5-VL-7B-Instruct (Vision-Language).

Training Dynamics

• Response Length: LUSPO consistently prevents response length collapse. On the Qwen2.5-VL-7B-Instruct model, GSPO exhibited a pronounced collapse in response length, whereas LUSPO maintained stable and significantly longer responses.
• Gradient Stability: The method ensures stable training dynamics in both dense and MoE architectures, addressing the instability issues often associated with RLVR in MoE models.

Performance Benchmarks

• Text-Only Models:
  • Qwen2.5-7B-Base: LUSPO improved accuracy by +2.9% on AIME24 and +7.4% on MATH500 compared to GSPO.
  • Qwen3-30B-A3B-Instruct: LUSPO achieved a +6.9% improvement on AIME24 and a +17.1% improvement on AIME25 over GSPO.
• Multimodal Models:
  • Qwen2.5-VL-7B-Instruct: LUSPO outperformed both GRPO and GSPO across multiple benchmarks. Notably, it achieved +1.6% higher accuracy than GRPO and +0.5% higher than GSPO on MathVista-Mini. On LogicVista, LUSPO showed a +6.0% improvement over GSPO.
• Generalization: The method demonstrated robustness across different datasets (ViRL39K and DAPO-MATH-17K), consistently outperforming GSPO even in scenarios where GSPO did not suffer from severe length collapse.

5. Significance and Claims

The paper positions LUSPO as a novel, state-of-the-art optimization strategy for RLVR training. The authors claim that by addressing the fundamental length bias in sequence-level policy optimization, LUSPO:

• Resolves Response Length Collapse: It prevents the degradation of response length that hinders the model's ability to perform complex reasoning.
• Enhances Reasoning Capabilities: By allowing models to generate longer, more coherent responses without penalty, LUSPO facilitates the exploration of wider reasoning patterns, leading to improved performance on complex mathematical and logical tasks.
• Broad Applicability: The method is effective across both dense and MoE architectures, as well as in both text-only and multimodal settings, making it a versatile solution for next-generation large model training.

The authors conclude that LUSPO represents a significant step forward in stabilizing RLVR training and maximizing the reasoning potential of large language models.