Technical Summary: Reward-Decorrelated Policy Optimization (RDPO)

1. Problem Statement

In complex reinforcement learning (RL) environments for Large Language Models (LLMs), training often involves multi-task settings with mixed-reward formulations. A single training run may incorporate diverse objectives such as correctness, instruction following, rubric satisfaction, preference-model scores, and response length.

Aggregating these heterogeneous reward signals into a single scalar advantage presents two primary challenges that destabilize training:

1. Heterogeneous Scales and Distributions: Rewards vary significantly in scale (binary, fractional, continuous) and distribution shape (skewed, tied, outlier-prone). Standard aggregation methods often fail to allocate advantages robustly at the prompt level, leading to situations where a single rollout dominates the update signal.
2. Reward Correlation: Reward dimensions within a task are often correlated (e.g., code correctness and response length, or rubric satisfaction and preference scores). Summing correlated rewards without adjustment introduces redundant variance, causing the model to "double-count" specific variations or cancel out useful signals, thereby reducing the effective information efficiency of the aggregated advantage.

Existing methods like GRPO (direct summation) obscure relative performance variations, while GDPO (independent Z-score normalization) remains sensitive to non-Gaussian distributions at the prompt level and fails to address inter-reward correlations.

2. Methodology: Reward-Decorrelated Policy Optimization (RDPO)

The authors propose RDPO, a lightweight, two-step reward-processing pipeline designed to explicitly target the failure modes of scale instability and correlation redundancy.

2.1 Diagnostic Metric: Effective Information Efficiency ($\eta_{eff}$)

Before detailing the solution, the paper introduces $\eta_{eff}$ as a diagnostic measure to evaluate mixed-reward aggregation. It decomposes into:

• $\eta_{proj}$ (Projection Efficiency): Measures how well the aggregation aligns with an equally weighted projection in the standardized reward space. It penalizes imbalances caused by varying reward scales.
• $\eta_{corr}$ (Correlation Efficiency): Quantifies the independent information retained after summing correlated rewards. It penalizes redundancy caused by positive or negative linear dependencies between reward dimensions.

2.2 Step 1: Magnitude-Aware Quantile (MAQ) Normalization

Goal: To stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards, addressing the limitations of Z-score normalization under non-Gaussian conditions.

Mechanism:

• Log-Compressed Gaps: Instead of linear scaling, MAQ computes the spacing between sorted rollout scores using a logarithmic compression function: $gap_j = \log(1 + |r_{j+1} - r_j| / (\beta \cdot \sigma_{global}))$. This restricts the influence of extreme outliers while preserving subtle distinctions in dense regions.
• CDF Allocation: Gaps are normalized to allocate Cumulative Distribution Function (CDF) positions proportionally.
• Inverse Normal Mapping: Values are mapped to a standard normal distribution via the inverse CDF ($\Phi^{-1}$).

Effect: MAQ produces advantages that are comparable across dimensions and less sensitive to pathological group statistics (ties, skew, outliers), ensuring that no single rollout disproportionately dominates the prompt-level update.

2.3 Step 2: Mahalanobis Whitening

Goal: To mitigate redundant variance among co-occurring reward dimensions within a specific task subspace.

Mechanism:

• Subspace Selection: In multi-task settings, a rollout rarely observes all reward dimensions. RDPO applies whitening exclusively over the observed reward subspace $S$ for a given task.
• Running Covariance Estimation: An Exponential Moving Average (EMA) is used to estimate the covariance matrix $\hat{\Sigma}_t$ of the MAQ-normalized advantages online, smoothing out batch-level noise.
• Whitening Transformation: The advantage vector $A^{(i,j)}$ is transformed using the inverse square root of the covariance matrix: $W^{(i,j)} = \hat{\Sigma}_S^{-1/2} A^{(i,j)}$. This shifts the active reward dimensions toward uncorrelated, unit-variance signals.
• Aggregation: The final scalar advantage is the sum of the whitened dimensions, followed by batch-wise normalization.

3. Experimental Setup

• Model: LongCat-Flash (post-training stage).
• Tasks: A unified mixture of four task types: Mathematical Reasoning, Code Generation, Instruction Following, and General Writing.
• Reward Subspaces:
  • Math: Math correctness + Length.
  • Code: Code correctness + Length.
  • Instruction Following: IFEval + Rubrics.
  • General Writing: Length + RM (Reward Model) + Rubrics.
• Baselines: GRPO (raw summation), GDPO (Z-score normalization), and the RL initialization model.
• Validation: Small-scale validation on a same-family smaller model to isolate the contributions of MAQ and Whitening before scaling to LongCat-Flash.

4. Key Results

4.1 Small-Scale Validation

On a smaller internal model, the full RDPO pipeline outperformed GDPO, GRPO, and the initialization model across several metrics:

• Instruction Following: Significant gains on IFEval (83.55% vs. 78.56% for GDPO).
• Reasoning & Coding: Improvements on AIME24, FullStackBench, HumanEval+, and MBPP+.
• Component Analysis: Both MAQ (Q) and Whitening (M) provided complementary benefits. MAQ was particularly effective on distribution-sensitive metrics (e.g., AH-Creative), while whitening aided correlation-sensitive settings.

4.2 Scaled LongCat-Flash Results

When applied to LongCat-Flash, RDPO demonstrated:

• Instruction Following & Writing: RDPO achieved the highest scores on IFEval (90.39%), GuideBench, SOP-Maze, WritingBench, and both ArenaHard v2 subsets (AH-Creative: 89.00%, AH-Hard: 76.10%). This confirms the utility of stabilizing prompt-level advantages and reducing redundancy for open-ended, preference-sensitive tasks.
• Reasoning & Coding: Results were broadly competitive but mixed. RDPO matched the best MATH500 score (98.60%) and remained competitive on AIME2025 and GPQA. In coding, RDPO led on MBPP+ and LiveCodeBench v6, while GRPO or the initialization model remained stronger on FullStackBench and HumanEval+.
• Overall Stability: The method successfully transferred from small-scale validation to large-scale training, maintaining stable reasoning and coding performance while significantly boosting instruction following and writing quality.

5. Significance and Claims

The paper claims that RDPO offers a robust solution for multi-objective RL in LLM post-training by explicitly addressing the dual failure modes of scale instability and correlation redundancy.

• Methodological Contribution: By combining Magnitude-Aware Quantile normalization with Mahalanobis whitening, RDPO ensures that the aggregated advantage signal is both geometrically consistent (equal contribution across dimensions) and informationally efficient (minimizing redundant variance).
• Practical Impact: The method enables the simultaneous optimization of heterogeneous tasks (e.g., math, code, writing) without the training instability typically associated with mixed-reward formulations.
• Performance: The authors conclude that RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations, validating its effectiveness as a general-purpose reward processing strategy for complex RL environments.