Technical Summary: ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training

1. Problem Statement

The paper addresses a critical instability in asynchronous Reinforcement Learning (RL) for Large Language Model (LLM) post-training. Modern systems often decouple response generation (rollout) from policy optimization (learning) to improve throughput. In this setup, rollout workers sample responses from recent policy snapshots while the learner updates a newer policy.

This asynchrony introduces distribution drift: the behavior policy ($\pi_b$) that generated a response differs from the current policy ($\pi_\theta$) used to train on it.

• Standard Solution: Methods like PPO use behavior-policy probabilities (log-probs) to compute importance ratios and apply clipping. This requires the rollout system to transmit token-aligned, versioned, and numerically consistent behavior log-probabilities to the learner, complicating the infrastructure and introducing potential numerical drift.
• The Failure Mode: The authors identify that naive current-policy-only objectives (which omit behavior log-probs) are unstable in asynchronous settings. Specifically, in group-relative RL where advantages sum to zero ($\sum A_g = 0$), positive and negative advantage responses can enter the loss function at vastly different current negative log-probability scales.
  • Stale negative-advantage responses often have much larger current negative log-probabilities than positive responses.
  • This causes the negative side of the loss to dominate, breaking the zero-sum cancellation intended by the group baseline.
  • The result is training collapse, where the model suppresses stale negative samples excessively without receiving compensating signals from positive samples.

2. Methodology

The paper proposes two current-policy-only objectives designed to stabilize training without requiring behavior-policy information. Both operate on the group-relative advantage $A_g$ and the current policy's average token negative log-probability $S_{\theta,g}$.

2.1 Scaled Policy Optimization (SPO)

SPO is a heuristic baseline that manually reduces the influence of negative-advantage responses.

• Mechanism: It applies a fixed scaling coefficient $\alpha \in (0, 1)$ to negative advantages ($A_g < 0$), while leaving positive advantages ($A_g \ge 0$) unchanged.
• Loss Function:
  $$L_{SPO}(\theta) = -\frac{1}{G} \sum_{g=1}^G \frac{1}{m_g} \sum_{i=1}^{m_g} \tilde{A}_g \log p_{g,i}(\theta)$$
  where $\tilde{A}_g = \alpha A_g$ if $A_g < 0$, else $A_g$.
• Limitation: The scaling factor $\alpha$ is manually tuned and does not adapt to the specific scale of each response or the degree of staleness.

2.2 Asymmetric-Scale Policy Optimization (ASymPO)

ASymPO replaces the manual rule with an adaptive, response-level normalization that balances loss contributions automatically.

• Mechanism: It normalizes the loss of each response by its own current average token negative log-probability ($S_{\theta,g}$). A stop-gradient operator ($sg$) is applied to the denominator to prevent the normalization term from being updated during backpropagation.
• Loss Function:
  $$L_{ASymPO}(\theta) = \frac{1}{G} \sum_{g=1}^G A_g \frac{S_{\theta,g}}{sg(S_{\theta,g})}$$
• Theoretical Guarantee: Theorem 2 proves that under zero-sum advantages, ASymPO ensures exact response-level loss balance. The forward loss contribution of positive responses equals the magnitude of the negative response contributions, regardless of their differing current log-probability scales.
• Gradient: The gradient remains a standard policy gradient direction, but the magnitude of the update for each response is normalized by its own current scale.

3. Key Contributions

1. Identification of Scale Imbalance: The paper formally identifies a specific failure mode in asynchronous, current-policy-only RL where stale negative samples dominate the loss due to scale discrepancies, breaking the zero-sum advantage balance.
2. Formulation of SPO and ASymPO: It introduces two objectives that stabilize training without behavior-policy probabilities. ASymPO is proven to exactly balance positive and negative loss contributions under zero-sum advantages.
3. Infrastructure Simplification: ASymPO allows for a strictly more compact rollout–learner interface. It eliminates the need to transport per-token behavior log-probabilities, policy-version tags, or perform logit recomputation at training precision (Table 1).
4. Empirical Validation: The methods are evaluated on asynchronous mathematical reasoning post-training across three model families (Qwen3-1.7B, Qwen3-4B, LLaMA-3.2-3B).

4. Experimental Results

The experiments were conducted on mathematical reasoning tasks (MATH, AIME, GSM8K, etc.) using asynchronous RL with a staleness threshold of 0.5.

• Stability:
  • Naive Loss and GPG (a method avoiding importance sampling but lacking group-balance) collapsed during training across all model families, failing to produce evaluable checkpoints.
  • GRPO (behavior-corrected), SPO, and ASymPO maintained stable training reward trajectories.
• Performance:
  • ASymPO generally achieved competitive or superior performance compared to GRPO (the behavior-corrected baseline) on aggregate metrics, particularly on LLaMA-3.2-3B-Instruct and Qwen3-4B-Base.
  • SPO also prevented collapse and remained competitive, though ASymPO's adaptive scaling often yielded better results on specific benchmarks (e.g., AIME24, MATH500).
  • On Qwen3-1.7B-Base, ASymPO achieved a mean@8 of 36.98, close to GRPO's 37.45, while SPO achieved 36.41.
• Ablation: The paper notes that SPO is sensitive to the choice of $\alpha$ (set to 0.2 in experiments). If $\alpha=1$, it collapses like the naive loss; if $\alpha=0$, it behaves like positive-only fine-tuning with weak performance.

5. Significance and Claims

The paper claims that behavior-policy probabilities are not strictly necessary for competitive and stable asynchronous RL post-training, provided that the scale imbalance between positive and negative responses is explicitly corrected.

• Practical Impact: ASymPO offers a mechanism to stabilize asynchronous group-relative RL while significantly simplifying the system architecture. It removes the "correctness condition" that ties the rollout–learner interface to behavior log-probability transport, versioning, and numerical consistency.
• Scope: The authors modestly frame the contribution as addressing a specific failure mode (loss-scale imbalance) rather than a general solution to all distribution drift. They acknowledge that ASymPO does not bound the policy ratio between generation and training like behavior-corrected methods, and its effectiveness is currently demonstrated primarily in mathematical reasoning tasks with verifiable rewards.
• Future Work: The paper suggests that further theoretical characterization is needed to understand when response-scale normalization approximates behavior correction and when explicit behavior-policy information remains unavoidable.