Technical Summary: Hölder Policy Optimisation (HölderPO)

1. Problem Statement

Group Relative Policy Optimisation (GRPO) has emerged as a dominant, compute-efficient algorithm for aligning Large Language Models (LLMs) with complex reasoning capabilities, notably by eliminating the need for an external critic model. However, GRPO relies on a fixed aggregation mechanism to map trajectory-level advantages to token-level policy updates. Specifically, it uses the arithmetic mean ($p=1$) to aggregate token-level importance ratios within a sequence.

The authors identify a critical limitation in this static approach: different reasoning tasks exhibit varying signal densities, and no single fixed aggregation function is universally optimal.

• Sparse-signal tasks (e.g., AIME, where correct reasoning steps are rare and high-magnitude): Fixed aggregations like the arithmetic mean often dilute these rare "aha moments" by averaging them with unremarkable tokens, or conversely, fail to amplify them sufficiently.
• Dense-signal tasks (e.g., MATH, where supervision is distributed across many tokens): Aggregations that overly concentrate on high-ratio tokens can induce high-variance gradient updates, leading to training collapse.

Empirical observations reveal a trade-off: fixed aggregations that prevent training collapse often fail to yield peak performance on sparse tasks, while those that maximize performance on sparse tasks often suffer from instability on dense tasks. The paper argues that the optimal probability aggregation is not a constant but a function of task signal density and training progression.

2. Methodology: HölderPO

To resolve this, the authors propose HölderPO, a generalised policy optimisation framework that unifies token-level probability aggregation via the Hölder mean (or $p$-norm).

Core Mechanism

Instead of a fixed mean, HölderPO parameterises the aggregation operator with a scalar $p \in \mathbb{R}$. Given a sequence of token-level importance ratios $\{r_{i,t}(\theta)\}$, the sequence-level surrogate term $\rho_{i,p}(\theta)$ is defined as:
$$\rho_{i,p}(\theta) = \begin{cases} \left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} r_{i,t}(\theta)^p \right)^{1/p} & \text{if } p \neq 0 \\ \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log r_{i,t}(\theta) \right) & \text{if } p = 0 \end{cases}$$
This formulation recovers standard GRPO ($p=1$, arithmetic mean) and GSPO/GMPO ($p \to 0$, geometric mean) as special cases, while extending the spectrum to negative values.

Theoretical Analysis: The Concentration-Stability Trade-off

The paper provides a theoretical characterisation of the parameter $p$'s role in the optimisation landscape:

1. Gradient Concentration ($p > 0$): As $p$ increases, the gradient weight distribution $W_{i,t}(p)$ concentrates monotonically on tokens with the highest importance ratios. This amplifies sparse, high-magnitude learning signals (critical for early-stage learning in sparse tasks) but loosens the variance bound.
2. Variance Bounds ($p \le 0$): As $p$ decreases (including negative values), the variance of the policy gradient estimator is strictly tightened. Negative $p$ values invert the concentration, upweighting tokens with ratios $<1$ (signalling hesitation), which promotes reasoning diversity and stabilises training on dense-signal tasks.
3. The Trade-off: No static $p$ can simultaneously maximise signal amplification and minimise variance. A high $p$ is optimal for early-stage signal discovery, while a low (or negative) $p$ is optimal for late-stage convergence stability.

Dynamic Scheduling Strategy

Recognising that the optimal $p$ shifts as training progresses, HölderPO introduces a dynamic annealing algorithm. The parameter $p$ is scheduled to decay monotonically from a positive initial value ($p_{high}$) to a low or negative terminal value ($p_{low}$) over the training lifecycle.

• Early Phase: High $p$ amplifies sparse, high-magnitude signals to drive initial policy improvement.
• Late Phase: Low/negative $p$ contracts the gradient variance bound to ensure stable convergence and prevent mode collapse.

3. Key Contributions

1. HölderPO Framework: A generalised policy optimisation framework that unifies various mean-based aggregations through the adaptable Hölder parameter $p$, extending the search space beyond static arithmetic or geometric means.
2. Theoretical Foundation: A formal proof of the two-sided role of $p$: larger $p$ concentrates gradients to amplify sparse signals, while smaller $p$ strictly bounds gradient variance. The authors prove that no static configuration can achieve both endpoints simultaneously, motivating dynamic scheduling.
3. Empirical Breakthroughs:
   • Demonstrated that a static high $p$ (e.g., $p=3$) breaks the performance ceiling on the challenging AIME benchmark (reaching 46.7% accuracy).
   • Showed that dynamic scheduling ($p: 2 \to -2$) unifies these advantages, achieving state-of-the-art (SOTA) results without additional computational overhead.

4. Experimental Results

The authors evaluated HölderPO on mathematical reasoning and agentic decision-making benchmarks using Qwen2.5-Math and Qwen2.5-Instruct base models.

• Mathematical Reasoning:
  • AIME24 (Sparse-Signal): Static $p=3$ achieved 46.7% accuracy, surpassing the previous GRPO record of 43.3%.
  • MATH500 (Dense-Signal): Lower $p$ values (e.g., $p=-1$) yielded superior stability and performance (85.0% accuracy).
  • Overall Performance: The dynamic annealing strategy ($p: 2 \to -2$) achieved a 54.9% average accuracy across five mathematical benchmarks (AIME, AMC, MATH, Minerva, OlympiadBench). This represents a 7.2% relative gain over standard GRPO and outperforms concurrent methods like PMPO.
• Agentic Tasks (ALFWorld):
  • On the ALFWorld benchmark, HölderPO achieved a 93.8% success rate, a 28.8% relative gain over GRPO (72.8%).
  • The authors note that for agentic tasks, a more conservative schedule ($p: 1 \to -1$) was optimal due to the base model's lack of domain-specific pre-training, highlighting the framework's adaptability.

5. Significance and Claims

The paper claims that HölderPO resolves the fundamental concentration-stability trade-off inherent in static policy optimisation methods like GRPO. By treating the aggregation mechanism as a continuous dial rather than a fixed hyperparameter, the framework allows the model to:

1. Aggressively amplify sparse, high-magnitude learning signals during the early stages of training.
2. Strictly control gradient variance to ensure stable convergence in later stages.

The authors emphasize that this dynamic adaptability is crucial for long-horizon reasoning tasks where the nature of the learning signal evolves. The method achieves SOTA performance across diverse benchmarks while maintaining the compute efficiency of critic-free RL.

Limitations Acknowledged:

• The schedule introduces hyperparameters ($p_{high}, p_{low}$, decay shape) that require empirical tuning per task.
• The positive-concentration regime ($p > 0$) may increase susceptibility to reward hacking if the verifier provides false-positive signals, as it aggressively amplifies high-ratio tokens.

Future Work:
The authors suggest developing an adaptive scheduler that adjusts $p$ based on real-time metrics (e.g., batch-level gradient variance or token-ratio dispersion) to remove the need for manual tuning.