Technical Summary: Growing Policy Optimization (GPO) for Legged Robot Locomotion

1. Problem Statement

Training reinforcement learning (RL) policies for legged robots, particularly under torque-based control, faces significant challenges due to high-dimensional continuous action spaces, strict hardware constraints, and the difficulty of exploration. While existing methods utilizing position-based control often succeed through environment-specific heuristics (e.g., reward shaping, curriculum learning, and manual initialization), they struggle with torque-based control. In torque control, actions are directly mapped to joint torques, creating strong coupling and stringent stability requirements.

The core issue identified is that standard policy gradient methods, such as Proximal Policy Optimization (PPO), implicitly assume a fixed action space throughout training. This design forces the policy to explore the full, high-variance action space from the beginning. Consequently, early-stage training suffers from:

• High gradient variance and unstable optimization landscapes.
• Difficulty in learning reliable, low-amplitude control behaviors before converging to an optimal policy.
• Inefficient data collection due to the risk of destabilizing the robot during exploration.

Current engineering practices often manually restrict action limits early in training and relax them later, but there is no principled framework that treats the evolution of the action space as an integral part of the learning algorithm, nor is there a clear theoretical analysis of its impact on optimization properties.

2. Methodology: Growing Policy Optimization (GPO)

The authors propose Growing Policy Optimization (GPO), a framework that applies a time-varying action transformation to restrict the effective action space during the early stages of training and progressively expand it.

Core Mechanism

GPO introduces a smooth, time-dependent transformation that maps a policy's latent action output ($a$) to an executable action ($\tilde{a}$):
$$\tilde{a} = \beta_t \tanh\left(\frac{a}{\beta_t}\right)$$
Here, $\beta_t = a_{limit} f(t)$ is a monotonically increasing growth function of the training step $t$, where $\lim_{t \to \infty} f(t) = 1$.

• Early Stage: $\beta_t$ is small, restricting the effective action space. The transformation operates in a near-linear regime for small actions, allowing the agent to learn stable, low-amplitude behaviors with reduced gradient noise.
• Late Stage: As $\beta_t$ increases, the effective action space expands, unlocking larger control authority and enabling broader exploration to achieve higher expected returns.
• Saturation: For large-magnitude actions, the output smoothly saturates at the hardware limit ($\pm a_{limit}$), functionally equivalent to hard clipping but preserving differentiability.

Theoretical Guarantees

The paper provides a theoretical analysis proving that GPO maintains the stability of standard PPO while improving optimization dynamics:

1. Update Equivalence: The importance sampling ratio under GPO is structurally identical to PPO ($r^{GPO}_t = r^{PPO}_t$), meaning the clipped surrogate objective remains unchanged up to a change of variables.
2. Bounded Gradient Distortion: The difference between GPO gradients and standard PPO gradients is bounded and decreases monotonically as $\beta_t$ increases. This ensures that the transformation introduces only vanishing gradient distortion.
3. Optimization Dynamics:
   • Early Stage: By restricting the action space, GPO reduces the variance of the policy gradient estimator (which scales quadratically with $\beta_t$). This increases the Signal-to-Noise Ratio (SNR), leading to faster local convergence and a tighter lower bound on early-stage expected return.
   • Late Stage: The gradual expansion ensures that the steady-state parameter error is no worse than (and potentially better than) a fixed-range baseline, allowing for improved asymptotic performance.

3. Key Contributions

• Novel Framework: Introduction of GPO, which explicitly models the evolution of the effective action space as a component of the RL algorithm rather than a static engineering heuristic.
• Theoretical Analysis: Proof that the time-varying transformation preserves PPO's update structure and introduces only bounded, vanishing gradient distortion, ensuring stable training.
• Performance Improvements: Demonstration that GPO achieves faster early-stage convergence and higher asymptotic returns compared to fixed-action-space baselines, specifically in the challenging domain of torque-based control.
• Generalizability: Validation of the framework across different robot morphologies (quadruped and hexapod) and control tasks (locomotion and whole-body control) without requiring task-specific reward shaping or gait priors.

4. Experimental Results

The authors evaluated GPO on quadruped and hexapod robots in both simulation and real-world hardware deployment.

Simulation Results

• Growth Functions: Comparisons between no growth (PPO), linear, sigmoid, and Gompertz growth functions showed that the Gompertz function (slow initial growth followed by accelerated expansion) yielded the best performance. It provided the fastest reward improvement and most stable learning.
• Tracking Performance: GPO-trained policies demonstrated significantly lower tracking errors for velocity and body pose commands compared to PPO. For instance, PPO failed to track x-direction velocities effectively, while GPO maintained close adherence.
• Gait Emergence: GPO policies exhibited more periodic and coordinated joint torques and velocities. In contrast, PPO policies often resulted in erratic behaviors and degenerate support allocation (e.g., over-reliance on specific legs), whereas GPO facilitated structured coordination patterns without explicit gait priors.

Hardware Validation

• Zero-Shot Sim-to-Real: Policies trained in simulation were deployed on a Unitree Go2 quadruped without fine-tuning.
• Robustness: GPO outperformed baselines (including torque-based PPO and DeCAP variants) in disturbance rejection tasks, including side pushes, vertical stomps, and locomotion over challenging terrains (muddy uphill, slippery downhill).
• Recovery: Under external perturbations (e.g., human lifting legs, dropped weights), GPO-enabled robots recovered balance smoothly without triggering unnecessary stepping motions, demonstrating superior closed-loop stability.

5. Significance and Claims

The paper claims that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion. Its significance lies in shifting the paradigm from modifying the training pipeline (via rewards or curricula) to modifying the underlying optimization mechanism itself.

By treating the action space evolution as a learnable, time-varying process, GPO addresses the fundamental trade-off between exploration and stability in high-dimensional continuous control. The authors assert that this approach enables:

1. Stable Early Optimization: Learning reliable low-amplitude control before tackling complex dynamics.
2. Enhanced Asymptotic Performance: Achieving higher expected returns by progressively unlocking the full action space.
3. Robust Sim-to-Real Transfer: Facilitating the deployment of torque-based policies on real hardware without the need for extensive manual tuning or gait priors.

The work suggests that managing the "effective" action space is a critical, yet previously under-explored, lever for improving embodied RL in contact-rich, uncertain environments.