Technical Summary: Guided Reinforcement Learning for Omnidirectional 3D Jumping in Quadruped Robots

Problem Statement

Quadruped robots face significant challenges when executing jumps, particularly those exceeding their leg length or occurring in non-flat environments. While optimization-based methods exist, they often suffer from high computational costs, sensitivity to initialization, and a reliance on precise system models that are difficult to maintain in unstructured settings. Conversely, standard End-to-End (E2E) Reinforcement Learning (RL) approaches, while promising, typically require millions of training steps due to sparse reward signals and lack interpretability, making safety certification difficult. The core problem addressed is the need for a control strategy that is sample-efficient, robust to model uncertainties, capable of omnidirectional 3D jumping (including elevation changes and rotations), and explainable enough to ensure physical feasibility.

Methodology

The authors propose a Guided Reinforcement Learning (GRL) framework that injects physical intuition into the learning process, moving away from raw E2E control. The methodology is structured around three main components:

1. Trajectory Parametrization

Instead of learning joint torques or positions directly, the policy learns a high-level trajectory in Cartesian space.

• Hybrid Parametrization: The thrust phase trajectory is modeled using a combination of Bézier curves and a Uniformly Accelerated Rectilinear Motion (UARM) model.
  • The initial phase uses a cubic Bézier curve to model the "charging" and decompression motion, ensuring smoothness and differentiability.
  • To overcome the expressiveness limits of Bézier curves during the explosive lift-off phase (which can lead to workspace violations like the robot's belly touching the ground), the trajectory transitions into a UARM segment. This allows for higher lift-off velocities while maintaining feasibility.
• Orientation Control: A separate Bézier curve parametrizes the trunk orientation (Euler angles), enabling dynamic maneuvers such as in-place twist jumps.
• Action Space: The RL agent outputs a reduced set of 13 parameters (e.g., time intervals, spherical coordinates for lift-off position/velocity, velocity multipliers, and orientation targets) rather than raw joint commands. This drastically reduces the search space.

2. Learning Framework

• Algorithm: The system uses Proximal Policy Optimization (PPO).
• State Space: The state is defined in task space (displacement to target in position and orientation), making the policy agnostic to specific robot morphologies. Joint-level information is excluded to prevent overfitting to specific hardware.
• Reward Function: A physically informative reward function is designed to balance task success with safety. It includes:
  • Positive Reward: A landing target reward that scales with jump distance and penalizes landing error.
  • Penalties: Costs for violating path constraints (joint limits, friction cones, torque limits), tracking errors, and unsafe touchdown conditions (bouncing, high angular velocity).
• Safety Filter: A post-hoc safety check uses ballistic equations to verify if a proposed action can physically reach the target height before execution, adding a layer of explainability.

3. Control Execution

The learned Cartesian trajectory is mapped to joint positions via Inverse Kinematics (IK). A low-level Proportional-Derivative (PD) controller with gravity compensation tracks these references. This separation allows the high-level RL policy to focus on trajectory generation while relying on classical control for robustness against friction and inertia uncertainties.

Key Contributions

1. Novel GRL Strategy: A sample-efficient RL method for omnidirectional 3D jumping that leverages physical intuition (Bézier + UARM) to guide the learning process, avoiding the "black box" nature of E2E approaches.
2. Robot-Agnostic Action Space: A formulation where actions are defined in task space (Cartesian position/orientation), allowing the policy to generalize across different quadruped morphologies (demonstrated via zero-shot transfer).
3. Predictability and Safety: The parametric nature of the trajectory allows for deterministic prediction of landing points and the implementation of safety filters, addressing the safety certification gap in RL.
4. Single-Stage Learning: Unlike curriculum-based approaches requiring multi-stage training, this method achieves the final goal in a single learning stage, significantly reducing training time and complexity.

Results

The method was validated through extensive simulations and real-world experiments on two quadruped robots: the Unitree Go1 and the Aliengo.

• Sample Efficiency: The policy converged in approximately 2,000 training steps (15 hours), a significant reduction compared to E2E methods that often require millions of steps.
• Accuracy and Feasibility:
  • Flat Jumps: Achieved landing errors below 5 cm for jumps within a 0.5-meter radius.
  • Vertical Jumps: Successfully executed upward jumps up to 0.26 meters and downward jumps up to 0.4 meters.
  • Rotations: Demonstrated the ability to perform in-place yaw rotations with a maximum orientation error of approximately 6 degrees.
• Robustness:
  • Sim-to-Sim: The policy maintained performance when transferred to a different simulator (Locosim) and under variations in mass (up to 50%) and joint damping, without requiring domain randomization during training.
  • Sim-to-Real: Real-world experiments on Go1 and Aliengo showed high repeatability. While absolute errors were slightly higher on the heavier Aliengo (attributed to unmodeled friction), the standard deviation remained low, indicating consistent behavior.
• Zero-Shot Transfer: A policy trained on the Go1 was successfully applied to the Aliengo robot (which has different mass and link lengths) with only low-level control adjustments, validating the task-space formulation.
• Comparison with SOTA: Compared to the state-of-the-art E2E approach by Atanassov et al. [39], the proposed method achieved comparable or superior accuracy with significantly fewer samples, higher jump elevations (0.26m vs. 0.1m), and a more uniform feasible region.

Significance and Claims

The paper claims that by integrating physical intuition (guided by Bézier curves and ballistic physics) into the RL framework, it is possible to overcome the primary limitations of both traditional optimization (computational cost, model dependency) and standard E2E RL (sample inefficiency, lack of interpretability).

The authors emphasize that their approach:

• Enables Real-Time Application: The lightweight neural network and pre-computable trajectory allow for millisecond-level planning.
• Ensures Physical Feasibility: The method inherently respects joint limits and friction constraints through the reward design and action space constraints, reducing the risk of unsafe behaviors.
• Facilitates Generalization: By decoupling the policy from specific joint configurations, the method supports transfer across different robot platforms without retraining from scratch.
• Simplifies Training: The single-stage learning process eliminates the need for complex curriculum scheduling and extensive reward engineering.

The authors conclude that while the current method is purely kinematic and assumes zero initial velocity, it represents a significant step forward in making dynamic jumping robust, safe, and efficient for legged robots in complex environments. Future work is identified as extending the method to torque-controlled dynamics, handling non-zero initial velocities, and incorporating deformable terrain.