Technical Summary: Affordance-Based Hierarchical Reinforcement Learning for Quadruped Pedipulation

Problem Statement

The autonomous manipulation of objects by quadruped robots remains a significant research challenge. While previous studies have successfully utilized reinforcement learning (RL) for low-level policy learning (locomotion and basic manipulation), task execution in open-world environments often relies on expert-designed high-level trajectories. Specifically, the autonomous selection of both a feasible robot base pose and an affordable interaction point on a target object is underexplored. This gap is critical because manipulating objects on uneven terrain introduces complexities such as unmodeled surface geometry, unknown friction, and the need to maintain balance on remaining limbs while one limb interacts with the object. Current approaches often lack the ability to autonomously identify affordance-aware poses that account for object reachability, local terrain characteristics, and surface features without human guidance.

Methodology

The authors propose a three-level hierarchical reinforcement learning (RL) framework that integrates vision-based affordance analysis with decoupled control policies. The system is designed to autonomously navigate to a target object, identify a feasible interaction point, and execute a "pedipulation" task (manipulation using the robot's legs).

1. Hierarchical Architecture

The framework consists of three distinct layers:

• High-Level (Pose Affordance & Navigation): A vision-based module processes LiDAR point cloud data to segment terrain and objects. It calculates local terrain slopes and object geometry to generate an affordable goal base pose (position and heading) for the robot. This pose ensures the robot is within reach of the object and oriented correctly relative to the terrain slope.
• Mid-Level (Navigation Policy): This policy receives the goal pose and the robot's state (velocity, orientation) to predict velocity commands. It guides the low-level controller in a closed-loop manner to reach the target base position.
• Low-Level (Locomotion & Pedipulation Policies):
  • Locomotion: A pretrained policy outputs joint positions to execute the velocity commands from the navigation layer.
  • Pedipulation: Once the robot reaches the goal pose, the system switches to the pedipulation policy. This policy is conditioned on high-level waypoints generated via cubic spline interpolation between a "home" position, a vision-identified contact point on the object, and an intermediate offset point. The policy controls the end-effector (specifically the front-right foot) to push the object while maintaining balance with the other three legs.

2. Affordance-Based Planning

The core innovation lies in the pose affordance module. Instead of relying on pre-defined trajectories, the system dynamically computes:

• Goal Base Pose: Derived from the object's center, the desired pushing heading, and the local terrain slope.
• Interaction Point: Determined by analyzing the object's surface geometry to find the point closest to the end-effector, maximizing the contact trajectory length.
• Trajectory Generation: A cubic spline is constructed to ensure smooth transitions with zero velocity constraints at start and end points, with height clamped for safety.

3. Training and Implementation

• Environment: Policies were trained in the Isaac Sim ecosystem using the Unitree Go2 quadruped platform.
• Algorithm: Actor-critic architecture with Proximal Policy Optimization (PPO).
• Domain Randomization: Training included randomization of base mass, center of mass, external forces, and terrain slopes to ensure robustness.
• Real-World Deployment: The system was deployed using a ROS2 framework with an Error State Extended Kalman Filter (ESEKF) for state estimation and a finite state machine (FSM) to manage operational modes.

Key Contributions

The paper outlines four primary contributions:

1. Visual-Affordance-Based Pose Command Generation: A module that generates goal poses for both the robot base and end-effector, explicitly accounting for reachability, terrain slope, and object surface features.
2. Multi-Level Hierarchical RL Framework: A system enabling fully autonomous navigation and pedipulation, where higher-level behaviors abstract lower-level policies as black boxes.
3. Sim-to-Real Transfer: A successful implementation demonstrating the system's ability to transfer from simulation to unstructured, outdoor real-world environments.
4. Object Interaction Dataset: A dataset containing robot telemetry (including end-effector force feedback), object features, and interaction poses, intended to facilitate further research in object exploration.

Experimental Results

The framework was evaluated in both simulation and real-world settings involving random object placements on varying slopes.

• Simulation:
  • Affordance Effectiveness: Experiments showed that using the affordance-based goal pose (incorporating terrain slope) resulted in higher object displacement efficiency compared to random or fixed goal poses.
  • Contact Point Selection: Selecting the interaction point closest to the end-effector yielded the highest efficiency (displacement per unit of force), outperforming strategies based on proximity to the robot base or geometric centers.
• Real-World:
  • Navigation: The robot achieved a mean position error of 0.12 m and a mean yaw error of 16.37° when reaching goal poses on slopes up to ~9°.
  • Pedipulation: The system achieved an 86% success rate for object interaction (defined by a force threshold of 25 N).
  • Performance: Across successful trials, the robot achieved a mean object displacement of 0.17 m with a mean peak contact force of 33 N, resulting in a mean efficiency score of 0.51 m N⁻¹.
  • Robustness: The system successfully handled dynamic transitions from navigation to manipulation without human intervention, relying only on safety checks via wireless controllers.

Significance and Claims

The paper claims that this work addresses a critical gap in autonomous robotics by enabling object-centric pedipulation planning in open-world environments. Unlike previous methods that rely on expert guidance or focus solely on low-level control, this framework allows a quadruped robot to:

• Autonomously identify feasible base poses and interaction points based on visual affordances.
• Execute complex manipulation tasks (pushing objects) without pre-designed trajectories or human demonstrations.
• Operate effectively in unstructured environments with varying terrain slopes.

The authors position this capability as highly relevant for applications such as planetary exploration, sample collection, and rescue operations, where real-time teleoperation is infeasible due to latency or danger. The study emphasizes that the system achieves dexterous manipulation using existing leg hardware, avoiding the need for additional manipulator arms, thereby leveraging the inherent maneuverability of quadruped robots. The authors modestly note that while the system is effective, the accumulation of errors across interconnected components (vision, state estimation, control) remains a challenge for high-precision tasks, and future work will focus on leveraging the collected data to learn better object-centric representations.