💻 computer science

Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL

This paper proposes a contact-guided exploration strategy within a Multi-Critic Reinforcement Learning framework to overcome the challenges of complex hybrid dynamics and sparse contact in non-prehensile manipulation, successfully demonstrating deployable skills for tasks like chair transportation on a real-world quadrupedal mobile manipulator.

Original authors: Simone Tolomei, Mayank Mittal, Franco Angelini, Manolo Garabini, Paolo Salaris, Marco Hutter

Published 2026-08-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Simone Tolomei, Mayank Mittal, Franco Angelini, Manolo Garabini, Paolo Salaris, Marco Hutter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the factory floor, where they lift heavy parts and place them with perfect precision. But step outside that controlled environment into a home or a warehouse, and the rules change. Here, objects are often too heavy to lift or too awkward to grab with a gripper. To move a bulky armchair or a large box, a robot cannot simply pick it up; it must push, pull, or slide it across the floor. This is known as non-prehensile manipulation. It is a difficult skill because the physics are messy. Unlike holding an object, where the connection is solid, pushing relies on friction and the angle of contact. If the robot pushes at the wrong angle, the object might slide away, tip over, or the robot might lose its own balance. For years, engineers struggled to teach robots how to navigate this delicate dance of contact without crashing or failing to move the object at all.

A team of researchers has now developed a new way to teach robots these skills, using a method that guides the robot's learning process from the very beginning. Instead of letting the robot guess randomly until it accidentally finds a way to push an object, the researchers gave it a map of where to touch. They used a computer algorithm to identify the most logical places on an object to make contact, such as the legs of a chair or the handle of a dishwasher. They then built a training system that rewards the robot for finding these specific spots. However, they knew that if the robot only cared about touching the right spot, it might never learn how to actually move the object to its destination. To solve this, they created a learning schedule that starts by focusing heavily on finding the contact point and gradually fades that focus away, forcing the robot to learn how to move the object efficiently once it has established a good grip.

The researchers tested this approach on a four-legged robot equipped with an arm, a machine designed to navigate uneven terrain while manipulating objects. In their simulations, the robot learned to push boxes and transport chairs to specific locations. The results showed that without this guided approach, the robot often failed to even touch the object, or it would push it in a way that caused it to tip over. By using their method, the robot successfully moved the objects in more than ninety percent of the trials. Crucially, the robot learned to adjust its body and arm to keep the object stable, often lowering its own center of gravity to prevent the heavy furniture from falling.

The true test came when the researchers moved the robot out of the computer and into the real world. They placed the robot in front of various chairs it had never seen before, including some with three legs and others with folding mechanisms. Despite the differences in shape and weight, the robot successfully pushed and pulled them to their targets. In one experiment, they added extra weight to a chair, making it heavier than the robot's arm was technically rated to lift. The robot managed to move it anyway by using its legs and the ground for support, proving that it had learned to use its entire body to do the work. The robot also showed an ability to recover from mistakes; if it slipped or lost contact, it would automatically reposition itself and try again without stopping.

The researchers also applied this technique to a more complex task: opening a dishwasher. This object has moving parts, requiring the robot to first grab the handle and then switch to pushing the door panel as it swings open. The robot learned to identify the handle as the starting point and then seamlessly transition to pushing the door until it was fully open. This demonstrated that the method could handle objects that change shape during the task, not just static blocks. The study suggests that by guiding a robot's initial curiosity toward meaningful contact points and then gradually letting it figure out the rest, engineers can teach machines to handle the heavy, awkward, and unpredictable objects found in our daily lives. While the system still relies on external cameras to track the object's position, the core learning strategy proved robust enough to handle real-world variations in weight, shape, and friction, bringing us a step closer to robots that can truly help with the heavy lifting in our homes.

Technical Summary: Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL

Problem Statement
Non-prehensile manipulation (pushing, pulling, sliding) combined with mobile platforms offers a versatile solution for moving heavy or bulky objects in unstructured environments. However, controlling these interactions is challenging due to the hybrid nature of the dynamics, which rely on unilateral contact constraints and friction cones rather than rigid grasps. While Deep Reinforcement Learning (DRL) offers a model-free alternative to traditional Model Predictive Control (MPC), it faces a significant "exploration bottleneck" in locomanipulation tasks. In these long-horizon tasks, random exploration rarely yields valid contact, leading to sparse rewards. Consequently, agents often converge to local minima where they avoid contact entirely to minimize energy or motion penalties, failing to learn the necessary interaction skills. Furthermore, existing methods often struggle with complex, non-convex object geometries where uniform surface sampling yields kinematically infeasible contact points.

Methodology
The authors propose a Contact-Guided Exploration strategy implemented within a Multi-Critic Reinforcement Learning (RL) framework. The core architecture and training process are as follows:

Object-Centric Contact Priors: Instead of uniform sampling, the system utilizes a general-purpose grasping algorithm to generate a set of candidate interaction points (e.g., chair legs, handles) on the object mesh. During training episodes, a specific target contact point is sampled from this set to guide the end-effector.
Multi-Critic Architecture: The framework employs a shared backbone with three specialized critic heads corresponding to distinct reward groups:
1. Task Reward ( $r_{task}$ ): Focuses on goal achievement (object position and velocity).
2. Exploration Reward ( $r_{exp}$ ): A dense reward encouraging the end-effector to reach the sampled contact point.
3. Regularization Reward ( $r_{reg}$ ): Penalizes abrupt motions, energy consumption, and violations of joint limits (e.g., hook height).
Weighted Advantage Mixing: The policy is updated using a composite advantage function, $A_t = \sum w_h A^h_t$ , where $w_h$ represents the weight of each reward group.
Decay Schedule: A critical component is the dynamic scheduling of critic weights. The exploration weight ( $w_{exp}$ ) is linearly annealed from 0.1 to 0.01 during the early stages of training (steps 5k–10k), while the regularization weight increases. This forces the policy to initially prioritize finding meaningful contact and progressively transition to optimizing for task performance and stability, decoupling the exploration phase from the final task objective.
System Control: The high-level policy outputs target arm joint positions, base velocity, and base height. The base height is actively modulated to prevent shoulder joint limits from being reached when the arm extends low. Low-level leg commands are handled by a frozen, pre-trained locomotion policy.

Key Contributions

Contact-Guided Exploration Strategy: A novel approach that guides RL exploration toward meaningful interaction regions using object-centric priors derived from grasping algorithms, rather than relying on uniform sampling or expert demonstrations.
Multi-Critic Formulation with Decay: The implementation of a dedicated exploration critic whose influence is progressively decayed. This allows the agent to learn contact-rich behaviors early on and then refine them into a robust, task-optimal policy without being locked into suboptimal exploration behaviors.
Systematic Evaluation and Hardware Validation: Extensive evaluation across box pushing, chair transportation, and dishwasher opening tasks. Crucially, the chair transportation policy was validated on a real-world quadrupedal mobile manipulator (ALMA), demonstrating zero-shot generalization to unseen object geometries and masses.

Results

Simulation Performance: The proposed method (Multi-Critic PPO + Weight Schedule) achieved a success rate of 94.1% in chair transportation, significantly outperforming standard PPO (which suffered from high "Missed Contact" rates) and fixed-weight Multi-Critic baselines (which suffered from higher "Tipover" rates due to persistent exploration incentives).
Failure Mode Analysis: The method effectively mitigated the "Missed Contact" problem (reducing it to negligible levels) and minimized "Tipover" events (4.4%) compared to baselines that struggled with unstable learning or local minima.
Real-World Deployment: On the ALMA quadruped, the policy achieved a 69.0% aggregate success rate across four unseen IKEA objects (including a folding chair and a three-legged table). The system demonstrated:
- Generalization: Successful manipulation of asymmetrical and non-standard geometries without fine-tuning.
- Reactive Recovery: The ability to autonomously re-plan and re-establish contact after a slip or failed attempt.
- High Payload Handling: Successful transport of a chair with an added 5 kg payload (total 6.5 kg), exceeding the arm's rated static payload by leveraging ground support and mobile base dynamics.
- Disturbance Rejection: Recovery from physical pushes during transport.
Articulated Objects: In the dishwasher opening task, the policy learned to dynamically sequence contacts, switching from the handle to the door panel as the kinematics changed, reducing the time the arm spent near joint limits by 59% compared to standard PPO.

Significance
The paper claims that this approach addresses the fundamental difficulty of sparse rewards in non-prehensile locomanipulation by structuring the learning process. By decoupling exploration and task objectives through a multi-critic architecture with a decay schedule, the system avoids the trade-off between avoiding penalties and finding contact. The work demonstrates that contact-rich, heavy-duty manipulation can be learned from scratch without expert demonstrations, achieving robust performance in simulation and deployable behaviors on physical hardware with varying object morphologies and masses. The authors note that while the results are promising, reliance on external motion capture limits field deployment, and future work could explore adaptive, performance-based curricula to reduce hyperparameter sensitivity.

More like this