💻 computer science

Passivity-Centric Safe Reinforcement Learning for Contact-Rich Robotic Tasks

This paper addresses the safety and stability limitations of traditional reinforcement learning in contact-rich robotic tasks by proposing a framework that integrates passivity-aware training with energy-based constraints and a passivity filter for deployment, thereby guaranteeing control stability and improving energy efficiency.

Original authors: Heng Zhang, Gokhan Solak, Sebastian Hjorth, Arash Ajoudani

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heng Zhang, Gokhan Solak, Sebastian Hjorth, Arash Ajoudani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot arm reaching out to touch a wall, not to push it away, but to feel its way through a dark, winding maze. This is the world of contact-rich robotics, where machines must interact physically with their surroundings to solve problems. For years, scientists have used a powerful tool called reinforcement learning to teach robots how to do this. Think of it as a digital trial-and-error process: the robot tries a movement, gets a reward if it succeeds, and learns from its mistakes. While this method has produced impressive results in computer simulations, a major hurdle remains when moving these robots into the real world. The algorithms are excellent at finding a path to a goal, but they often ignore a fundamental rule of physics: energy. If a robot learns a strategy that requires it to pump too much energy into its joints or the environment, it can become unstable, shake violently, or even damage itself and the objects it touches. Ensuring that a robot's movements are safe and stable is not just a theoretical concern; it is a practical necessity for any machine that will ever work alongside humans.

Researchers at the Istituto Italiano di Tecnologia in Genoa, Italy, set out to solve this specific problem. They asked a simple but critical question: can we teach a robot to be not only good at a task but also inherently safe regarding how it uses energy? Their work focuses on a concept called passivity. In plain terms, passivity means that a system cannot create energy out of nothing; it can only store what it receives or dissipate it. A passive robot is one that behaves like a well-damped spring, absorbing shocks rather than amplifying them. The team discovered that standard reinforcement learning policies, which are trained to maximize task success, often fail to respect this rule. In their experiments, these standard policies learned to complete tasks quickly but did so by generating control commands that violated energy limits, leading to instability when deployed in the real world.

To fix this, the researchers developed a new approach that combines the learning power of artificial intelligence with strict energy rules. They created a system with two distinct parts working together. The first part happens during the training phase. Here, they taught the robot to be "passivity-aware." Instead of just rewarding the robot for reaching the exit of a maze, they added invisible constraints that penalized it for using too much energy or changing its stiffness too quickly. This forced the robot to learn a more economical way of moving, discovering strategies that were naturally gentler and more stable. The second part happens when the robot is actually working. Even with the better training, the researchers knew that a computer program could still make a mistake. So, they added a safety filter, a final layer of protection that sits between the robot's brain and its muscles. This filter constantly monitors the energy flow. If the robot tries to make a move that would inject too much energy into the system, the filter automatically scales it back, ensuring the robot stays within safe limits.

The team tested this method in a challenging scenario: a maze exploration task where a robot arm had to find its way out by blindly touching the walls. They compared four different types of robots: one that ignored energy rules entirely, one that was trained to care about energy but had no safety filter, one that had a safety filter but was trained without energy rules, and finally, their new method that combined both energy-aware training and the safety filter. The results were clear. The robot trained without energy rules could finish the maze in simulations, but when they tried to run it on a real physical robot, it failed. It moved too aggressively, applying excessive force at the turns, which caused it to lose control. In contrast, the robots that were trained with energy constraints learned to move more conservatively. They took slightly longer to learn the task during the training phase, but once deployed, they were far more reliable.

When the researchers put the best-trained models to the test in the real world, the difference was stark. The robot using their combined method completed the maze successfully in every single trial, never violating force limits or running out of its energy budget. The standard robot, even when protected by the safety filter, struggled because it had not learned to manage its energy efficiently in the first place. The safety filter alone could stop a disaster, but it could not make the robot efficient. The energy-aware training alone made the robot efficient, but it could not guarantee safety if the robot made a sudden, unpredictable error. Only by combining both did they achieve a system that was both safe and efficient. The researchers found that the robot trained with strict energy limits used its energy budget much more slowly and evenly, allowing it to handle complex turns and obstacles without running out of steam.

This work highlights a crucial shift in how we build intelligent machines. It suggests that for robots to work safely in our physical world, they cannot just be taught to be smart; they must be taught to be physically responsible. By embedding the laws of energy conservation directly into the learning process and backing it up with a real-time safety filter, the researchers have shown a path toward robots that are not only capable but also trustworthy. Their experiments, conducted on a seven-jointed robotic arm, prove that it is possible to teach a machine to navigate a difficult, contact-heavy environment without risking its own stability or the safety of its surroundings. The method does not rely on complex mathematical models of the world that might be wrong; instead, it relies on the robot learning the limits of its own energy through experience, guided by rules that keep it grounded in physical reality.

Technical Summary: Passivity-Centric Safe Reinforcement Learning for Contact-Rich Robotic Tasks

Problem Statement

Reinforcement Learning (RL) has demonstrated significant success in robotic control but faces critical barriers when deployed in real-world, contact-rich environments. Traditional RL policies often prioritize task completion and learning efficiency while neglecting system stability and safety. In contact-rich scenarios, such as wall-following or force-regulated manipulation, policies lacking passivity guarantees can lead to system instability, unbounded energy accumulation, and high-frequency vibrations. These behaviors pose risks to the robot, the environment, and human operators.

While Passivity-Based Control (PBC) is a established method for ensuring stability in physical interactions, its integration into RL remains underexplored. Existing literature often focuses on designing energy tanks or parameters for classical control, with limited investigation into learning-based passive system design. Furthermore, simply bounding total energy is insufficient; the energy flow between the controller and the robot must also be restricted to ensure passivity. The paper identifies a gap in combining RL with PBC to simultaneously address safety, stability, and energy efficiency in contact-rich tasks.

Methodology

The authors propose a Passivity-Ensured Safe RL framework that integrates energy-based constraints into both the training and deployment phases. The approach is built upon a Safe RL baseline that utilizes Variable Impedance Control (VIC) and a safety critic for contact-rich tasks. The methodology distinguishes between four approaches:

Passivity-Agnostic: Standard RL without passivity considerations.
Passivity-Filtered: RL trained traditionally, with a passivity filter applied at deployment.
Passivity-Aware: RL trained with explicit energy constraints to learn passive behavior.
Passivity-Ensured: A combination of passivity-aware training and a passivity filter at deployment.

Core Components

Augmented Energy Tank: The system employs an energy tank ( $E_T$ ) to store and limit energy. Unlike previous works that only bound the total energy, this method introduces a flow limit ( $\dot{E}_T$ ) to restrict the rate of energy injection, addressing the insufficiency of total energy bounding alone.
Passivity-Aware Training (Sec. 2.4): The Constrained Markov Decision Process (CMDP) is extended to include two new constraints:
- Budget Constraint ( $c_{eb}$ ): Triggers if the energy tank drops below a lower bound ( $E_T \leq \underline{E}_T$ ).
- Flow Constraint ( $c_{ef}$ ): Triggers if the energy draining rate exceeds a threshold ( $\dot{E}_T < \underline{\dot{E}}_T$ ).
  These constraints are used to train a safety critic, enabling the agent to learn energy-efficient policies without augmenting the state space with tank variables, thus keeping the state compact.
Passivity-Ensured Control (Sec. 2.5): A passivity layer is applied to the RL agent's output during deployment. This layer acts as a filter:
- It calculates the scaling factor $\alpha$ based on the ratio of allowed energy flow to actual flow.
- It modifies the Cartesian stiffness and control torques ( $\tau_{IC}$ ) to ensure the energy flow condition is met.
- If the energy tank is depleted, stiffness updates are halted or modified to prevent further energy injection, ensuring the system remains passive.

Key Contributions

The paper outlines three primary contributions:

Systematic Comparison: An extensive comparison of different RL-based passive control learning approaches (passivity-agnostic, filtered, aware, and ensured), evaluating their effects on training, deployment, safety, stability, and energy efficiency.
Augmented Passivity Formulation: The design and evaluation of an augmented energy tank-based passivity formulation within RL, specifically incorporating varying energy tank sizes and, crucially, energy flow constraints.
Passivity-Ensured Framework: The introduction of a framework that simultaneously addresses control passivity and contact-rich safety by combining passivity-aware training with a deployment-time passivity filter.

Experimental Results

The method was evaluated on a contact-rich maze exploration task where a robot must navigate an unknown maze by blindly touching walls, requiring precise force regulation. Experiments were conducted in MuJoCo simulation and on a real-world Franka Panda robot arm.

Training Phase

Constraint Impact: Introducing passivity constraints ( $c_{eb}$ and $c_{ef}$ ) directly affected task success rates during training. While overly restrictive constraints (e.g., tight flow limits) initially hindered exploration and success, they forced the agent to learn more conservative, energy-efficient behaviors.
Learning Dynamics: Agents trained with energy constraints (e.g., $E_b4$ ) learned to regulate stiffness more economically, resulting in reduced and more evenly distributed energy expenditure compared to passivity-agnostic agents.

Deployment Phase (Simulation & Real World)

Stability Guarantee: The passivity-ensured approach (passivity-aware training + passivity filter) was the only method to guarantee system passivity. Passivity-agnostic policies, even when successful in task completion, frequently violated energy constraints and exhibited aggressive energy usage.
Performance vs. Efficiency:
- Passivity-Agnostic: Achieved high task success in isolation but depleted energy budgets rapidly and failed in real-world trials due to aggressive force application.
- Passivity-Ensured: Achieved 100% success rate in real-world trials with no constraint violations. While some constrained policies (like $E_b4$ ) converged slower in training, they demonstrated superior energy efficiency and robustness in deployment.
Real-World Validation: In real-world experiments, the passivity-agnostic baseline failed in one out of three trials due to excessive force at a turning point. All passivity-ensured configurations succeeded without violating force or energy constraints. Energy consumption analysis confirmed that passivity-aware policies maintained lower energy usage rates than the agnostic baseline.

Significance and Claims

The paper claims that traditional RL policies are insufficient for contact-rich tasks due to a lack of inherent stability guarantees. The proposed Passivity-Ensured Safe RL framework addresses this by:

Guaranteeing Stability: The deployment-time passivity filter ensures that the system remains passive regardless of the RL agent's output, preventing instability and unsafe behaviors.
Improving Energy Efficiency: By explicitly training the agent with energy constraints, the system learns to use energy more economically, leading to better performance in complex, long-duration tasks.
Bridging the Gap: The work demonstrates that combining learning-based policy optimization with classical passivity formalisms is a viable path toward safe, stable, and efficient robotic deployment in unstructured, contact-rich environments.

The authors conclude that while passivity-aware training improves energy efficiency, the passivity filter is essential for guaranteeing safety during deployment, as learning alone cannot ensure strict passivity due to the stochastic nature of RL and data incompleteness.