⚡ electrical engineering

Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamics

This paper addresses the failure of existing physics-informed goal-conditioned reinforcement learning methods in contact-rich manipulation tasks by introducing contact-aware and hierarchical formulations that selectively apply physics-informed inductive biases to handle hybrid dynamics and nonsmooth value landscapes.

Original authors: Vittorio Giammarino, Anastasios Manganaris, Ahmed H. Qureshi

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vittorio Giammarino, Anastasios Manganaris, Ahmed H. Qureshi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to move a box across a table. Sometimes the box is just sitting there, and the robot has to walk over, grab it, and carry it. Other times, the robot is already holding the box.

This paper is about teaching robots to learn these tasks using Reinforcement Learning (trial and error), but with a special trick: instead of just guessing, the robot uses physics to help it figure out the best path.

Here is the breakdown of what the authors found and how they fixed it, using some everyday analogies.

The Problem: The "One-Size-Fits-All" Map

The researchers were using a method called Physics-Informed Goal-Conditioned Reinforcement Learning (Pi-GCRL). Think of this as giving the robot a "smart map" that tells it how hard it is to get from Point A to Point B.

In simple tasks (like a robot moving in an empty room), this map works great. It assumes that if you want to go somewhere, you can move in any direction, and the "cost" (effort) is the same in all directions. It's like walking on a flat, open field where you can run North, South, East, or West equally well.

But robots that manipulate objects are different.
When a robot is not touching an object, it can move its arm freely, but the object stays stuck where it is. The robot cannot "move" the object directly until it grabs it.

The Analogy: Imagine you are trying to move a heavy sofa.
- Phase 1 (No Contact): You can walk around the sofa freely, but you can't push the sofa. If you try to "push" the sofa while walking away from it, physics says that's impossible.
- Phase 2 (Contact): Once you grab the sofa, you can move it. Now, your movement and the sofa's movement are linked.

The authors discovered that the old "smart map" (the Eikonal equation) was broken for this scenario. It tried to force the robot to believe it could move the sofa even when it wasn't touching it.

The Result: The map got distorted. It told the robot, "You can reach the goal by pushing the sofa from across the room," which is physically impossible. This confused the robot and made it learn slower or fail.

The Solution: Two New Tools

To fix this, the authors introduced two new ideas to make the robot's "map" respect the rules of contact.

1. The "Contact-Aware" Compass

Instead of telling the robot how to move in every direction, this new method only checks the directions the robot can actually move right now.

The Analogy: Imagine you are in a maze with a locked door. The old map said, "You can walk through the door." The new map says, "You can only walk through the door if you have a key (or if you are currently holding the door handle)."
How it works: If the robot isn't touching the object, the map ignores the object's position when calculating the "effort" to move. It only cares about the robot's arm. Once the robot grabs the object, the map updates to include the object's movement. This prevents the robot from trying to do the impossible.

2. The "Manager and Worker" Team (Hierarchical Learning)

The second fix was to split the job into two levels: a High-Level Manager and a Low-Level Worker.

The Manager (High-Level): This part of the brain looks at the big picture. It decides what the robot needs to do next (e.g., "Go to the object," "Grab it," "Move it to the goal"). It doesn't worry about the tiny details of the physics.
The Worker (Low-Level): This part handles the actual movement. Crucially, the Worker only looks at the things it can control right now.
- If the robot isn't holding the object, the Worker ignores the object's position and just focuses on moving the arm to the object.
- If the robot is holding the object, the Worker focuses on moving both together.
The Analogy: Think of a construction site. The Foreman (Manager) tells the Crew (Worker), "Go get that beam." The Crew doesn't worry about the crane's engine or the truck's GPS; they just focus on the specific task of grabbing the beam. By separating the "big picture" from the "immediate physics," the robot learns much faster.

What They Tested

The team tested these ideas in three ways:

A Simple 1D Game: They created a tiny, simple simulation where a robot had to move a block. They showed that the old method got confused and drew a messy map, while their new method drew a clean, correct map.
Complex Simulations: They used a virtual robot arm (a UR5e) to stack multiple cubes. They found that their new methods (especially the "Manager and Worker" team) were much better at stacking cubes than the old methods, which often got stuck or failed.
Real-World Robot: They tried it on a real robot arm in a lab. Even with a small amount of data, the robot using their new "Contact-Aware" and "Hierarchical" methods was much better at picking up an object and placing it in the center of a table compared to the other methods.

The Bottom Line

The paper claims that when robots have to touch and move objects, you can't use a "one-size-fits-all" physics map. You have to be smart about when the robot can move the object and when it can't.

By building a system that understands these "contact modes" (touching vs. not touching) and splitting the thinking into a high-level planner and a low-level controller, robots can learn to manipulate objects much more reliably.

Note: The authors explicitly state that while their method works well in simulations and controlled lab settings, it does not yet guarantee safety for humans or handle complex force limits automatically. They view this as a step toward safer, more robust robot learning, but not a final safety solution.

Technical Summary: Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamics

Problem Statement

Goal-Conditioned Reinforcement Learning (GCRL) aims to learn policies that generalize across arbitrary goals using sparse feedback. While Physics-informed GCRL (Pi-GCRL) has successfully applied optimal-control-inspired inductive biases (specifically Partial Differential Equation constraints) to navigation and object-free goal-reaching, its application to contact-rich robotic manipulation remains problematic.

In contact-rich environments, system dynamics are hybrid, alternating between free-space motion, contact-mediated interaction, and object transport. These transitions induce mode-dependent controllability: certain state coordinates (e.g., object positions) may be locally uncontrollable in specific contact modes (e.g., when not grasping). The authors argue that existing Pi-GCRL methods, which typically apply full-state Eikonal regularization, fail in these settings because the underlying assumptions of the Eikonal equation (isotropic, fully controllable dynamics) are violated. This misalignment causes the value function to be distorted, penalizing valid variations along uncontrollable coordinates and degrading policy performance.

Methodology

The paper proposes a principled analysis of this failure mode and introduces two complementary formulations to address it: Contact-Aware Regularization and Hierarchical Decomposition.

1. Theoretical Analysis: Eikonal Mismatch

The authors formalize the failure of full-state Eikonal regularization in hybrid systems. They model manipulation as a hybrid system where the state $s = (x, z)$ consists of controllable coordinates $x$ and locally uncontrollable coordinates $z$ (e.g., object pose when not grasped).

Proposition 4.1 demonstrates that if the optimal cost-to-go $d^*$ depends on locally uncontrollable coordinates ( $\nabla_z d^* \neq 0$ ), the full-state Eikonal constraint ( $\|\nabla_s d^*\| = 1$ ) contradicts the local Hamilton-Jacobi-Bellman (HJB) characterization.
The Eikonal constraint incorrectly allocates the gradient budget to directions where no instantaneous motion is possible, whereas the HJB equation correctly constrains gradients only along feasible directions.

2. Contact-Aware Physics-Informed Value Learning

To resolve the mismatch, the authors propose replacing the full-state gradient norm constraint with a directional constraint aligned with observed feasible transitions.

Instead of penalizing the full gradient norm, the method approximates the continuous-time dynamics direction using the empirical transition vector $(s' - s)$ .
This yields a Contact-Aware HJB-style residual (Eq. 4):
$\mathcal{L}_{CA-HJB} = \mathbb{E} \left[ \left( \nabla_s d_\theta(s, g)^\top \frac{s' - s}{\|s' - s\| + \epsilon} + 1 \right)^2 \right]$
This formulation ensures that value gradients are only constrained along directions where motion is actually observed, effectively ignoring uncontrollable coordinates during specific contact modes.

3. Physics-informed Hierarchical Flow (Pi-H-Flow)

The authors introduce a hierarchical framework to further align regularization with controllability:

High-Level Policy ( $\pi_{hi}$ ): A flow-based model generates subgoals $x_g$ in a controllable representation $X$ (excluding object-pose coordinates). This separates high-level object-centric reasoning from low-level execution.
Low-Level Policy ( $\pi_{lo}$ ): Operates entirely within the controllable representation $X$ , mapping current state and subgoal to primitive actions.
Regularization: Physics-informed constraints (Eikonal or HJB) are applied only at the low level within $X$ , where the dynamics are fully controllable and isotropic assumptions are more valid.
The high-level flow is trained to generate future controllable states, while the low-level learner uses the contact-aware HJB residual to learn the value function.

Key Contributions

Principled Analysis: The paper identifies and proves that full-state Eikonal regularization is structurally misaligned with hybrid contact dynamics due to locally uncontrollable coordinates, distinct from issues caused by anisotropic dynamics.
Novel Formulations:
- A contact-aware residual that constrains value gradients only along feasible transition directions.
- A hierarchical decomposition that isolates controllable dynamics for physics-informed regularization, separating them from object-centric state variables.
Empirical Validation: The authors validate their approach across offline goal-conditioned manipulation tasks in simulation (OGBench) and on a real-world UR5e robot.

Experimental Results

The proposed methods were evaluated on the OGBench suite (Double, Triple, and Quadruple cube rearrangement tasks) and a real-world pick-and-place task.

Simulation (OGBench):
- Failure of Baselines: Standard Eikonal-regularized methods (Eik-GCIVL, Eik-QRL) failed to converge reliably, confirming the theoretical misalignment.
- Contact-Aware Improvement: The contact-aware HJB variant (HJB-GCIVL) significantly outperformed full-state Eikonal methods, particularly in complex multi-object scenarios.
- Hierarchical Benefit: The hierarchical framework (Pi-H-Flow-HJB-GCIVL) achieved the highest success rates and fastest convergence.
- Ablation: Using the controllable "Joints+Contacts" representation at the low level consistently improved performance (10–20% gain in success rate) over using the full state, validating the benefit of state decomposition.
Real-World Experiments:
- Trained on an offline dataset of ~60k steps collected from a scripted MPC policy.
- The proposed Pi-H-Flow-HJB-GCIVL achieved the highest success rates (75% at 10cm threshold, 60% at 5cm, 50% at 3cm), outperforming both non-physics-informed baselines and non-hierarchical physics-informed variants.
- Qualitative results showed the policy could recover from misaligned grasps and utilize alternative interaction modes (e.g., sliding objects).

Significance and Claims

The paper claims to provide a principled step toward extending Pi-GCRL to contact-rich manipulation. By analyzing the geometric mismatch between Eikonal constraints and hybrid dynamics, the authors demonstrate that naive application of physics-informed biases can be detrimental.

The significance lies in:

Correcting a Theoretical Flaw: Showing that Lipschitz continuity (enforced by Eikonal constraints) is insufficient for hybrid systems and that constraints must respect local controllability.
Practical Framework: Offering a scalable, hierarchical architecture that combines flow-based subgoal generation with contact-aware value learning.
Real-World Applicability: Demonstrating that these theoretical insights translate to improved performance on physical robots using limited offline data.

The authors remain modest regarding limitations, noting that current algorithms remain data-hungry and that the HJB regularizer does not inherently guarantee safety or respect actuator limits. They suggest future work should integrate more explicit physics models and safety constraints.