💻 computer science

Inverse Manipulation through Symbolic Planning and Residual Operator Learning

This paper proposes a hybrid framework for robotic inverse manipulation that combines automatically extracted symbolic operators with residual Reinforcement Learning to refine coarse symbolic plans into physically grounded skills, demonstrated effectively on the ManiSkill3 PushCube task.

Original authors: Yigit Yildirim, Giuseppe Rauso, Riccardo Caccavale, Alberto Finzi

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yigit Yildirim, Giuseppe Rauso, Riccardo Caccavale, Alberto Finzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to push a cube across a table. That's the "forward" task. Now, imagine you need to teach that same robot how to undo that exact action and get the cube back to where it started.

This sounds simple, but it's actually a tricky puzzle for a robot. If you just tell the robot to "play the movie backward," it often fails. Why? Because if the robot didn't grab the cube (it just pushed it), it can't simply pull its arm back to undo the push. The cube is now sitting somewhere else, and the robot needs a new plan to get it back.

This paper presents a clever "hybrid" system that solves this problem by combining logical planning (like a chess player thinking ahead) with trial-and-error learning (like a baby learning to walk).

Here is how their system works, broken down into simple steps:

1. The "Magic Translator" (Symbolic Extraction)

First, the robot watches a human (or a script) perform the forward task, like pushing a cube. Instead of just memorizing the arm movements, the system acts like a translator. It watches the scene and writes down a simple "recipe" or rule in plain logic.

Before: "The cube is at the start."
After: "The cube is at the finish."
The Rule: "If the cube is at the start, I can push it to the finish."

2. The "Reverse Recipe" (Inverse Planning)

Once the robot has the rule, it automatically writes a "reverse recipe." It looks at the rule and asks, "What do I need to do to get back to the start?"

It realizes it needs to: "Put the cube back at the start," "Make sure the gripper is open," and "Make sure the robot arm is near the cube."
The robot then uses a standard planner (like a GPS) to figure out a sequence of basic moves to try and fix the situation. For example, it might try to "Pick up the cube" and "Place it at the start."

3. The "Good Enough" Handoff

Here is where the magic happens. The robot's basic planner is good, but not perfect. It might successfully pick up the cube and drop it roughly near the start, but it might be off by a few inches.

In many other systems, this would be a failure.
In this system, the robot stops and says, "Okay, I got the cube close enough. Now, I need to fine-tune the position."

4. The "Fine-Tuner" (Residual Learning)

This is the second half of the team: a Reinforcement Learning (RL) agent. Think of this as a highly skilled micro-adjuster.

The system tells the RL agent: "You must move the cube the remaining distance to the exact starting spot, but do not mess up the things we already fixed (like keeping the gripper open)."
The RL agent tries thousands of tiny movements. If it moves the cube closer, it gets a "treat" (a reward). If it knocks the cube away from the spot we already fixed, it gets a "scolding" (a penalty).
Eventually, the RL agent learns the perfect, tiny nudges needed to slide the cube exactly into place.

The Result: A Perfect Undo

The authors tested this on a task called "PushCube."

The Problem: The robot pushed a cube.
The Old Way (Just reversing): Couldn't do it because the robot didn't grab the cube.
The "Logic Only" Way: Got the cube close, but missed the target by about 17 millimeters (about the width of a pencil eraser).
The New Hybrid Way: The logic part got the cube close, and the learning part nudged it the rest of the way. The result? The cube ended up within 1.4 millimeters of the start, with a 90% success rate.

The Big Picture

The paper claims that by splitting the job into two parts—Symbolic Planning for the big picture and Residual Learning for the fine details—you can teach robots to undo complex tasks that they couldn't reverse before. It turns a "rough guess" into a "physically perfect undo."

In short: The robot uses its brain to figure out the general strategy to undo a task, and then uses its "muscle memory" (learned through trial and error) to make the final, precise adjustments to get it exactly right.

Technical Summary: Inverse Manipulation through Symbolic Planning and Residual Operator Learning

Problem Statement

The paper addresses the challenge of skill inversion in robotic manipulation: the ability to restore a physical environment to its state prior to a forward execution. The authors argue that simply rewinding motor trajectories is insufficient for many tasks, particularly those involving non-prehensile interactions (e.g., pushing) where the robot does not grasp the object. While symbolic planning can generate inverse sequences, it often fails to fully restore continuous physical states due to limited action primitives and precision requirements. Conversely, pure trajectory reversal is often physically inapplicable. The core problem is bridging the gap between high-level symbolic restoration objectives and low-level continuous control to achieve physically grounded inverse skills.

Methodology

The proposed framework is a hybrid system that combines symbolic planning with Residual Reinforcement Learning (RL). The process operates in three main stages:

1. Symbolic Operator Extraction

The system analyzes forward skill demonstrations to automatically extract symbolic operators in STRIPS formalism (Fikes and Nilsson 1971).

Soft Predicates: A predefined repertoire of predicates (e.g., AT POSE, GRIPPER OPEN) is grounded on the scene graph using soft, differentiable geometric predicates. These yield continuous scores $V_p(s) \in [0, 1]$ based on signed margins $m_p(s)$ .
Operator Definition: An operator $o = \langle Pre, Add, Del \rangle$ is constructed by analyzing the start and end states of demonstrations. Predicates with high start scores form the precondition set ($Pre$), while changes in scores determine the add ($Add$) and delete ($Del$) effects.

2. Inverse Target Generation and Symbolic Planning

The inverse objective is defined not as a time-reversal, but as a restoration problem. The target state $I(o)$ is constructed to satisfy the original preconditions, restore the deleted effects, and negate the added effects:
$I(o) = Pre \cup Del \cup \neg Add$
A Breadth-First Search (BFS) planner attempts to satisfy $I(o)$ using a limited set of scripted action primitives (e.g., PICK, PLACE).

3. Residual Operator Learning

Since the symbolic planner may only partially satisfy the inverse target, the system partitions the remaining objectives:

Fence Set ( $F$ ): Predicates already satisfied by the symbolic plan ( $\pi_{sym}$ ) that must be preserved.
Active Residual Set ( $A$ ): Unresolved predicates that constitute the residual objective.

A Residual RL policy ( $\pi_{res}$ ) is trained to satisfy the active set $A$ while maintaining the fence set $F$ .

Reward Function: The reward is derived directly from the symbolic plan's predicates.
- For active predicates ( $A$ ): A signed normalized margin using $\tanh$ to provide bounded rewards for restoration.
- For fence predicates ( $F$ ): A one-sided penalty using $\min(0, \cdot)$ to penalize violations without rewarding "do-nothing" behavior.
Execution: The final inverse skill is the concatenation of the symbolic sequence $\pi_{sym}$ and the learned residual policy $\pi_{res}$ .

Key Contributions

The paper outlines three primary contributions:

Hybrid Formulation: Addressing skill inversion as a hybrid symbolic-continuous restoration problem, deriving inverse targets from symbolic forward operators extracted from demonstrations rather than relying on temporal reversal.
Residual Operator Learning Framework: Introducing a mechanism where RL is activated only when symbolic planning cannot fully satisfy the inverse objective, effectively handling the "residual" part of the task.
Predicate-Derived Rewards: Deriving RL reward signals directly from the symbolic plan's operator effects, eliminating the need for manual reward engineering for each specific task.

Experimental Results

The framework was evaluated on the ManiSkill3 PushCube task using a Franka Panda manipulator. The forward skill involved pushing a cube to a goal position. The inverse task required restoring the cube to its original source position.

Performance: The hybrid approach (Symbolic + RL) achieved a 90% success rate within a 1 cm tolerance, with a mean precision of 1.4 mm.
Comparisons:
- Symbolic Prefix Only: Achieved only 10% success, leaving a mean residual distance of 16.6 mm.
- Symbolic + Random Action: Achieved 50% success with high variance (18.8 mm mean), indicating that random exploration occasionally succeeds by chance but lacks reliability.
- Pure Trajectory Inverse: Deemed inapplicable due to the non-prehensile nature of the forward push.
Ablation Studies: Experiments confirmed that using a one-sided $\tanh$ penalty for fence predicates was critical. Removing the penalty led to the agent staying still (saturated rewards), while removing the $\tanh$ function entirely caused value function divergence and large reward drifts.

Significance and Claims

The paper claims that predicate-derived residual control can transform an approximate symbolic inverse into a physically grounded inverse skill. By treating inverse skills as distinct symbolic operators with their own restoration objectives, the system avoids the limitations of time-reversal assumptions found in prior work (e.g., Nair et al. 2020; Edwards et al. 2018).

The authors position this work as a step toward enabling human-robot collaboration and assembly-disassembly lines where the ability to "undo" tasks is essential. They note that while the system currently relies on hand-designed predicates and scripted low-level primitives, the results suggest that inverse skills can be effectively solved as a planning problem followed by a residual RL problem with a derived reward. The paper remains modest regarding its scope, acknowledging limitations in predicate design and the reliance on scripted primitives, and suggests future work in learning these components from demonstrations.