💻 computer science

Learning Stable In-Grasp Manipulation in a Non-Dropping Action Space

This paper proposes an efficient approach to learning stable in-grasp manipulation by decomposing complex dexterous skills into simpler components that are trained with constraints and guidance derived from classical physics and control theory, thereby overcoming the instability and inefficiency of traditional end-to-end reinforcement learning.

Original authors: Ha Thang Long Doan, Hikaru Arita, Kazuto Nakashima, Kenji Tahara

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ha Thang Long Doan, Hikaru Arita, Kazuto Nakashima, Kenji Tahara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot hand to juggle, spin a coin, or rearrange a deck of cards while holding it. This is the world of dexterous manipulation, a field where scientists try to make robots as nimble as human fingers. For a long time, engineers tried to solve this by writing perfect math equations that described exactly how every finger, spring, and piece of friction should behave. But real life is messy; rubber slips, metal bends, and friction changes, so these perfect math models often fail when the robot actually tries to move.

Recently, scientists started using a different trick called Reinforcement Learning (RL). Think of this like a video game where the robot is a character that learns by trying things over and over. If it drops the object, it gets a "game over" and tries again. If it succeeds, it gets a point. The problem is that without a guide, the robot learns very slowly. It might accidentally discover a "cheat code" where it tosses the object in the air and catches it, or it might drop the object in a way that looks like a mistake but actually teaches it nothing. The robot gets stuck in a loop of unstable, dangerous moves because it doesn't understand the basic rules of physics that keep things from falling.

This paper, titled "Learning Stable In-Grasp Manipulation in a Non-Dropping Action Space," proposes a clever middle ground. Instead of letting the robot learn from scratch or relying on perfect math, the authors suggest giving the robot a "training wheels" system based on solid physics. They take a known, stable method for holding objects and use it to build a safe playground where the robot can learn. Inside this playground, the robot is free to experiment and get better, but it is physically impossible for it to drop the object or break its own fingers. The result is a robot that learns to manipulate objects much faster and more accurately than before, all while staying safe.

The "Training Wheels" for Robot Hands

The authors, a team from Kyushu University in Japan, tackled a specific headache in robotics: how to teach a robot to move an object while holding it without ever letting go. They call this "in-grasp manipulation." Imagine holding a tennis ball in your hand and trying to rotate it so the logo faces you, all without your fingers slipping off.

The paper argues that trying to teach a robot this skill from scratch using pure Reinforcement Learning is like teaching a toddler to drive a race car on a cliff edge. The robot might eventually figure out how to drive, but it will likely crash a lot first. In the world of RL, this is called the "long-tail instability problem." The robot finds weird, unstable ways to move that look like they work for a second but eventually cause the object to drop. The authors found that when the robot drops the object, the learning process gets confused because it doesn't know why it failed, and it wastes time trying to fix things that shouldn't be broken in the first place.

The Solution: A Safe Sandbox

To fix this, the authors didn't throw away the math; they just changed how they used it. They started with a proven physics method called FTODG (Fingers-Thumb Opposability-based Dynamic Grasping). Think of FTODG as a very strict, very smart teacher who knows exactly how to hold an object so it never falls. This teacher uses specific rules about force and balance to keep the object stable.

However, this "teacher" has a flaw: it's a bit rigid. It can hold the object perfectly, but it's not great at moving it precisely to a new spot or spinning it exactly how you want. It's like a teacher who can keep you from falling off a bike but can't teach you how to do a wheelie.

The paper's big idea is to combine the teacher with the student. They created a system called TSIGL (Theoretically Stable In-Grasp Learning). Here is how it works:

The Safe Zone: They built a "stable action space." Imagine a sandbox with high walls. Inside the sandbox, the robot is free to move its fingers and try different ways to spin or move the object.
The Safety Net: They used a mathematical tool called Control Barrier Functions (CBFs). Think of these as invisible force fields around the sandbox. If the robot tries to make a move that would cause it to drop the object or squeeze it too hard, the force field instantly stops the move and gently pushes it back to a safe position.
The Learning: The robot (using RL) is allowed to explore only inside this safe zone. It learns to find the best way to move the object by tweaking the force and speed of its fingers, but it never has to worry about the "game over" scenario of dropping the object.

What They Found

The team tested this idea in a computer simulation using a robotic hand called the "Shadow Dexterous Hand." They taught the robot three skills: holding an object with three fingers, moving the object to a new spot without letting go, and spinning the object.

The results were clear and impressive. When they let the robot learn without their "safe sandbox" (using standard end-to-end learning), the robot dropped the object thousands of times. In one test, the standard method dropped a can 3,138 times while trying to learn to spin it. In contrast, the TSIGL method with the safety net dropped the object zero times.

Because the robot never wasted time on failed attempts, it learned much faster. In the simulation, the TSIGL robot reached a streak of 42 successful moves in a row, while the standard method struggled to get even one or two in a row before dropping the object. Furthermore, once the robot learned the skill, it was actually better at it than the original physics teacher (FTODG) was on its own. The robot learned to adjust its grip just right to move the object more accurately than the rigid math model could.

Why It Matters

The authors are careful to note that this was tested in a simulation, not yet in the real world with a physical robot. However, the simulation showed that by combining the safety of physics with the flexibility of learning, robots can learn complex skills much more efficiently.

The paper explicitly rules out the idea that we should just rely on Reinforcement Learning alone to solve these problems, showing that without the "safety net," the learning is too slow and unstable. They also show that simply adding a penalty for dropping the object (telling the robot "don't drop it") isn't enough; the robot will still find loopholes to drop it while trying to learn. The only way to truly fix the problem, they found, is to physically constrain the robot's actions so that dropping the object is mathematically impossible.

In short, this paper suggests that the best way to teach a robot to be dexterous isn't to let it crash and burn, but to give it a safe playground where it can practice until it masters the skill, all while being protected by the laws of physics.

Technical Summary: Learning Stable In-Grasp Manipulation in a Non-Dropping Action Space

Problem Statement
The paper addresses the challenge of achieving stable and accurate in-hand manipulation with multifingered robotic hands. Traditional approaches rely on analytic models constrained by strong assumptions regarding physical parameters (e.g., fixed friction, stiffness, damping), which often lead to inaccurate regulation when applied to objects with varying properties. Conversely, end-to-end Reinforcement Learning (RL) avoids complex modeling but suffers from inefficiency and a "long-tail" instability problem. In pure RL, agents may discover "unstable" actions—such as unnecessary finger releases or object tossing—that appear successful in the short term but cause eventual object dropping. These failures are difficult to identify and correct due to conflicting learning objectives and the lack of explicit stability guarantees.

Methodology: The TSIGL Framework
The authors propose the Theoretically Stable In-Grasp Learning (TSIGL) framework, which decomposes dexterous skills into analyzable components guided by classical physics and control theory. The core methodology integrates the Fingers-Thumb Opposability-based Dynamic Grasping (FTODG) control theory with RL, constrained by Control Barrier Functions (CBFs).

Stable Action Space via FTODG:
The framework utilizes FTODG to define a "stable action space." FTODG relies on a Virtual Frame (VF) generated by the fingertips to compute stable grasp signals ( $u_{stable}$ ) that ensure force/torque equilibrium. It employs the Principle of Manipulation Signal Superposition, where the final control signal ( $u_i$ ) is the sum of the stable grasp term, a position manipulation term ( $u_{pos}$ ), and an orientation manipulation term ( $u_{ori}$ ). This design ensures that the object remains stable in the hand even while being repositioned or reoriented, using only internal joint sensor data (external sensorless).
RL for Accuracy:
While FTODG guarantees stability, it may lack accuracy due to model assumptions and varying object properties. TSIGL addresses this by using RL to learn the specific gain parameters ( $P_i, O_{n,i}, O_{t,i}$ ) that determine the fingertip forces and moments for repositioning and reorientation. The RL policy ( $\pi_{RL}$ ) maps observations to these gain parameters.
CBF Filtering for Safety:
To prevent the RL agent from exploring actions that violate the stability conditions of FTODG, the framework employs a Control Barrier Function (CBF) filter. The raw actions ( $K_{RL}$ ) from the RL policy are processed through a quadratic programming (QP) solver. This filter minimally adjusts the actions to satisfy strict implementation requirements:
- Contact Maintenance: Ensuring fingertips do not break contact with the object.
- Force Limits: Constraining fingertip forces to remain within a safe range ( $F_{min}$ to $F_{max}$ ) to prevent slippage or excessive force.
- Static Model Approximation: The filter uses a simple static model to estimate future fingertip forces based on current controller torques, allowing for safety constraints even in the presence of sensor noise and latency.

Key Contributions

Stable Action Space: The authors develop an FTODG-based action space grounded in safe RL principles, ensuring that exploration occurs only within theoretically stable bounds.
TSIGL Framework: They propose a novel learning architecture that leverages CBFs to constrain RL exploration within the stable action space, effectively decoupling stability (handled by theory) from accuracy (handled by learning).
Empirical Validation: The study demonstrates that TSIGL prevents object dropping and unstable finger motions (such as "quick release and catch" or "unstable regrasp" behaviors) while achieving higher sample efficiency and accuracy compared to baseline end-to-end RL methods.

Results
The framework was evaluated in the NVIDIA Isaac Lab simulation environment using a Shadow Dexterous Hand and various objects (cubes, cylinders, cans, etc.) under conditions of sensor noise, latency, and randomized friction.

Stability: In Skill 1 (precision grasp), the FTODG controller successfully withstood external forces up to 8.6 N without slippage.
Sample Efficiency: In Skill 2 (position manipulation), the TSIGL framework achieved 0 object drops during exploration and converged to approximately 44 consecutive successes. In contrast, baseline methods using soft penalties (Soft 1, 2, 3) suffered from learning divergence, high drop counts (up to 738 drops), and significantly lower success rates.
Comparison with End-to-End RL: In Skill 3 (orientation manipulation), standard end-to-end RL methods dropped objects thousands of times (e.g., 3,138 drops) and failed to achieve consistent success. The TSIGL framework achieved 0 drops and over 42 consecutive successes.
Accuracy Improvement: When deployed, the learned TSIGL policies outperformed controllers with fixed FTODG parameters. Consecutive success rates improved by 130–155% for position manipulation and 39–81% for orientation manipulation across various objects.

Significance and Claims
The paper claims that stability should not be left solely to RL optimization. By shifting the perspective to decouple skills into simpler components and guiding RL with scientific knowledge (specifically FTODG and CBFs), the authors demonstrate that stable in-grasp manipulation can be learned efficiently. The significance lies in the ability to explore complex manipulation skills without the "long-tail" instability problem, achieving zero object drops during training while simultaneously improving manipulation accuracy beyond what is possible with the theoretical controller alone. The authors note that while the work is currently simulation-based, the framework offers a promising path toward safe, sample-efficient dexterous manipulation in unstructured environments.

The "Training Wheels" for Robot Hands

The Solution: A Safe Sandbox

What They Found

Why It Matters

More like this