💻 computer science

MoDex: A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

MoDex is a two-stage diffusion policy that enables a dexterous hand to sequentially grasp multiple objects without releasing previously held ones by using an opposition space condition to reserve degrees of freedom for future tasks, achieving superior success rates in both simulation and real-world experiments compared to existing baselines.

Original authors: Haofei Lu, Hongjia Liu, Yifei Dong, Florian T. Pokorny, Jens Lundell, Danica Kragic

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haofei Lu, Hongjia Liu, Yifei Dong, Florian T. Pokorny, Jens Lundell, Danica Kragic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot hand as a highly skilled juggler. Usually, when robots try to pick up an object, they use their entire hand—every single finger—to grab just one thing. It's like using a whole team of movers to carry a single coffee cup. Once they have that cup, they have to put it down or let go of it to pick up the next one. They are "all in" on one object, leaving no spare fingers for anything else.

MoDex is a new robot brain that changes the game. Instead of using all its fingers for one grab, it learns to be a strategic saver. It picks up an object using only a few fingers (like just the thumb and index finger), leaving the other fingers free and ready. Then, without letting go of the first item, it reaches out, grabs a second object with a different set of fingers, and then a third. It's like a juggler who catches a ball, holds it in one hand, catches a second ball with the other hand, and then catches a third ball with their feet, all without dropping the first two.

Here is how the paper explains this magic, broken down into simple concepts:

1. The "Opposition Space" (The Rulebook)

The secret sauce of MoDex is something the authors call the Opposition Space (OS). Think of this as a rulebook for every grab.

Before the robot tries to pick up an object, a "rule" tells it: "For this specific object, only use your thumb and middle finger. Leave the ring finger and pinky free."
This ensures the robot doesn't accidentally use a finger it needs for a future object. It's like a chef who only uses a specific set of knives for a specific dish, saving the other knives for later courses.

2. The "Memory" (The Grasp History)

The robot also has a short-term memory called Grasp History.

If the robot is picking up the second object, it remembers: "I'm already holding a ball with my thumb and index finger."
This memory prevents the robot from trying to use those same fingers again. It forces the robot to find a new way to grab the next item using the fingers that are currently "off-duty."

3. The "Training Camp" (Two-Step Learning)

The robot didn't learn this skill overnight. The paper describes a two-step training process, similar to how a human athlete might train:

Step 1: Imitation Learning (Watching the Pros): First, the robot watched thousands of videos of expert human-like demonstrations in a computer simulation. It learned to copy these moves, like a student copying a teacher's handwriting. This gave it a good starting point.
Step 2: Reinforcement Learning (Trial and Error): Then, the robot was put in a "gym" where it had to practice on its own. It was given a reward system:
- Good: Picking up a new object and keeping the old ones from falling.
- Bad: Dropping an object, hitting the table, or wiggling fingers that aren't supposed to move.
- This step fine-tuned the robot, making it much more reliable than just copying the videos.

4. The Results: Simulation vs. Reality

The researchers tested MoDex in two places:

In the Computer (Simulation): They used a virtual robot arm (a Franka Panda) with a dexterous hand (an Allegro Hand). They asked it to pick up three objects in a row. MoDex was much better than other methods, succeeding about 56% of the time on average, while other methods struggled or failed completely as the number of objects increased.
In the Real World: They put the same code on a real robot in a real lab. Even though the robot had never seen real objects before (it only saw simulations), it still worked! It successfully picked up real-world items, though it was slightly less perfect than in the computer (dropping from ~56% to ~35% success on the hardest tasks). This proves the "sim-to-real" transfer works.

What MoDex Can't Do Yet (The Limitations)

The paper is honest about what the robot can't do yet:

No "In-Hand" Magic: Humans can pick up a pen with two fingers, then shift it to a more comfortable grip without letting go. MoDex cannot do this. Once it grabs an object with a specific set of fingers, it sticks with that grip until the end.
No Strategic Planning: The robot doesn't decide which fingers to use or which object to pick up first. Humans decide that; the robot just follows the instructions given to it.

The Big Picture

The paper concludes that current robots are "underutilizing" their fancy, multi-fingered hands. By teaching them to use only a few fingers at a time and save the rest for later, we can make them much better at handling multiple items without dropping them. MoDex is the first system to prove this "sequential saving" strategy works, turning a clumsy robot hand into a careful, multi-tasking juggler.

Technical Summary: MoDex – A Diffusion Policy for Sequential Multi-Object Dexterous Grasping

Problem Formulation

Current dexterous grasping methods often commit all available degrees of freedom (DoFs) of a hand to a single object. While this maximizes stability for that specific grasp, it underutilizes the hand's dexterity and leaves no redundancy for subsequent grasps. Furthermore, existing multi-object grasping approaches frequently rely on known object models, decoupled grasp generation and execution, or fixed object poses, limiting their applicability in dynamic, real-world scenarios.

This work addresses the challenge of sequential multi-object grasping: the task of grasping the $i$ -th target object while securely holding all previously grasped $i-1$ objects, without releasing them. The kinematic constraints of the Allegro Hand used in this study limit the maximum number of graspable objects to three ( $I \le 3$ ). The core difficulty lies in learning a control policy that maps partial observations to actions while dynamically managing which subset of fingers (DoFs) are active for the current object versus those reserved for future objects.

Methodology: MoDex

MoDex is a Diffusion Policy (DP) designed to predict the next gripper pose directly from observations. It operates within a two-stage training framework: imitation learning (IL) pre-training followed by reinforcement learning (RL) fine-tuning.

1. Policy Architecture and Conditioning

MoDex follows an encoder-denoiser architecture (specifically a 1D convolutional U-Net) that processes action chunks over a prediction horizon. The policy is conditioned on three key inputs:

Partial Point Clouds ( $P_t$ ): Raw perception of the environment.
Opposition Space (OS) Vector ( $f_i$ ): A binary indicator vector specifying which fingers participate in the current grasp. This explicitly restricts the active DoFs for the current object, reserving the remaining DoFs for subsequent grasps.
Grasp History Context ( $h_{i-1}$ ): A vector concatenating the OS vectors of all previous grasps. This informs the policy which fingers are already occupied, preventing the reuse of active DoFs for new objects.

The OS and history are concatenated with the noisy action chunk at every denoising step, allowing the policy to reason about kinematic redundancy and finger allocation.

2. Training Pipeline

Stage 1: Imitation Learning (Pre-training): MoDex is pre-trained on a dataset of expert sequential grasping trajectories generated in simulation (Robosuite). A heuristic pose generator and an operational space controller (OSC) produce these demonstrations. The policy is trained using the behavior cloning objective to denoise action chunks.
Stage 2: Reinforcement Learning Fine-tuning: To improve robustness and handle out-of-distribution states, the pre-trained policy is fine-tuned using Diffusion Policy Policy Optimization (DPPO). This treats the denoising process as a nested Markov Decision Process (MDP).
- Reward Function: A dense, task-specific reward ( $r_{base}$ $r_{ba se}$ ) is used, comprising:
  - $r_{succ}$ : A large constant reward for successfully lifting the target while holding all previous objects.
  - $r_{grasp}$ : Encourages active fingers to contact the target.
  - $r_{maintain}$ : Rewards keeping previously grasped objects secure and stationary.
  - $r_{avoid}$ : Penalizes collisions with the table or unnecessary contacts by idle fingers.
  - $r_{ina}$ : A penalty to prevent idle fingers from twitching or drifting, which is crucial for maintaining stability in subsequent grasps.

Key Contributions

MoDex: The first Diffusion Policy for sequential multi-object grasping that directly maps raw observations to grasp actions while preserving kinematic redundancy.
Conditioning Mechanisms: The introduction of Opposition Space (OS) and Grasp History Context as explicit conditioning signals, enabling a single policy to manage complex, multi-stage grasping tasks without releasing held objects.
Data Collection Algorithm: A novel algorithm for collecting stage-indexed expert trajectories that systematically pairs objects with valid OS configurations, ensuring diverse and valid training data.
Comprehensive Evaluation: An extensive benchmarking of MoDex against representative Imitation Learning (BC-RNN, MoDex-BC) and Reinforcement Learning (PPO, SeqDiffuser) baselines in both simulation and real-world hardware.

Experimental Results

The authors evaluated MoDex on a Franka Emika Panda arm equipped with an Allegro Hand across 15 objects of varying geometries (spheres, cylinders, boxes).

Simulation Performance: MoDex achieved the highest success rates across all grasp stages (1, 2, and 3).
- Stage 1: 75.00% (vs. 57.08% for MoDex-BC).
- Stage 2: 49.58% (vs. 50.00% for MoDex-BC, but significantly outperforming baselines like PPO which achieved 0%).
- Stage 3: 45.00% (vs. 42.08% for MoDex-BC).
- Overall Improvement: MoDex improved performance by 2.92–17.92% over the best learning-based baselines in simulation.
Real-World Transfer: MoDex was deployed on real hardware with zero-shot transfer to unseen objects.
- It achieved success rates of 57.78% (Stage 1), 26.67% (Stage 2), and 20.00% (Stage 3).
- This represents an improvement of 6.67–17.78% over the MoDex-BC baseline (pre-trained only, no RL fine-tuning), demonstrating the critical role of DPPO fine-tuning in bridging the sim-to-real gap.
Ablation Studies: Removing the OS condition or Grasp History Context caused significant performance drops in Stages 2 and 3, confirming that explicit finger-allocation signals are essential for sequential grasping. Similarly, removing DPPO fine-tuning resulted in lower success rates, particularly in the real world.

Significance and Limitations

The paper posits that current methods may underutilize the potential of dexterous hands by failing to manage DoF allocation across sequential tasks. MoDex demonstrates that a single diffusion policy, conditioned on opposition space and grasp history, can effectively solve this problem, outperforming methods that rely on decoupled planning or standard RL from scratch.

Limitations:
The authors acknowledge that MoDex does not yet achieve human-level dexterity in two specific areas:

In-hand Regrasping: The policy commits to a single Opposition Space per object and cannot perform in-hand manipulation (e.g., transferring an object from a pinch to a power grasp).
Task Planning: MoDex assumes the OS assignment and the order of object picking are provided as inputs; it does not reason about the optimal OS or picking sequence autonomously.

The work concludes that while MoDex successfully addresses the sequential grasping problem, future work should focus on integrating in-hand manipulation and autonomous task planning.