RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation

The paper proposes RoTri-Diff, a diffusion-based imitation learning framework that explicitly models the spatial triadic relationship between two robot arms and an object to generate stable, coordinated bimanual manipulation trajectories, outperforming state-of-the-art baselines in both simulation and real-world tasks.

The Gist

Imagine you are trying to move a heavy, fragile cake from a table to a shelf. You need two people (or two robot arms) to do it. One person has to tilt the cake slightly, while the other person slides their hand underneath to catch it.

If the two people don't talk to each other, or if they only focus on their own hands without looking at the cake, disaster strikes: the cake slips, the hands collide, or the cake gets crushed.

This is exactly the problem the paper RoTri-Diff solves. Here is the breakdown in simple terms:

1. The Problem: The "Clumsy Dancers"

Existing robot learning methods are like clumsy dancers. They fall into three bad habits:

• The "Key Pose" Dancer: They only look at the start and finish points (like a dance instructor saying "Start here, end there"). They don't care about the steps in between, so they often trip over their own feet (collide) or drop the object.
• The "Continuous Flow" Dancer: They try to memorize every single tiny movement perfectly. But if the situation changes slightly (like the cake is a millimeter to the left), they get confused and fail because they are too rigid.
• The "Object-Only" Dancer: They focus entirely on the cake, ignoring how their own hands are positioned relative to it. They might try to grab the cake while their other hand is already in the way.

The Result: Robots fail at tasks requiring fine coordination, like picking up a plate where one arm tilts it and the other grabs it.

2. The Human Secret: The "Triadic Triangle"

When humans do this task, we don't just think about "My Left Hand" and "My Right Hand." We instinctively think about a triangle formed by three points:

1. Left Hand
2. Right Hand
3. The Object (The Cake/Plate)

We constantly adjust this triangle. If the cake tilts, we instantly know how to move our hands to keep the triangle stable. We are aware of the relationship between all three, not just the parts.

3. The Solution: RoTri-Diff

The authors created a new AI framework called RoTri-Diff that teaches robots to think like humans by modeling this "Triadic Interaction."

• RoTri (Robot-Object Triadic Interaction): This is the robot's "mental map" of that triangle. Instead of just knowing where the hands are in the room, the robot calculates the exact 3D relationship between Hand A, Hand B, and the Object. It treats them as a single, connected unit.
• Diffusion Model: Think of this as a "sculptor." Imagine a block of marble (random noise). A sculptor chips away the excess to reveal a statue. A diffusion model starts with a random, messy plan for how the robot should move, and it "chips away" the bad ideas step-by-step until a perfect, smooth, collision-free movement plan remains.

4. How It Works (The Recipe)

RoTri-Diff uses a hierarchical (layered) approach, like a construction project:

1. The Blueprint (Keyposes): First, it figures out the major milestones (e.g., "Arm A must be here, Arm B must be there").
2. The Motion (Object Flow): It predicts how the object will move (e.g., "The plate will tilt 15 degrees").
3. The Glue (RoTri): This is the magic sauce. It constantly checks the "triangle" between the two arms and the object. It ensures that as the robot moves, the arms never crash into each other and the object never slips.

It combines all three signals into one powerful brain that generates a smooth, continuous dance for the robot arms.

5. The Results: From Simulation to Reality

The team tested this on a computer simulation with 11 different difficult tasks (like putting a laptop in a drawer or sweeping dust into a pan) and then on real physical robots.

• In the Computer: It beat the previous best robots by a huge margin (10.2% better success rate). It solved tasks that other robots failed completely, like the tricky "Pick Plate" task.
• In the Real World: They tested it on real robot arms with real cameras. It successfully picked up tomatoes, washed plates, and lifted heavy baskets without dropping anything or crashing.

The Bottom Line

RoTri-Diff is like giving a robot a "sixth sense" for spatial relationships. Instead of just seeing two arms and an object as separate things, it sees them as a connected team. By explicitly teaching the robot to maintain the geometric "triangle" between its hands and the object, it can perform delicate, coordinated tasks that were previously impossible for machines.

In short: It stopped robots from being clumsy dancers and taught them to be a synchronized, aware dance troupe.

Technical Summary

Here is a detailed technical summary of the paper "RoTri-Diff: A Spatial Robot–Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation."

1. Problem Statement

Bimanual manipulation requires precise, continuous coordination between two robotic arms to perform complex tasks (e.g., lifting a tray, picking a plate). While Imitation Learning (IL) is the dominant paradigm for acquiring these skills, existing approaches suffer from critical limitations:

• Robot-Centric Methods: Often rely on sparse keyposes (leading to poor intermediate state control and collisions) or dense continuous actions (leading to trajectory overfitting and weak perception).
• Object-Centric Methods: Incorporate object motion but fail to explicitly model the dynamic geometric relationship between the two arms and the object.
• The Core Gap: Current systems lack spatial triadic awareness. They do not explicitly reason about the relative 6D poses between the left arm, the right arm, and the manipulated object simultaneously. This leads to failures in tasks requiring fine-grained coordination, such as one arm tilting an object while the other grasps it, resulting in dropped objects, self-collisions, or unstable grasps.

2. Methodology: RoTri-Diff

The authors propose RoTri-Diff, a hierarchical diffusion-based imitation learning framework that explicitly models the Robot–Object Triadic Interaction (RoTri).

A. Core Representation: RoTri

The central innovation is the RoTri vector, which encodes the relative 6D poses between the two end-effectors and the object.

• Definition: $R_t = [p_{left \to right}, p_{left \to obj}, p_{right \to obj}] \in \mathbb{R}^{21}$.
• Structure: It consists of three 7-dimensional components (3D position + 4D quaternion) representing the relative pose of the left arm to the right arm, the left arm to the object, and the right arm to the object.
• Function: This creates a continuous triangular geometric constraint, allowing the policy to maintain stable spatial relationships and avoid collisions dynamically.

B. Hierarchical Diffusion Architecture

RoTri-Diff operates as a hierarchical diffusion model that integrates three guidance signals:

1. Robot Keyposes: High-level waypoints for long-horizon planning.
2. Object Pointflow: The predicted motion of the object's point cloud (to handle occlusion and dynamics).
3. RoTri Relationship: The continuous spatial constraints between arms and object.

The action prediction process involves three stages:

1. Simultaneous Prediction: The model predicts the future Object Pointflow and a continuous RoTri segment (the evolution of the triadic relationship).
2. Keypose Generation: Using the predicted pointflow and the RoTri state at specific keypose timesteps, the model generates target end-effector poses (keyposes).
3. Continuous Action Generation: The model integrates the predicted pointflow, the full RoTri segment, and the generated keyposes to denoise and generate a dense sequence of continuous actions.

C. Visual Perception & Training

• Perception: Uses a 3D semantic feature extractor (combining DINOv2 for semantic features and PointNet++ for geometric compression) to process multi-view RGB-D inputs.
• Training Strategy: The model learns to predict the change ($\Delta R_t$) in the RoTri representation incrementally, rather than absolute poses, shifting focus to relative interaction dynamics.
• Loss Function: A weighted sum of losses for continuous actions, keyposes, object pointflow, and RoTri prediction changes.

3. Key Contributions

1. RoTri Representation: Introduction of a novel triadic interaction representation that explicitly encodes the relative 6D poses of two arms and an object, establishing continuous geometric constraints for stable coordination.
2. RoTri-Diff Framework: A hierarchical diffusion model that synergistically combines robot keyposes, object dynamics, and RoTri constraints. It is the first bimanual IL framework to integrate all three signals (Keyposes, Object Movement, and Robot-Object Interaction).
3. State-of-the-Art Performance: Extensive validation showing superior performance in both simulation and real-world scenarios, particularly in tasks requiring fine-grained, asynchronous coordination.

4. Experimental Results

Simulation (RLBench2 Benchmark)

• Tasks: Evaluated on 11 representative bimanual tasks covering symmetric, synchronous, and asynchronous coordination (e.g., Pick Plate, Handover Item, Lift Tray).
• Performance: RoTri-Diff achieved an average success rate of 80.9%, outperforming the previous state-of-the-art (PPI) by 10.2%.
• Key Wins:
  • Pick Plate: Achieved 40.7% vs. 0.0% for PPI (which failed due to lack of triadic awareness).
  • Handover (Hard): 52.3% vs. 15.0% for AnyBimanual.
  • Put Item into Drawer: 87.0% vs. 52.7% for 3D Diffuser Actor.

Real-World Experiments

• Setup: Two xArm6 robots with multi-view cameras (Intel RealSense) performing four challenging tasks: Pick Tomato & Banana, Pick Plate, Wash Plate, and Lift Basket.
• Results:
  • Pick Tomato & Banana: 5/5 success (Symmetric coordination).
  • Pick Plate: 3/5 success (Asymmetric/Sequential: one arm tilts, other grasps).
  • Wash Plate: 4/5 success.
  • Lift Basket: 4/5 success (Synchronous lifting under load).
• Significance: Demonstrated robustness in handling strict spatial constraints and temporal dependencies in unstructured real-world environments.

Ablation Studies

• Hierarchy Necessity: Removing either the keypose module or the continuous action module significantly degraded performance, proving that both high-level planning and low-level execution are required.
• Guidance Density: Dense, per-timestep RoTri guidance was found to be superior to sparse guidance, preventing error accumulation and drift.

5. Significance and Conclusion

RoTri-Diff addresses a fundamental gap in bimanual robotics: the lack of explicit modeling of the spatial relationship between two agents and an object. By treating the robot-object system as a triad and enforcing geometric constraints through a diffusion process, the method achieves human-like stability in complex manipulation tasks.

• Impact: It enables robots to perform tasks previously considered too unstable for current IL methods, such as tilting an object with one arm while the other grasps it.
• Limitations: The current approach relies on rigid-body assumptions and accurate 6D pose estimation, limiting its application to deformable objects or highly unstructured environments.
• Future Work: Extending the triadic representation to handle deformable objects and enabling cross-embodiment transfer.

In summary, RoTri-Diff represents a significant step forward in robotic dexterity, moving beyond simple arm-centric or object-centric views to a holistic triadic interaction perspective.