🤖 AI

HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions

HeteroGenManip is a task-conditioned, two-stage framework that enhances generalizable robot manipulation for heterogeneous objects by decoupling contact point localization from interaction trajectory planning, utilizing foundation-model-guided grasping and a multi-model diffusion policy to achieve significant performance gains in both simulation and real-world scenarios.

Original authors: Zhenhao Shen, Zeming Yang, Yue Chen, Yuran Wang, Shengqiang Xu, Mingleyang Li, Hao Dong, Ruihai Wu

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhenhao Shen, Zeming Yang, Yue Chen, Yuran Wang, Shengqiang Xu, Mingleyang Li, Hao Dong, Ruihai Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do chores. The paper introduces a new system called HeteroGenManip (let's call it "The Smart Butler") designed to help robots handle a very tricky problem: dealing with a mix of different kinds of objects at the same time.

Think of a robot's day. It might need to pick up a rigid coffee mug (hard, doesn't change shape), a deformable t-shirt (soft, floppy, changes shape), and a hinged cabinet door (moves in specific ways). Most robots are great at one type but get confused when you mix them up.

Here is how this new system works, broken down into simple concepts:

The Two Big Questions

To do any task, a robot has to answer two questions:

Where to touch? (Where should the robot grab the object?)
How to move? (Once it's holding it, how does it move it to the goal?)

Old robot brains often try to answer both questions at once in one giant, messy step. This is like trying to drive a car while simultaneously learning how to park it; if you make a small mistake at the start, the whole trip goes wrong.

The Solution: A Two-Step Dance

The authors propose splitting the job into two distinct steps, like a dance routine with a specific partner for each move.

Step 1: The "Where to Touch" Guide (The Matchmaker)

Before the robot even tries to move, it needs to know exactly where to grab the object.

The Problem: A t-shirt looks different every time you throw it on the floor. A mug handle might be on the left or right.
The Fix: The system uses a "Matchmaker." It looks at a picture of how a human did the task before (the demonstration) and finds the "soulmate" point on the new object.
The Analogy: Imagine you are looking for a specific button on a coat. Even if the coat is wrinkled or a different color, the Matchmaker says, "That button right there is the one, just like the one in the photo." It uses a pre-trained "brain" (a foundation model) that is really good at recognizing specific parts of objects, ensuring the robot grabs the right spot every time, no matter how the object is twisted.

Step 2: The "How to Move" Guide (The Specialized Chefs)

Once the robot has a firm grip, it needs to plan the path to move the object.

The Problem: Moving a rigid box requires a different "brain" than moving a floppy shirt. A rigid box doesn't stretch; a shirt does. If you use the same brain for both, it gets confused.
The Fix: The system has a Menu of Chefs.
- If the object is a rigid mug, it calls the "Rigid Chef" (a specific AI model trained on hard objects).
- If the object is a floppy shirt, it calls the "Deformable Chef" (a model trained on soft, stretchy things).
The Analogy: Think of a restaurant. You wouldn't ask a sushi chef to grill a steak, and you wouldn't ask a grill master to fold origami. This system is smart enough to route the "steak" (rigid object) to the grill master and the "origami" (soft object) to the sushi chef. They then work together to figure out the perfect path to move the object without dropping it or tearing it.

Why It's Better (The Results)

The paper tested this "Smart Butler" in two ways:

In the Simulation (The Video Game): They created a digital world with many different tasks, like hanging a shirt on a line, stacking clothes, or putting a mug in a cabinet.
- The Result: The new system was 31% better than the best existing robots. It didn't get confused when the objects looked different or were in new places.
In the Real World (The Actual Kitchen): They tested it on a real robot arm with real objects (hanging tops, inserting hangers, placing mugs).
- The Result: It succeeded 76.7% of the time on new, unseen objects, while other methods only succeeded about 33-40% of the time.

The Bottom Line

The paper claims that by splitting the job (finding the grab point first, then planning the move) and using specialized experts for different types of objects (hard vs. soft), robots can finally handle the messy, mixed-up reality of our world much better than before. It's like giving the robot a specialized toolkit instead of a single, blunt hammer.

Technical Summary: HeteroGenManip

Problem Statement

Generalizable robotic manipulation involving heterogeneous object interactions (e.g., rigid-deformable, rigid-articulated, deformable-deformable) remains a critical challenge. Existing foundation-model-based approaches typically rely on end-to-end learning that conflates two distinct stages: "where to manipulate" (contact point localization) and "how to manipulate" (trajectory planning). This conflation leads to error accumulation in long-horizon tasks and redundant motion modeling. Furthermore, current methods often employ a single uniform foundation model for all object types. This fails to capture category-specific features, as models optimized for rigid objects struggle with the non-rigid shape changes of deformable objects, and vice versa. Consequently, these approaches exhibit poor generalization across diverse physical properties and arbitrary shape variations.

Methodology

The authors propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution. The framework addresses the "where" and "how" challenges through two synergistic components:

1. Foundation-Correspondence-Guided Grasp

This module addresses the "where to manipulate" question by leveraging structural priors to identify optimal contact points across heterogeneous objects.

Mechanism: It utilizes a pre-trained, category-specific foundation model ( $M_C$ ) corresponding to the object's category (e.g., rigid or deformable).
Process: The model extracts point-wise structural and semantic features from both demonstration and target point clouds. By computing the cosine similarity between features, it identifies the exact counterpart of a pre-defined manipulation point on the target object.
Orientation: For rigid objects with rotational variations, auxiliary reference points are matched to construct a directional vector for 3D orientation. For deformable objects in flat states, fixed orientation vectors are used.
Benefit: This spatial anchoring decouples localization from trajectory planning, significantly reducing pose uncertainty and simplifying policy learning.

2. Multi-Foundation-Model Diffusion Policy (MFMDP)

This module addresses the "how to manipulate" question by generating complex interaction trajectories using category-specialized models.

Feature Extraction: Instead of a single model, MFMDP routes objects to specialized foundation models:
- Deformable Objects: Uses the Garment Affordance Model (GAM) to extract features prioritizing point-correspondence and topological relations.
- Rigid/Articulated Objects: Uses Uni3D (modified to output point-wise features) to capture fine-grained geometric-semantic details.
Unified Descriptor: To reconcile distinct feature representations, the framework compresses high-dimensional semantic vectors (via PCA for Uni3D) and computes similarity-based vectors (for GAM). These are concatenated with raw 3D coordinates to form unified point descriptors.
Dual-Stream Cross-Attention: To prevent semantic features from overshadowing spatial geometry, the policy employs a dual-stream encoding mechanism:
- A Spatial Coordinate Stream preserves raw 3D geometry.
- A Semantic Descriptor Stream encodes the unified features.
- Cross-Attention fuses these streams, using semantic descriptors as queries and spatial coordinates as keys/values. This ensures semantic attributes enhance rather than suppress spatial cues.
Holistic Integration: The framework integrates global dynamic point clouds, robot joint states, and relational features from the cross-attention module into a DDIM-based diffusion model for action prediction.

Key Contributions

HeteroGenManip Framework: A novel architecture that decomposes long-horizon manipulation into two stages and integrates multiple category-specific foundation models to overcome the generalization bottleneck in heterogeneous interactions.
Foundation-Correspondence-Guided Grasp: A module that adapts to heterogeneous objects by leveraging structural priors to explicitly answer "where to manipulate," reducing policy complexity and state-action space.
Multi-Foundation-Model Diffusion Policy (MFMDP): A policy that embeds category-specific feature extraction strategies, tailoring foundation models to specific object types (rigid vs. deformable) to resolve the limitations of single-model approaches.
Empirical Validation: Comprehensive evaluation on established benchmarks (RoboTwin, DexGarmentLab) and novel tasks, demonstrating superior generalization over state-of-the-art methods.

Experimental Results

The framework was evaluated on eight simulation tasks and four real-world tasks covering rigid-rigid, rigid-deformable, rigid-articulated, and deformable-deformable interactions.

Simulation Performance: HeteroGenManip achieved an average 31% performance improvement over the best baseline in simulation tasks with broad type settings. In the "Test" setting (novel environments with varying shapes and poses), the proposed method maintained a success rate of 0.65, significantly outperforming the best baseline (0.34).
Real-World Performance: In real-world experiments using an ARX-X7s humanoid robot, the method achieved an average success rate of 76.7% in test settings, compared to 33.3% for the best baseline (DP). Notably, the gap between training and testing performance was minimal for HeteroGenManip, indicating strong generalization and resistance to overfitting.
Ablation Studies: Removing the correspondence-guided grasp caused a 24% average performance drop. Removing the multi-foundation model (replacing with a single Uni3D) led to significant failures in rigid-deformable tasks (e.g., dropping from 0.62 to 0.30 in "Hang Tops"), confirming the necessity of specialized models for different physical properties.

Significance and Claims

The paper claims that HeteroGenManip provides a new paradigm for heterogeneous object interaction manipulation. By explicitly decoupling contact localization from trajectory planning and utilizing a multi-foundation model routing strategy, the framework effectively reconciles the conflicting representational needs of objects with different physical properties. The authors assert that this approach eliminates redundant modeling and enables robust generalization to unseen objects and complex cross-type interactions in real-world scenarios, overcoming the limitations of single-model approaches and underutilized structural priors.

Note: The authors acknowledge limitations, specifically the reliance on single-view point clouds which depends heavily on foundation model capabilities and camera perspectives, and the dependency on 2D segmentation for point cloud generation, which can struggle with thin-edged objects like hangers.