🤖 AI

CoToGrasp: Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning

CoToGrasp is a novel, object-agnostic generative framework that synthesizes diverse, stable dexterous grasps conditioned on specific contact topologies by learning in a feature-based canonical workspace, thereby achieving zero-shot generalization to unseen objects without requiring expensive annotated datasets.

Original authors: Julien Merand, Boris Meden, Liming Chen, Mathieu Grossard

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Julien Merand, Boris Meden, Liming Chen, Mathieu Grossard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of moving heavy objects, but they struggle with the delicate art of holding things the way a human does. For decades, engineers treated grasping as a simple geometry problem: find a spot where a robot hand can hold an object without dropping it. This approach works well for picking up a box to move it from one place to another, but it fails when the robot needs to perform a specific task, like turning a screwdriver or holding a cup by its handle. In these situations, how the object is held matters just as much as that it is held. A robot needs to know not just where to grab, but which fingers to use and how to arrange them to support a future action. This shift from simple stability to functional intent is the next great challenge for robots that are meant to live and work alongside people.

A team of researchers has developed a new system called CoToGrasp to solve this problem. Instead of teaching a robot to memorize how to hold thousands of different objects, they taught it to understand the "shape" of a grip itself. The researchers realized that while every object in the world is different, the way a hand touches an object follows a limited set of patterns. They borrowed a classification system originally designed for human hands, which sorts grasps into categories like "precision" (using just the fingertips for fine control), "power" (wrapping the whole hand around something for strength), and "object-specific" (using a unique grip for a specific tool). By focusing on these contact patterns rather than the objects themselves, the system can learn to generate the correct grip for a tool it has never seen before.

The core of this breakthrough is a method that separates the idea of a grip from the object being held. Imagine a robot hand that learns to recognize the feeling of a pinch or a squeeze in its own frame of reference, completely independent of what it is holding. The researchers created a virtual workspace anchored to the robot's hand, where they mapped out exactly which parts of the fingers should touch an object for a specific task. When the system needs to grasp a new item, it projects the object's surface features into this hand-centered space. It then asks, "If I want a precision pinch, where on my fingers should the contact happen?" The system generates a map of contact points based on the desired task, and then the robot figures out how to move its joints to match that map. This allows the robot to ignore the confusing details of the object's shape and focus entirely on the functional goal.

To test this idea, the researchers trained their system using only data about the robot hand itself, never showing it a single picture of an object during the learning phase. This "object-agnostic" approach meant the system did not need to memorize millions of object-hand pairs, which is usually required for such tasks. Instead, it learned the intrinsic capabilities of the hand. When they tested the system on a large dataset of unseen objects, it successfully synthesized grasps that matched specific functional categories. The results showed that the system could produce a wide variety of grips, from delicate precision holds to strong power grips, without defaulting to the same few easy solutions that other robots often use. In fact, the system generated grasps with a much higher variety of functional types than previous methods, which tended to get stuck in a loop of only using simple, enveloping grips.

The researchers also compared their system to other advanced planners that try to follow similar rules. While those other systems often failed when the object's shape didn't perfectly match a pre-programmed template, CoToGrasp adapted smoothly. It successfully generated valid grasps for difficult, highly constrained tasks, such as holding a tool in a very specific way, with a success rate that outperformed the competition. The system was able to filter out impossible grasps before the robot even tried to move, ensuring that only physically viable options were selected. This filtering process is crucial because it prevents the robot from wasting time trying to grab something in a way that is physically impossible.

Finally, the team took the system out of the computer simulation and onto a real robot arm equipped with a multi-fingered hand. They placed various everyday objects on a table and asked the robot to grasp them using specific contact patterns. The robot successfully executed these complex grips in the real world, holding objects steady and demonstrating the physical viability of the generated plans. The experiments confirmed that the system could translate a high-level instruction like "use a precision grip" into a concrete, stable physical action on an object it had never encountered before. This work suggests that by teaching robots to understand the functional logic of a grip rather than just the geometry of an object, we can build machines that are far more capable of the nuanced, task-oriented manipulation required in our daily lives.

Technical Summary: CoToGrasp

Problem Statement

Current dexterous grasp planners primarily optimize for physical stability (force closure), focusing on whether an object can be grasped rather than how it should be grasped to support specific downstream functional tasks. While high Degrees of Freedom (DoF) robotic hands offer rich contact capabilities, existing data-driven planners exhibit a severe generative bias toward energetically stable but functionally uniform "enveloping power grasps." This mode collapse renders them inefficient for tasks requiring specific contact topologies, such as precision tool insertion or in-hand reorientation.

Furthermore, conditioning grasp synthesis on specific human grasp taxonomies (e.g., precision vs. power) typically requires prohibitively expensive, object-annotated datasets. Existing taxonomy-conditioned methods often couple contact topologies rigidly to specific object geometries during training. Consequently, when presented with unseen objects whose local geometry does not perfectly match pre-computed joint templates, these methods fail or collapse into incorrect grasp types, hindering zero-shot generalization.

Methodology: CoToGrasp

The authors propose CoToGrasp (Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning), a generative framework designed to synthesize diverse, stable grasps strictly conditioned on specific contact topologies while remaining object-agnostic.

1. Object-Agnostic Training Paradigm

Unlike standard methods that learn contact maps directly on object point clouds, CoToGrasp decouples functional intent from object identity by training exclusively on gripper point clouds.

Canonical Workspace: The framework introduces a feature-based canonical workspace anchored to the gripper frame. Local geometric features are extracted from the gripper (during training) or the object (during inference) and projected onto a fixed set of spatial basis points using a weighted k-Nearest Neighbors (kNN) aggregation.
Symmetric Contact Duality: The method leverages the geometric duality of contact. Instead of searching for valid locations on an unbounded object surface, the model learns the intrinsic contact manifold of the gripper within a bounded, fixed domain. The object is transformed into the gripper's canonical frame (inverted as a "negative mold"), allowing the network to learn grasp templates as fundamental capabilities of the hand, independent of the target object's geometry.

2. Semantic Taxonomy Conditioning

The system adapts the Gonzalez taxonomy [12], which categorizes 21 distinct contact topologies based strictly on the hand's active contact surfaces rather than object shape.

Input: The network receives the projected local features of the workspace and a learnable embedding of the requested contact topology (e.g., Precision, Power, or Object-Specific).
Architecture: A Transformer encoder processes the workspace embeddings, utilizing self-attention to model the spatial distribution and co-occurrence patterns of contact points inherent to each topology.
Generative Model: Features are distilled into a compact latent descriptor via a Set Transformer and fed into a Conditional Variational Auto-Encoder (CVAE). The decoder reconstructs probabilistic contact masks (zone probabilities) over the canonical workspace.

3. Grasp Synthesis and Validation Pipeline

During inference, the system synthesizes a physical grasp through a cascading validation pipeline to ensure kinematic feasibility and physical stability:

Prediction: The network predicts a contact template over the canonical workspace for a given object pose and requested topology.
Label-Consistency Check: A strict binary check verifies that the predicted contact zones are geometrically feasible for the specific object (preventing "hallucinated" contacts on non-existent surfaces).
Force-Closure Validation: A rapid, approximated force-closure estimator validates the physical stability of the proposed contact distribution. If unstable, the latent variable is resampled (up to 20 iterations) to generate a new template.
Energy-Based Optimization: Finally, an energy-based optimization aligns the gripper's active surfaces with the predicted 3D workspace points. This step minimizes contact alignment error, object penetration, self-collision, and joint limits, while introducing a repulsive term to enforce the specific topology by keeping unused fingers away from the object.

Key Contributions

CoToGrasp Framework: A novel generative planner that synthesizes grasps conditioned on structured contact topologies derived from human taxonomies, operating in an object-agnostic manner.
Canonical Feature-Based Workspace: A method to project local object features into a unified, gripper-centric domain, effectively decoupling semantic functional intent from arbitrary object geometry. This enables zero-shot generalization to unseen objects without requiring object-annotated training data.
Rigorous Evaluation Methodology: The authors establish a new evaluation framework to quantify the functional diversity and semantic bias of grasp planners. They introduce metrics such as Topology Compliance (TC) and Semantic Entropy ( $H_{TC}$ ) to measure how well planners adhere to requested functional intents versus defaulting to power grasps.
Real-World Validation: Demonstration of the physical viability and kinematic feasibility of synthesized contact topologies on a physical Allegro Hand robot.

Experimental Results

The authors evaluated CoToGrasp on the large-scale DexGraspNet dataset and the MultiDex subset, comparing it against both taxonomy-unaware baselines (e.g., DFC, GenDexGrasp) and taxonomy-aware baselines (Dexonomy).

vs. Unconditioned Planners: While unconditioned baselines achieve higher raw Success Rates (SR) by defaulting to stable power grasps, they exhibit severe mode collapse with low semantic entropy. Specifically, the DFC baseline reports an $H_{TC}$ of 0.7389, while other baselines show even lower values. In contrast, CoToGrasp achieves the highest semantic entropy ( $H_{TC} = 0.83$ ) and generation speed (0.11s/grasp), successfully synthesizing diverse topologies including precision grasps, albeit with a lower overall SR (36.94%) due to the difficulty of the constrained task.
vs. Taxonomy-Aware Baselines: Compared to Dexonomy, CoToGrasp demonstrates superior performance in both physical stability (SR: 29.75% vs. 27.16%) and semantic accuracy (TC: 17.18% vs. 14.28%). Crucially, CoToGrasp outperforms Dexonomy in highly constrained precision grasps (e.g., Topology M2: 30.3% vs. 10.5%). The authors note that Dexonomy's higher scores on certain topologies are artifacts of mode collapse, where it defaults to unverified enveloping grasps when object geometry conflicts with rigid joint templates.
Real-World Performance: The system successfully planned and executed diverse grasps on YCB objects using a UR10 manipulator with an Allegro Hand, confirming that the generated topologies are physically executable.

Significance and Claims

The paper claims that CoToGrasp addresses the critical limitation of existing planners: the inability to generate task-aligned contact topologies without expensive, object-specific training data. By shifting the learning domain from the unbounded object space to the fixed, intrinsic gripper manifold, the method achieves zero-shot generalization to unseen geometries.

The authors emphasize that their approach allows robotic systems to move beyond simple geometric stability optimization toward functional, task-level intent. The rigorous validation pipeline ensures that the synthesized grasps are not only semantically correct but also kinematically feasible and physically stable. The successful real-world deployment demonstrates that the structural properties of the generated contact topologies can be executed on physical hardware, bridging the gap between high-level semantic reasoning and low-level dexterous manipulation.