💻 computer science

KPGrasp: Scalable Keypoint Flow Matching for Dexterous Grasp Generation

KPGrasp introduces a scalable flow-matching framework that generates high-quality dexterous grasps by coupling an all-Euclidean 3D hand-keypoint parameterization with a Transformer model, achieving state-of-the-art performance on simulation benchmarks and successful real-world deployment without relying on contact losses or test-time refinement.

Original authors: Yuansen Huang, Jiayi Chen, Haoran Liu, Yubin Ke, Bing Han, Jiangran Lyu, Mi Yan, Li Yi, He Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuansen Huang, Jiayi Chen, Haoran Liu, Yubin Ke, Bing Han, Jiangran Lyu, Mi Yan, Li Yi, He Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot hand to pick up a coffee mug, a teddy bear, or a weirdly shaped vase. For a long time, robots have struggled with this because their "brains" were trying to solve a math problem that was way too complicated. They were trying to calculate the exact angle of every single finger joint and the exact 3D position of the wrist all at once, while also worrying about not crushing the object. It was like trying to drive a car while simultaneously solving a complex calculus equation and balancing a stack of plates.

The paper introduces KPGrasp, a new way to teach robots how to grab things. Instead of making the robot do heavy math, KPGrasp teaches it to "see" the shape of a hand in a much simpler, more intuitive way.

Here is how it works, broken down into simple concepts:

1. The Old Way: The "Mixed-Up" Instruction Manual

Previously, when a robot needed to grab something, it was given a confusing instruction manual. It had to figure out two different things at once:

The Wrist: Where is the hand in space? (This is tricky because rotation is mathematically weird and "bumpy").
The Fingers: How bent should each finger be?

This is like trying to bake a cake by being told, "Rotate the oven 45 degrees clockwise, then add 2.5 cups of flour, then rotate the oven 10 degrees counter-clockwise." The instructions are in different "languages" (rotation vs. straight lines), which makes it hard for the robot to learn the pattern. If the robot makes a tiny mistake with the wrist rotation, the whole hand ends up in the wrong place, and the fingers might crash into the object.

2. The New Way: The "Dot-to-Dot" Drawing

KPGrasp changes the game. Instead of giving the robot complex rotation angles, it tells the robot to simply place dots in 3D space.

Imagine you have a picture of a hand. Instead of describing the angles of the joints, you just put a dot on the tip of the thumb, a dot on the tip of the pinky, and a few dots on the palm.

The Magic: These dots exist in normal, straight-line space (Euclidean space). It's much easier for a computer to learn how to move dots around than to learn how to rotate a wrist.
The Result: The robot learns to place these dots exactly where they need to be to hug the object perfectly. Once the dots are placed, a simple "translator" (called Inverse Kinematics) instantly figures out what the finger angles need to be to match those dots.

3. The "Flow" Model: Learning from a Massive Library

How does the robot learn where to put these dots?

The Old Way: Researchers had to write strict rules (like "don't touch the object too hard" or "don't let the fingers go through the object"). If the rules were too strict, the robot froze. If they were too loose, the robot crushed the object. It was a constant game of "tuning the knobs."
The KPGrasp Way: The researchers fed the robot a massive library of 9.5 million successful grabs. They used a technique called Flow Matching.
- Analogy: Imagine a river flowing from a chaotic ocean (random noise) into a calm, organized lake (perfect grasps). The robot learns the "current" of this river. It starts with a random guess and follows the flow until it lands on a perfect grasp.
- Because it learned from so many examples, it doesn't need strict rules or "tuning knobs." It just knows what a good grab looks like because it has seen millions of them.

4. The Results: Fast, Accurate, and Real

The paper tested this new method against the best existing robots:

Success Rate: On a tough test called "Dexonomy," KPGrasp succeeded 76.3% of the time. The next best robot only succeeded about 29% of the time. That's a massive jump.
No Crushing: The robot barely touched the objects (penetration depth was only 2.4 mm), meaning it didn't crush or squeeze things.
Speed: It is incredibly fast. It can generate a grasp in 0.032 seconds. The old methods that tried to "fix" their mistakes after generating them took about 2.8 seconds. KPGrasp is roughly 87 times faster.
Real World: They tested it on a real robot arm with a real camera. Even though the camera only saw one side of the object (not a perfect 3D scan), the robot still succeeded 83% of the time.

Summary

KPGrasp is like teaching a robot to grab things by showing it a million pictures of successful grabs and asking it to "connect the dots" on a hand, rather than forcing it to solve complex geometry equations. By simplifying the language the robot speaks (using dots instead of angles) and feeding it a massive diet of good examples, the robot learns to grab almost anything quickly and gently, without needing a human to constantly tweak its settings.

Technical Summary: KPGrasp: Scalable Keypoint Flow Matching for Dexterous Grasp Generation

Problem Statement

Generating high-quality dexterous grasps remains a significant challenge for learning-based methods. Traditional approaches often rely on carefully tuned contact losses, physics-based objectives, or costly contact-based test-time refinement to ensure grasp stability. These components inherit drawbacks from analytic methods, including brittle parameter tuning, slow inference, and the potential for generating grasps with large penetrations or poor stability. Furthermore, existing learning-based methods frequently reintroduce explicit physics priors (e.g., contact penalties, discriminators) into the training or inference pipeline, raising the question of whether large-scale, high-quality data can implicitly provide sufficient grasp priors without such hand-crafted designs.

Additionally, the standard output parameterization for dexterous grasps—a hybrid tuple of non-Euclidean SE(3) wrist pose and Euclidean joint angles—presents challenges for scalable generative modeling. This "pose-joint" representation introduces manifold discontinuities (specifically in SO(3)), leads to kinematic error accumulation along the kinematic chain, and requires complex loss balancing between mixed output spaces.

Methodology

The authors propose KPGrasp, a flow-matching framework that learns dexterous grasp priors directly from large-scale offline synthesized data, eliminating the need for explicit contact losses or test-time refinement. The method is built on two core innovations:

1. All-Euclidean 3D Hand-Keypoint Parameterization

Instead of predicting a wrist pose and joint angles, KPGrasp predicts a set of 3D hand keypoints ( $K \in \mathbb{R}^{J \times 3}$ ) directly in the same coordinate frame as the input object point cloud.

Definition: For the Shadow Hand, $J=21$ keypoints are defined, located at mechanical joints, fingertips, and five points on the rigid palm (center and four finger roots).
Advantages: This representation places the hand output entirely in Euclidean space, aligning naturally with object point clouds and enabling native spatial reasoning. It avoids SO(3) discontinuities and allows the network to reason about spatial proximity during hand-object interaction.
Conversion: During training, labels are derived by applying forward kinematics to standard pose-joint grasps. During inference, predicted keypoints are converted back to executable robot commands: palm keypoints determine the wrist pose ( $R, t$ ) via SVD-based rigid registration, and joint angles ( $\theta$ ) are recovered using a standard inverse kinematics (IK) solver. This IK step projects keypoints onto the kinematic manifold without optimizing for contact or penetration.

2. Scalable Transformer Flow Model

KPGrasp employs a conditional Transformer-based flow model to learn the grasp distribution.

Architecture: The model uses a Diffusion Transformer (DiT) backbone. The input object point cloud is processed by a MinkUNet to generate geometry tokens. Noisy hand keypoints are embedded as tokens. The network alternates between keypoint self-attention (modeling hand structure) and object-conditioned cross-attention (aligning hand geometry with the object surface).
Training Objective: The model is trained exclusively using the standard Flow Matching (FM) loss, which minimizes the distance between the predicted velocity field and the conditional velocity of the data path. No auxiliary contact losses, discriminators, or physics-based penalties are used.
Inference: Samples are generated by solving the learned Ordinary Differential Equation (ODE) via Euler integration. To select high-quality grasps, the model utilizes the tractable likelihood of the Continuous Normalizing Flow (CNF), estimated via the Hutchinson trace estimator, to rank candidates without external evaluators.

Key Contributions

KPGrasp Framework: A dexterous grasp generator that learns priors directly from large-scale data without contact losses or contact-based test-time refinement.
Keypoint Parameterization: The introduction of an all-Euclidean 3D hand-keypoint output space, replacing the conventional hybrid SE(3) pose and joint-angle representation to facilitate scalable generative learning.
Scalable Flow Model: A Transformer-based flow model that demonstrates effective scaling with dataset size, model capacity, and batch size under a standard flow-matching objective.
Empirical Validation: Demonstration of state-of-the-art (SOTA) performance on simulation benchmarks and successful deployment in real-world experiments.

Experimental Results

Simulation Benchmarks

Dexonomy Benchmark: KPGrasp achieved a 76.3% Grasp Success Rate (GSR), outperforming the strongest directly comparable baseline (Dexonomy without classifier) by 47.4 percentage points. It also surpassed the classifier-based Dexonomy variant (63.9% GSR) without requiring additional generated training data. Crucially, it reduced the average penetration depth to 2.4 mm and achieved a 97.3% Object Success Rate (OSR).
DexGrasp Anything Benchmark: Without fine-tuning, the model trained on Dexonomy achieved the best average success rate (63.8%) and lowest average penetration depth (12.9 mm) across five diverse object sets, demonstrating strong generalization.

Ablation Studies

Parameterization: The keypoint parameterization significantly outperformed pose-joint variants (both Euclidean and SE(3) flow), confirming that the Euclidean representation is superior for this task.
Tokenization: The performance gain was attributed to the multi-token reasoning capability of the keypoint representation; pose-joint outputs did not recover this benefit even with increased token counts.
Likelihood Ranking: Using the flow likelihood to rank candidates improved GSR from 67.0% to 76.3%, validating the likelihood score as an effective selection metric.
Scaling: Performance improved consistently with increases in dataset size, model capacity (number of DiT blocks), and batch size.

Efficiency and Real-World Deployment

Inference Speed: KPGrasp requires only 0.032 seconds per grasp during batched inference (including sampling, ranking, and IK), which is approximately 87 times faster than DexGrasp Anything (2.8s) and offers a throughput of ~24 valid grasps per second per GPU.
Real-World Experiments: Tested on 20 diverse objects using a Galbot platform with a Shadow Hand and a single-view point cloud setup. The pipeline achieved an 83% success rate (83/100 trials) without fine-tuning the core architecture, though it noted sensitivity to small object-pose perturbations.

Significance and Claims

The paper claims that KPGrasp demonstrates that large-scale, high-quality data can implicitly provide sufficient grasp priors for learning, rendering explicit physics-based objectives and test-time refinement unnecessary. By shifting to an all-Euclidean keypoint parameterization, the authors show that dexterous grasp generation can be framed as a clean Euclidean learning problem compatible with scalable Transformer architectures. The results suggest that this approach not only achieves superior performance in terms of success rate and penetration depth but also offers significant advantages in inference speed and scalability compared to prior methods that rely on complex loss tuning or iterative refinement. The authors position KPGrasp as a step toward efficient, data-driven dexterous manipulation that avoids the brittleness of analytic methods and the computational overhead of physics-guided inference.