💻 computer science

Human Universal Grasping

This paper introduces HUG, a flow-matching model trained on a massive 1-million-frame egocentric human grasp dataset that generates diverse, retargetable grasps for any object from a single RGB-D image, achieving state-of-the-art performance in zero-shot robotic grasping across various embodiments and environments.

Original authors: Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to pick up a coffee mug. Usually, engineers have to spend months programming the robot, showing it thousands of examples of how that specific robot should move its fingers. It's like teaching a child to tie their shoes by only showing them how their hands do it, then expecting them to figure out how to tie shoes with a different pair of hands.

This paper, titled "Human Universal Grasping" (HUG), proposes a much simpler, more natural solution: Just watch humans do it.

Here is the story of how they did it, explained in everyday terms.

1. The Problem: Robots are Clumsy Students

Robots are great at repetitive tasks in factories, but they struggle with the messy, unpredictable world of a home. If you give a robot a weirdly shaped object (like a pineapple or a hairbrush), it often doesn't know how to grab it without knocking it over. Most robots are trained on "simulated" data (computer games) or by humans manually controlling them, which is slow and tedious.

2. The Solution: The "Smart Glasses" Field Trip

The researchers realized that humans are already experts at grabbing things. We pick up thousands of objects every day without thinking about it. Instead of teaching robots from scratch, they decided to copy the human playbook.

The Data Collection: They gave people a pair of smart glasses (called Aria Gen 2) that look like regular eyewear but have cameras and sensors.
The Mission: People wore these glasses and went about their day in 41 different buildings (homes, offices, etc.). They picked up 6,707 different objects.
The Result: They created a massive library called 1M-HUGS. It contains 1 million images of human hands grabbing things. It's like a "YouTube" of human grasping, but with 3D depth information so the computer knows exactly how far away the hand is.

3. The Brain: A "Flow" Model

Once they had the data, they needed a way to teach a robot to use it. They built a model called HUG.

Think of HUG as a chameleon or a universal translator:

Input: You show it a single photo (with depth) of an object, like a toaster on a counter. You click on the toaster with your mouse.
Processing: The model looks at its library of 1 million human grabs. It asks, "If a human saw this toaster, how would they grab it?"
Output: It doesn't just say "grab it." It calculates the exact position, rotation, and finger shape needed to hold that specific object.

The magic trick is that this model learns how humans move, not how a specific robot moves. It learns the concept of a grasp.

4. The Translation: From Human Hands to Robot Hands

Here is the clever part. The model predicts a grasp using a standard human hand shape (called MANO). But robots have different hands—some have three fingers, some have four, some are big, some are small.

The researchers built a retargeting system. Imagine you are a dancer. You learn a routine (the human grasp). Now, you have to perform that same routine on a stage with a different floor layout, or with a partner who is taller than you. You don't learn a new dance; you just adjust your steps to fit your new partner.

HUG does this instantly. It takes the human hand pose it predicted and mathematically "retargets" it to fit the robot's specific fingers.

No retraining needed: You don't have to teach the robot again. You just plug in the new robot, and it works.
Zero-shot: This means it works on objects the robot has never seen before, in rooms it has never visited.

5. The Test: The "HUG-BENCH" Challenge

To prove this works, they didn't just test it on easy things like balls or boxes. They built a "final exam" called HUG-BENCH.

They gathered 90 tricky objects: things with handles, weird shapes, tiny items, and large, unwieldy items.
They tested the system in two ways:
1. In Simulation: A computer world where they could test thousands of times quickly.
2. In the Real World: They took a real robot into a real house and tried to pick up these 90 objects.

The Results:

HUG succeeded 66.7% of the time on the tabletop and 62% in the wild (messy house).
It beat the current best robot methods by a huge margin (23% to 34% better).
While other robots failed on tricky items like a picnic basket or a spray bottle, HUG figured out how to grab them because it had "seen" humans do similar things before.

Summary

The paper claims that by collecting 1 million real-world human grasps using smart glasses, they created a system that allows robots to learn to grab anything, anywhere, with any hand, simply by copying human intuition. It bridges the gap between human dexterity and robot clumsiness without needing to program the robot for every single new object.

What they didn't claim:

They did not claim this works for medical surgery or delicate clinical tasks.
They did not claim the robot can plan complex multi-step tasks (like making a sandwich); it only solves the specific problem of "how do I grab this one object right now?"
They noted limitations: The system currently only models right-handed grasps and struggles with very tiny objects or objects that are completely hidden from view.

Technical Summary: HUG (Human Universal Grasping)

Problem Statement

While humans grasp arbitrary objects effortlessly, multi-fingered robotic hands lack comparable generality. A primary bottleneck is the scarcity of diverse, real-world grasping data for robots. Existing approaches face significant limitations:

Synthetic Data: Methods relying on simulation (optimizing force-closure or using RL) suffer from the sim-to-real gap and often require retraining for each new robot embodiment.
Teleoperation: While providing real grasps, teleoperation is tedious, does not scale to the open world, and is difficult to collect for dexterous hands.
Data Scarcity: Robots lack the thousands of daily grasping experiences humans accumulate, hindering the learning of natural, executable grasp distributions.

The paper argues that the most natural source of robot grasping data is humans themselves. However, leveraging this requires overcoming challenges in data collection scale, precise 3D annotation, and the morphology gap between human and robot hands.

Methodology: HUG Framework

The authors propose HUG (Human Universal Grasping), a framework that learns dexterous grasping purely from in-the-wild human data, enabling zero-shot transfer to various robot hands. The pipeline consists of three main components:

1. Dataset: 1M-HUGS

The authors collected 1M-HUGS, an egocentric dataset comprising 1 million image-grasp pairs.

Collection: Data was recorded using Aria Gen 2 smart glasses across 41 buildings, capturing 6,707 object instances in 27.8 hours of video.
Protocol: Wearers recorded static scenes from diverse viewpoints before reaching in with their right hand to grasp an object. The grasp pose is propagated back to preceding frames (where hands are invisible) using camera poses, generating hundreds of (object-only image, grasp) pairs per physical grasp.
Processing: A curation pipeline uses vision-language models and SAM3 for object masking. Frames are filtered based on depth confidence and hand-object intersection.
Labels: Since Aria provides only 21 hand landmarks, the authors fit a full articulated MANO hand model to these landmarks. The MANO shape parameter ( $\beta$ ) is fixed to a canonical value to remove inter-person size variations, while the pose parameter ( $\theta$ ) captures the grasp. The dataset includes synchronized RGB, grayscale, metric depth, object masks, and MANO poses.

2. Model Architecture: Flow-Matching Transformer

HUG is a flow-matching model that predicts a MANO grasp state from a single RGB-D image and a user-specified 2D query point on the object.

Input: An RGB-D image and a 2D pixel click $(u, v)$ , which is lifted to a 3D query point $p_q$ using depth and intrinsics.
Encoders:
- RGB: Encoded using a frozen DINOv2-Base ViT with register tokens.
- Point Cloud (PC): The depth image is back-projected to a metric point cloud, cropped to a 0.3m radius around $p_q$ , and processed by a trainable PointNeXt U-Net.
Fusion: The model employs point painting, projecting point cloud centroids to the RGB image to sample DINO features, which are concatenated with PC tokens and fused via an MLP.
Flow Transformer: The grasp state is decomposed into wrist translation ( $t$ ), wrist rotation ( $R_{6d}$ ), and finger pose ( $\theta_{6d}$ ). These are processed by a DiT (Diffusion Transformer) block that cross-attends to the fused scene tokens. The model is conditioned on timesteps via AdaLN-Zero modulation.
Output: The model predicts a 99-dimensional grasp state, which is integrated via a flow-matching ODE to produce the final MANO pose.

3. Retargeting

Predicted human grasps (MANO poses) are retargeted to various robot hands (e.g., Ability, WUJI) using learned retargeting methods. Crucially, no per-embodiment training is required; the model generalizes zero-shot across different robot morphologies.

Key Contributions

First Human-Only Framework: HUG is the first grasping framework trained purely on human data (no robot data) that is deployable across multiple robot embodiments.
1M-HUGS Dataset: A large-scale, egocentric dataset of 1M natural human grasps with metric depth and MANO-fit hand poses, covering diverse environments and object instances.
HUG-BENCH: A new evaluation benchmark consisting of 90 unseen objects (30 for real-world testing) across five geometric categories and three size bins. It includes metric-scale 3D meshes for paired simulation and real-world evaluation.
Methodology: A point-conditioned flow-matching model that fuses RGB and depth to predict diverse, natural grasps without requiring per-hand training.

Experimental Results

Simulation Evaluation (HUG-BENCH)

Evaluated in MuJoCo on 30 unseen test objects:

Performance: HUG achieved a 73.0% success rate (SR) and 14.6 mm fingertip contact (FC) error.
Baselines: Outperformed the "Human Grasp Oracle" (94.0% SR) by a margin consistent with simulator limitations (tracking errors, open-loop execution).
Ablations:
- Removing the 3D geometric loss dropped SR to 32.7%, highlighting its necessity for accurate fingertip placement.
- Removing the 0.3m crop or point painting reduced performance by ~10-15 SR points.
- Modality: RGB-only failed significantly (29.7% SR), while PC-only was strong (70.7% SR) but lacked semantic grounding. The dual-modal RGB+PC approach was essential for resolving ambiguous cases (e.g., grasping the handle of a brush vs. bristles).
Scaling: Performance improved monotonically from 25K to 1M frames, suggesting the model is still data-bound rather than capacity-bound.

Real-World Evaluation

Evaluated on 30 objects across two settings:

Tabletop (xArm + Ability Hand): HUG achieved 66.7% SR, outperforming Dex1B (43.7%) and CAP (32.7%) by +23% and +34%, respectively. HUG successfully grasped 28/30 objects at least once.
In-the-Wild (YOR Mobile Manipulator + WUJI Hand + Aria Glasses): HUG achieved 62.0% SR in uncontrolled household environments without any onsite tuning or per-embodiment retraining.

Failure Modes: Most failures occurred during the pre-grasp to grasp transition (hitting the object or table) or during the lift (slipping). These are attributed to open-loop execution and lack of force feedback, rather than incorrect grasp prediction.
Generalization: HUG demonstrated robustness across varying object sizes (small erasers to large bins), geometries (amorphous, appendaged), and environments.

Significance and Claims

The paper claims that HUG demonstrates that dexterous robot grasping can be learned directly from the natural behavior of humans, bypassing the need for expensive robot-specific data collection or simulation-heavy training.

Generality: By modeling the distribution of natural human grasps rather than all physically valid grasps, the system produces reliably executable grasps that transfer zero-shot to new cameras, robot hands, and unseen homes.
Scalability: The use of lightweight smart glasses allows for the collection of massive datasets (1M frames) that were previously infeasible.
Benchmarking: The introduction of HUG-BENCH provides a standardized, challenging, and metric-scale benchmark for evaluating dexterous grasping in both simulation and the real world, addressing the lack of rigorous evaluation standards in the field.

The authors conclude that while limitations exist (e.g., right-hand only, open-loop execution, occlusion handling), the framework represents a significant step toward achieving the generality of human grasping in robotics.