Technical Summary: GraspIT

Problem Statement

Robotic manipulation of everyday objects remains a significant challenge, particularly regarding generalization to novel objects, scenes, and lighting conditions. A primary bottleneck is the lack of training data that simultaneously provides:

1. Photorealistic RGB-D observations linked to real-world sensor data.
2. Physically validated grasp quality annotations that go beyond static force-closure metrics.
3. A principled bridge between simulation and reality (Real↔Sim) to ensure simulation labels are valid for real sensor data.

Existing datasets often lack one or more of these components. For instance, benchmarks like GraspNet-1Billion provide massive scale but rely on analytic force-closure metrics that correlate poorly with real-world success under dynamic perturbation and lack robot reachability filtering. Conversely, large-scale imitation learning datasets (e.g., Open X-Embodiment, DROID) provide demonstration sequences but lack explicit 3D geometry, CAD models, or physically grounded grasp quality scores. This gap leaves visual encoders in modern Vision-Language-Action (VLA) models without the metric geometry and physical grounding necessary to predict manipulation success.

Methodology

GraspIT addresses these gaps through a closed-loop generation system that integrates real-world capture with high-fidelity simulation in NVIDIA Isaac Sim. The methodology rests on four design pillars:

1. Quality over Quantity via Four-Stage Slip Testing

Instead of generating billions of analytically scored candidates, GraspIT validates every grasp candidate through a four-stage physical slip test executed by parallel Franka Panda instances in simulation. This produces a continuous quality score $s \in [0, 1]$:

• Stage 1 (Trajectory + Close): Validates collision-free trajectory planning to antipodal force-closure.
• Stage 2 (Lift): Tests vertical displacement under gravity to detect insufficient friction.
• Stage 3 (Horizontal Oscillation): Applies linear perturbation to expose rotational instability.
• Stage 4 (Pendulum Swing): Rotates the joint to test resistance to moment arms from off-center mass.

A base score $b$ is assigned based on the highest stage passed (0.00 to 1.00). This is further refined by a penalty function (Eq. 2) that subtracts points if positional slip exceeds 2 cm or orientational slip exceeds 10° in any executed stage. This design captures failure modes (e.g., an object that lifts but slips during oscillation) that static force-closure metrics miss.

2. Robot-in-the-Loop Reachability Filtering

Before physics validation, every candidate undergoes collision-free trajectory planning using cuRobo and Lula for the Franka Panda arm. Candidates requiring approaches through obstacles or exceeding workspace limits are discarded, ensuring all positive labels are actionable for a deployed system.

3. Real↔Sim Bidirectional Link

The system establishes a consistent annotation coordinate frame across domains:

• Real-to-Sim: Real-world objects are 3D scanned using TRELLIS (generative image-to-3D) and registered into Isaac Sim via multi-view ICP. Real scenes are captured by a robot-mounted camera and registered into the simulator.
• Sim-to-Real: Physics-validated labels generated in simulation are back-projected onto all real camera frames.
  This allows a single real-world capture session to yield both real RGB-D images and unlimited additional photorealistic synthetic views with full annotation.

4. Open and Extensible Infrastructure

The entire pipeline is containerized in Docker, enabling any lab to reproduce, extend, or contribute to the dataset. The system supports arbitrary scene generation from the ABC dataset (~10k CAD meshes) and can integrate existing datasets (e.g., BOP, GraspNet) by importing their meshes into the validation pipeline.

Key Contributions

The paper presents three primary contributions:

1. GraspIT Dataset: The first dataset providing photorealistic RGB-D observations, robot-reachability-filtered grasps, and staged slip-test quality scores simultaneously. It includes ~316k annotated RGBD frame sets across 1,035 simulated and 100 real scenes.
2. Real↔Sim Registration Pipeline: A method that validates simulation annotations for transfer to real sensor data through metric 3D registration, addressing the domain gap.
3. Scalable Generation System: An open, containerized system enabling unlimited new scene generation with minimal manual labeling, utilizing ~7.4k unique object identities.

Results and Statistics

The initial release of GraspIT contains:

• Scenes: 1,035 simulated scenes and 100 real-world scenes.
• Frames: ~316k annotated frame sets (256 views per scene).
• Objects: ~7,396 unique object identities from the ABC dataset plus 100 real scanned objects.
• Grasp Candidates: ~2.3M candidates evaluated by the Robot School.
  • 82.94% (1.88M) are annotated as "good" ($s \ge 0.50$).
  • 17.06% (388k) are annotated as "bad" ($s < 0.50$).
  • Mean Score: $\bar{s} = 0.72 \pm 0.30$ across all grasps; $\bar{s}_{good} = 0.85$.
• Filtering Impact:
  • 11.46% of candidates failed trajectory planning (reachability).
  • 17.06% of candidates passed force-closure but failed the slip-test (providing graded hard negatives).
  • A specific "penalized" bucket of 94,475 grasps passed multiple stages but were downgraded due to excessive slip, highlighting the value of the continuous scoring over binary pass/fail metrics.

Significance and Claims

The paper positions GraspIT as a foundational resource for next-generation object manipulation models, specifically addressing the "physical grounding" deficit in current Vision-Language-Action (VLA) and diffusion policies.

• Bridging the Gap: By providing a bidirectional Real↔Sim link, GraspIT allows simulation-trained policies to be validated against real-world physics, a capability absent in prior datasets like ACRONYM or GraspNet.
• Beyond Force-Closure: The staged slip-test demonstrates that antipodal force-closure is insufficient for high-quality grasping. The inclusion of "hard negatives" (candidates that look geometrically plausible but fail under perturbation) provides critical training data for grasp quality regressors.
• Pretraining Potential: The authors argue that the physically grounded labels (shape, centroid, grasp quality) can serve as a pretraining stage for VLAs, potentially reducing the depth-estimation and physical reasoning deficits observed in current encoders.
• Modesty on Limitations: The authors acknowledge current limitations, including the restriction to parallel-jaw grippers, the use of a single robot model (Franka Panda), and the reliance on CAD-derived physical properties (ignoring material-dependent friction). They note that downstream evaluation benchmarks are not yet established and that the dataset is intended as a starting point for community expansion.

The GraspIT tools and data are publicly available to encourage the community to contribute new scenes, objects, and task demonstrations, aiming to grow the dataset into a shared foundation for generalized robotic manipulation.