Technical Summary: DexVerse

Problem Statement

Building general-purpose robots requires dexterous manipulation capabilities that extend beyond simple gripper-based reaching and grasping. True dexterity involves coordinated control of high-degree-of-freedom (DoF) hands and arms under intermittent, contact-rich interactions, grounded in object geometry, visual affordances, and long-horizon task structures.

A critical bottleneck in advancing this field is the lack of benchmarks that jointly evaluate the major axes of dexterous generalization. Existing benchmarks are often limited in one or more of the following areas:

• Task Diversity: Many focus on gripper-based manipulation (e.g., CALVIN, LIBERO) or narrow dexterous settings without expert demonstrations.
• Embodiment Coverage: Few support multiple arm-hand combinations, hindering the study of embodiment transfer.
• Visual and Data Variability: Limited support for controllable visual variations (textures, lighting, viewpoints) and synchronized multi-modal demonstration data.
• Evaluation Scope: Difficulty in comparing policies across different contact regimes, visual conditions, and embodiments simultaneously.

Consequently, it remains unclear how well current generalist robot policies scale from isolated skills to functionally diverse, long-horizon, contact-rich dexterous manipulation across different embodiments and environments.

Methodology: The DexVerse Benchmark

The authors present DexVerse, a large-scale, modular benchmark suite designed to unify broad dexterous task coverage, multi-embodiment control, and visuomotor generalization evaluation.

1. Task Suite and Taxonomy

DexVerse comprises 100 distinct dexterous manipulation tasks organized into eight categories based on interaction patterns and manipulation challenges:

• Primitive (9 tasks): Direct interaction with simple goals (e.g., PickCube).
• Functional (11 tasks): Affordance-aware interactions (e.g., HammerStrike, PourCan).
• Articulation (18 tasks): Controlling object parts and joints (e.g., OpenLaptop, SqueezeScissors).
• Non-prehensile (5 tasks): Pushing, sliding, or pivoting without grasping.
• Contact-rich (8 tasks): Precise alignment under sustained contact (e.g., InsertPeg, NutThread).
• Bimanual Coordination (5 tasks): Cooperative manipulation across two hands/arms.
• Multi-goal (39 tasks): Satisfying multiple or compositional objectives.
• Long-horizon (5 tasks): Temporally extended multi-stage procedures (e.g., MakeCoffee, OvenBake).

2. Modular Environment Design

Built on Isaac Lab, the environment utilizes a configuration-driven architecture:

• Modularity: Tasks are defined by structured components (scene layout, assets, robot embodiment, observation/action interfaces, initialization, success conditions).
• Extensibility: A hierarchy of base environments (Main Base, Task Family Base, Task Environment) allows for reusable templates and easy instantiation of new tasks, assets, or robot configurations without rewriting core logic.
• Embodiment Support: The benchmark supports 3 robot arms (Franka Research 3, UR10e, xArm 7) and 6 dexterous hands (Sharpa Wave, WUJI, Shadow, Inspire, Allegro, LEAP). It supports both fixed and floating wrist variants.
• Visual Variation: Configurable visual randomization is integrated, allowing for changes in object/scene textures, lighting, skyboxes, exposure, and camera viewpoints to test visuomotor robustness.

3. Dataset and Teleoperation

To address the scarcity of high-quality dexterous demonstrations, the authors developed a VR-based teleoperation pipeline:

• Interface: Uses Apple Vision Pro with Isaac Lab's CloudXR interface for immersive control.
• Retargeting: Human wrist poses drive robot end-effectors via inverse kinematics. Human hand motions are retargeted to various dexterous hands using optimization-based dex-retargeting.
• Data Collection: The dataset contains 3,180 demonstration trajectories.
  • 56 single-goal tasks: 55 demos each (50 with Shadow Hand, 1 each with other 5 hands).
  • 5 long-horizon tasks: 20 demos each.
• Multi-modal Observations: Each demonstration includes synchronized proprioceptive, RGB, depth, point-cloud, and simulator state observations.
• Replay Mechanism: A utility allows state replay to regenerate observations locally, ensuring portability and avoiding physics simulation drift across different hardware.

4. Policy Evaluation

The authors benchmarked four representative policy families on a baseline split of 19 tasks (950 episodes total):

1. π0.5: Vision-language-action (VLA) transformer.
2. OpenVLA: VLA transformer.
3. Diffusion Policy (DP): From-scratch diffusion policy.
4. 3D Diffusion Policy (DP3): From-scratch diffusion policy utilizing point clouds.

Key Results

Evaluation across the 19 tasks revealed significant challenges for current methods:

• Overall Performance: The mean online success rate for the best-performing baselines was only 34%. No single method consistently dominated across all task categories.
• Method-Specific Strengths:
  • Diffusion Policy (DP): Performed best on simple "Pick-and-Lift" tasks (0.51 mean), where 2D images and low-dimensional states suffice.
  • 3D Diffusion Policy (DP3): Excelled in "Functional Tool Use" (0.35 mean), where point-cloud geometry aids in localizing tool tips.
  • π0.5: Led in "Articulated-object Manipulation" (0.35) and "Precision Contact" (0.29), benefiting from language conditioning and flow-matching experts for multi-stage subgoals.
  • OpenVLA: Lagged behind diffusion baselines (0.19 mean), suggesting that internet-scale pretraining priors do not yet translate effectively to high-DoF dexterous control manifolds.
• Failure Modes:
  • Contact-Rich Tasks: Tasks requiring sub-centimeter alignment and sustained force regulation (e.g., PushT, InsertPen, SlideUtilityKnife) resulted in 0% success for all baselines.
  • Generalization: Policies struggled to generalize across different visual conditions and embodiments without explicit retargeting or embodiment-aware representations.

Significance and Contributions

The paper claims three primary contributions:

1. Unified Benchmark: DexVerse provides the first modular benchmark unifying 100 diverse dexterous tasks, multiple arm-hand embodiments, controllable visual variation, and a unified evaluation framework. It fills the gap between gripper-based benchmarks and narrow dexterous studies.
2. Multi-modal Dataset: The release of 3,180 expert demonstrations collected via VR teleoperation, synchronized across proprioceptive, RGB, depth, point-cloud, and state modalities, enables the evaluation of state-based, image-based, and 3D policy learning methods under consistent conditions.
3. Empirical Baselines: By benchmarking state-of-the-art policies, the authors demonstrate that current approaches are far from solving general dexterous manipulation. The results highlight specific open challenges, particularly in fine-grained visuomotor control, contact-rich reasoning, and embodiment transfer.

The authors position DexVerse not as a solved problem, but as a challenging and extensible testbed necessary for developing robust, general-purpose dexterous manipulation policies. The benchmark exposes the limitations of current imitation learning and VLA approaches, specifically their inability to handle tight-tolerance contact tasks and complex multi-stage coordination without explicit force feedback or closed-loop correction.