💻 computer science

UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data

This paper introduces UMI-Bench 1.0, the first open and reproducible real-world benchmark designed to standardize the evaluation of Universal Manipulation Interface (UMI)-style policies by unifying data collection, scene reset, execution, and analysis within a single auditable protocol.

Original authors: Shi Jin, Yuntian Wang, Yuhui Duan, Di Wu, Gaoqi Dong, Xiaohang Liu, Xiaotong Li, Hongfei Jia, Zehao Zhang, Tianyu Wang, Zhongjie Jia, Yuanqi Yao, Chenjia Bai, Zhaxizhuoma, Siao Liu, Nieqing Cao, Jin W

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shi Jin, Yuntian Wang, Yuhui Duan, Di Wu, Gaoqi Dong, Xiaohang Liu, Xiaotong Li, Hongfei Jia, Zehao Zhang, Tianyu Wang, Zhongjie Jia, Yuanqi Yao, Chenjia Bai, Zhaxizhuoma, Siao Liu, Nieqing Cao, Jin Wang, Chao Yu, Yan Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to fold laundry, stack cups, or pour beans. You've shown it thousands of videos of humans doing these tasks. But when you actually put the robot in a real kitchen, it might trip over a shadow, drop a cup because the lighting is slightly different, or get confused if the cup is blue instead of red.

For a long time, scientists tested these robots in video games (simulations) or with very specific, perfect setups. But as the paper explains, simulations are like driving a car in a video game: you can learn the rules, but you haven't felt the wind, the slippery road, or the surprise of a squirrel running across the street.

This paper introduces UMI-Bench 1.0, which is essentially a standardized "driving test" for real-world robots.

Here is a breakdown of what they did, using simple analogies:

1. The Problem: The "Recipe" vs. The "Kitchen"

The authors noticed that many robot learning systems use a specific way of collecting data called UMI (Universal Manipulation Interface). Think of UMI like a specific type of recipe book where the instructions are written from the robot's own "eyes" (a camera on its wrist) and its "hands" (the gripper).

The problem was that while everyone was using this "recipe book" to train robots, they weren't testing them in the same "kitchen."

One team might test on a table with a red mat.
Another might test with a blue mat and different lighting.
A third might reset the objects in a slightly different spot.

If Robot A fails and Robot B succeeds, is it because Robot B is smarter? Or is it just because Robot B got lucky with the red mat? There was no fair way to compare them.

2. The Solution: A Standardized "Driving Test"

The authors built UMI-Bench 1.0. Imagine this as a strictly regulated driving test center where every single car (robot) faces the exact same conditions.

The Same Track: They built a specific table with a grid on it (like a giant game board). Every robot must start with objects in the exact same spots.
The Same Camera: Every robot must look at the world through a camera mounted on its wrist, just like the training data. No "third-person" views (like a security camera looking down from the ceiling).
The Same Rules: They created 10 specific tasks, ranging from simple to complex.
- Simple: Stacking three baskets.
- Medium: Pouring beans from a cup into a basket using two hands.
- Hard: Folding a pair of pants or sorting mahjong tiles.
The "Surprise" Elements: To test if the robot is truly smart or just memorized the test, they changed things up.
- Factor A (The "New Outfit"): They swapped the objects for different colors or shapes (e.g., a green ball instead of a red one).
- Factor B (The "New Location"): They moved the objects to different spots on the grid.

3. How They Tested It

They took three different "student" robots (AI models named π0, π0.5, and DreamZero) and ran them through this standardized test 50 times for each task.

They didn't just ask, "Did it finish the task?" (Yes/No). They graded them like a teacher grading a math test:

Full Success: Did it get an A+? (Perfectly stacked, no spills).
Progress Score: Did it get partial credit? (It picked up the cup, but spilled a few beans).

4. What They Found (The Report Card)

The results were revealing:

The "Good" News: When the robot faced the exact same setup it was trained on, it did pretty well. It could handle the "Seen" conditions.
The "Bad" News: When the robot faced a new location (Factor B), it struggled much more than when it faced a new object (Factor A).
- Analogy: It's like a student who can solve a math problem perfectly if the numbers are in the same order, but gets completely confused if you move the numbers to a different page. The robots rely heavily on memorizing where things are, rather than understanding the geometry of the task.
The Hardest Tasks: Tasks that required long chains of actions (like rearranging a whole kitchen shelf) or very delicate timing (catching a can on a spinning turntable) were extremely difficult. All robots failed these almost 100% of the time.

5. Why This Matters

The paper argues that we need to stop comparing robots based on "who did the best in their own private lab."

UMI-Bench 1.0 is like a standardized national exam. It allows researchers to say, "Okay, Robot A is better at folding clothes, but Robot B is better at pouring liquids," with confidence that the difference is due to the robot's brain, not the lighting in the room.

It doesn't claim that robots are ready to replace humans in your home yet. Instead, it provides the ruler we need to measure how far we actually are from that goal, ensuring that when we say a robot is "learning," we are measuring real-world skill, not just simulation tricks.

Technical Summary: UMI-Bench 1.0

Problem Statement

While Vision-Language-Action (VLA) models have advanced rapidly in robotic manipulation, their evaluation on physical robots remains challenging. Existing real-world benchmarks (e.g., RoboChallenge, RoboArena, ManipArena) often fail to account for the specific coupling required by Universal Manipulation Interface (UMI)-style policies. UMI policies rely on a tight integration of wrist-view observations, action representations, data collection methods, and physical deployment. When evaluation protocols vary in hardware setup, camera placement, scene reset procedures, or action interfaces, performance differences may reflect evaluation noise rather than model capability. Furthermore, informal scene specifications hinder reliable failure analysis, making it difficult to distinguish between object-level generalization, spatial reasoning deficits, or execution instability. There is a lack of a standardized, reproducible benchmark that aligns UMI data collection with real-world deployment evaluation.

Methodology

UMI-Bench 1.0 addresses this gap by establishing a local-first, end-to-end real-robot benchmark specifically designed for UMI-style wrist-view manipulation policies. The methodology is structured around a unified protocol that links data collection, scene specification, and evaluation execution.

1. Standardized Infrastructure

Hardware Platform: The benchmark utilizes a fixed tabletop workstation (1.2m x 1.0m) equipped with a standardized robot arm, gripper, and wrist-mounted RGB camera (1280x1280 at 100 FPS). A transparent acrylic grid board with 5cm cells provides a repeatable reference for object reset and metadata recording, visible to operators but not assumed to be decoded by the policy.
Observation Interface: Policies are evaluated strictly using wrist-view RGB observations (optionally with proprioception). Third-person cameras are excluded from policy inputs to maintain alignment with UMI training assumptions.
Data-to-Evaluation Pipeline: The system connects a standardized data collection workflow (using FastUMI Pro) with a unified evaluation runner. This ensures that the training distribution and test distribution are comparable, with explicit metadata linking episodes to specific task factors.

2. Task Suite and Generalization Design

The benchmark comprises 10 real-world tabletop tasks: 4 single-arm and 6 bimanual tasks. These tasks cover a range of manipulation skills including spatial alignment, articulated container manipulation, tool use, bimanual coordination, dynamic grasping, semantic sorting, deformable object folding, and long-horizon rearrangement.

To diagnose generalization, each task is evaluated under a two-factor shift taxonomy:

Factor A (Object/Appearance Shifts): Introduces held-out object instances, appearances, categories, or combinations (e.g., different colors, materials, or object types).
Factor B (Layout/Dynamics Shifts): Introduces held-out positions, layouts, poses, or task dynamics (e.g., different grid locations, turntable speeds, or object orientations).

This creates four evaluation cells per task: Seen/Seen, Seen/Unseen (Factor A), Unseen/Seen (Factor B), and Unseen/Unseen. The benchmark includes approximately 20,000 demonstrations across 10 tasks and 50 evaluation episodes per task.

3. Evaluation Protocol and Metrics

Scoring: Evaluation is based on the final stable physical state of the rollout. Two primary metrics are reported:
- Full Success Rate (FSR): The fraction of episodes satisfying all task-critical subgoals simultaneously.
- Progress Score: A 0–100 partial-credit score based on task-specific subgoals (e.g., grasp, lift, place, stabilize).
Generalization Gap: Defined as $\Delta_{gen} = S_{seen} - S_{unseen}$ , computed separately for Factor A and Factor B to distinguish between object/appearance failures and spatial/dynamic failures.
Reproducibility: The protocol includes detailed checklists for scene reset, logging, and human scoring to ensure auditability.

Key Contributions

First UMI-Dedicated Benchmark: UMI-Bench 1.0 is the first benchmark dedicated to the real-world evaluation of UMI-based manipulation models, addressing the misalignment between UMI data collection and physical deployment.
Unified Evaluation Protocol: It standardizes the entire pipeline from data collection to result logging, reducing variation in hardware, observations, and scene reset to ensure reproducible comparisons.
Structured Generalization Analysis: By organizing unseen conditions into object/appearance (Factor A) and spatial/dynamic (Factor B) shifts, the benchmark enables interpretable diagnosis of policy failures.
Comprehensive Task Suite: It provides a diverse set of 10 tasks (including bimanual and deformable manipulation) with ~20k demonstrations and standardized evaluation episodes.

Experimental Results

The authors evaluated three state-of-the-art models: $\pi_0$ , $\pi_0.5$ , and DreamZero.

Overall Performance: $\pi_0.5$ achieved the highest average Overall Progress Score (55.84), outperforming $\pi_0$ (48.90) and DreamZero (40.59). $\pi_0.5$ ranked first on six of the ten tasks, demonstrating robustness across both single-arm and bimanual settings.
Task-Specific Strengths:
- $\pi_0$ performed best on single-arm placement tasks (T1, T2).
- DreamZero showed competitive performance on tool-mediated stamping (T3) and cooperative transport (T6) but trailed on average.
- All models achieved 0% Full Success Rate on T3 (Tool-Mediated Stamping) and T9 (Long-Horizon Rearrangement), highlighting current limitations in long-horizon stage estimation and tight final-placement accuracy.
Generalization Trends: Performance degradation was significantly more severe under Factor B (spatial/dynamic) shifts than under Factor A (object/appearance) shifts. The mean score dropped from 59.62 (Seen/Seen) to 45.33 (Factor B) and 40.19 (Combined shifts). This suggests that current policies rely heavily on demonstration-aligned motion priors and struggle when task-relevant geometry or dynamics change.
Latency Observations: DreamZero exhibited significantly higher inference latency (1630ms) compared to $\pi_0$ and $\pi_0.5$ , which may contribute to its poor performance on dynamic tasks like T7 (Dynamic Pick-and-Place).

Significance and Claims

The paper claims that UMI-Bench 1.0 provides a practical infrastructure for measuring and diagnosing real-world manipulation policies. By treating data collection, scene specification, and evaluation as a single coupled protocol, it makes the relationship between training distributions and test distributions explicit and analyzable.

The authors emphasize that the benchmark is designed for iterative model development within a specific data and deployment paradigm (local-first, wrist-view), rather than replacing remote robot platforms. The results indicate that while policies retain capability when training and evaluation distributions are aligned, significant gaps remain in handling spatial shifts and long-horizon planning. The benchmark aims to facilitate the identification of these specific failure modes, guiding future improvements in data collection strategies and model architectures for scalable robot learning.