🤖 machine learning

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

This paper introduces "Humanoid Everyday," a large-scale, diverse dataset comprising over 10,000 multimodal trajectories across 260 tasks designed to advance open-world humanoid manipulation research, accompanied by an analysis of policy learning methods and a standardized cloud-based evaluation platform.

Original authors: Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, Yue Wang

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, Yue Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to be a helpful roommate. You want it to walk around the house, pick up a cup, fold a towel, and even hand you a tool. But right now, most robot training data is like a library full of books about how to move a single robotic arm sitting on a table. It's great for that arm, but it doesn't teach a whole robot how to walk, balance, and use two hands at the same time.

The paper "Humanoid Everyday" introduces a massive new solution to this problem. Think of it as the "Encyclopedia of Robot Daily Life."

Here is a simple breakdown of what the researchers built and why it matters:

1. The Dataset: A "Robot Reality Show"

The researchers created a huge collection of video and sensor data called Humanoid Everyday.

The Cast: They used two advanced humanoid robots (the Unitree G1 and H1) that look and move like humans, complete with dexterous hands that can feel touch.
The Plot: Instead of just moving a box from point A to point B, they recorded 260 different "scenes" or tasks. These range from simple things like picking up a dumpling toy to complex ones like walking while carrying a door handle or folding a towel.
The Variety: The tasks cover seven different categories, including:
- Basic chores (picking things up).
- Dealing with squishy things (like cloth or towels).
- Using tools (like an eraser).
- Teamwork (handing objects to a human).
- Walking and working (moving around while doing tasks).
The Scale: They recorded over 10,000 video clips (trajectories) and more than 3 million frames of data. Every clip includes video, depth (3D distance), laser scans (LiDAR), and even what the robot's hands "feel" (tactile data).

2. The Secret Sauce: A "Super-Fast Camera Crew"

Collecting this data is hard because robots move fast, and if the camera lags even a tiny bit, the data is useless.

The Problem: The official robot software was like a slow, single-lane road where data got stuck in traffic.
The Fix: The team built a new "teleoperation" system (a way for humans to control the robot remotely). They treated the data collection like a high-speed relay race. Instead of one person doing everything, they split the work into different "processes" (like different runners) that run simultaneously.
The Result: They cut the delay (lag) from 500 milliseconds down to just 2 milliseconds. This is like going from a snail's pace to a sprint, allowing them to capture smooth, high-quality movements that actually look like a human doing them.

3. The Cloud "Test Track"

One of the biggest headaches in robot research is that not every scientist owns a $100,000 humanoid robot.

The Innovation: The team built a cloud-based evaluation platform.
How it works: Imagine a remote control track. Researchers from anywhere in the world can upload their robot "brain" (their AI policy) to the cloud. The cloud then sends the robot's eyes (camera feed) to the researcher's computer. The researcher's computer figures out what to do and sends the commands back to the physical robot in the lab.
The Benefit: This creates a fair playing field. Everyone tests their robot on the exact same real-world tasks, in the exact same room, with the exact same robot. No more guessing if your robot is better because it has a different camera or a different table.

4. The Reality Check: What the Robots Can (and Can't) Do

The researchers didn't just collect data; they tested it. They took several famous AI models (the "brains" currently used in robotics) and tried to teach them these new tasks.

The Good News: The models got better at some things, especially when they had seen a lot of similar data before (pre-training). The best performer was a model called GR00T N1.5, which succeeded in about 51% of the trials.
The Bad News: The robots still struggle with the hardest stuff.
- The "Rose in a Vase" Problem: In a task where the robot had to walk over and put a thin rose stem into a tiny vase opening, almost every model failed (0% success rate).
- Why? The robots are good at big movements but terrible at the tiny, precise adjustments needed for delicate tasks. They also get confused when they have to walk and move their hands at the same time.

Summary

Humanoid Everyday is a massive, high-quality library of a robot's daily life, collected with a super-fast system that captures every detail. It comes with a "cloud test track" so anyone can test their robot brains on real hardware.

The paper shows that while we are making progress, current robot "brains" are still like toddlers learning to walk: they can do simple things, but they stumble when asked to walk and do fine motor skills (like threading a needle) at the same time. This dataset is the training ground needed to help them grow up.

Technical Summary: Humanoid Everyday

Problem Statement

Despite significant strides in humanoid robotics regarding full-body capabilities (locomotion, dancing, running), the development of general-purpose humanoid manipulation policies is hindered by three primary gaps:

Dataset Limitations: Existing datasets predominantly focus on stationary robotic arms or simple mobile platforms. The few existing humanoid datasets are often confined to fixed environments, lack task diversity, omit lower-body locomotion, and fail to capture complex human-humanoid interactions.
Evaluation Scarcity: There is a lack of standardized evaluation platforms for benchmarking learning-based policies on humanoid data. Current frameworks are largely restricted to robotic arms or simulated agents, making fair, rigorous comparisons across diverse humanoid policies difficult.
Embodiment Gap: Current datasets often fail to capture the full spectrum of human-like capabilities required for real-world scenarios, including bimanual coordination, full-body motion, and dexterous interaction with deformable objects and tools.

Methodology

1. Data Collection Pipeline

The authors developed a highly efficient, human-supervised teleoperation pipeline to collect the Humanoid Everyday dataset.

Hardware: Data was collected using two Unitree humanoid robots: the G1 (29 DoF, equipped with 7-DoF Dex3-1 dexterous hands with tactile sensors) and the H1 (27 DoF, equipped with 6-DoF INSPIRE hands). Both robots carry Intel RealSense RGB-D cameras and Livox LiDAR systems.
Teleoperation Interface: Operators wear an Apple Vision Pro to capture wrist and finger keypoints. Finger motions are mapped to robot hands via a dex-retargeting system, while wrist poses are converted to arm joint commands using a Pinocchio-based inverse kinematics (IK) algorithm.
Pipeline Optimization: Unlike the blocking, synchronous design of the official Unitree teleoperation script, the authors implemented a multi-processing, asynchronous I/O pipeline. This decouples data streaming, writing, robot control, and IK computation into separate processes/threads using shared memory buffers.
- Performance: This re-engineering reduced control delay from 500 ms to 2 ms and halved the data collection time compared to the official system, enabling high-frequency (30 Hz) control and synchronized multimodal data capture.

2. Dataset Composition

The dataset comprises 10.3k trajectories and over 3 million frames covering 260 unique tasks across 7 categories:

Basic Manipulation: Pick-and-place tasks.
Deformable Manipulation: Interacting with cloths or soft objects.
Articulated Manipulation: Operating hinged or jointed structures.
Tool Use: Utilizing external objects to achieve goals.
High-Precision Manipulation: Tasks requiring high accuracy (e.g., inserting a stem).
Human-Robot Interaction (HRI): Cooperative actions with humans.
Loco-Manipulation: Combining locomotion with manipulation.

Modalities: Each episode includes RGB video, depth maps, LiDAR scans, tactile feedback, IMU data, joint poses, joint actions, and natural language annotations.

3. Cloud-Based Evaluation Platform

To address the lack of standardized evaluation, the authors introduced a cloud-based evaluation platform.

Functionality: Researchers can upload policies to a server, which streams real-time visual inputs (RGB, Depth) and robot states from a physical G1 or H1 robot to the client.
Execution: Users run inference locally and send action commands back to the server for execution on the physical robot.
Monitoring: A web interface provides third-person and egocentric views of task execution.
Efficiency: The system demonstrated the ability to run continuously for over 100 minutes with minimal human intervention (only 3 interventions for motor overheating), enabling reproducible, hardware-agnostic benchmarking.

4. Policy Analysis Experiments

The authors evaluated a suite of representative imitation learning and Vision-Language-Action (VLA) models on the dataset:

Models Tested: Diffusion Policy (DP), 3D Diffusion Policy (DP3), Action Chunking with Transformers (ACT), OpenVLA, $\pi_0$ -FAST, $\pi_0.5$ , and GR00T N1.5.
Training Strategy: A two-stage fine-tuning approach was used for VLA models: pre-training on the full Humanoid Everyday dataset followed by adaptation to specific task categories.
Ablation Study: A comparison was conducted between direct task-specific fine-tuning and the two-stage pre-training strategy.

Key Results

1. Policy Performance

General Struggle: All end-to-end imitation policies struggled significantly due to the high-dimensional action space (28 DoFs).
Model Specifics:
- DP3 generally outperformed DP, attributed to the robustness of 3D point cloud observations, though it failed in Loco-Manipulation tasks where large frame-to-frame changes made point clouds less reliable than RGB.
- ACT performed poorly, overfitting to demonstrated trajectories and failing to adapt to scene variations.
- OpenVLA struggled with high-frequency (30 Hz) data; downsampling to 2 Hz improved motion generation but reduced smoothness.
- $\pi_0$ -FAST faced decoding errors due to high-dimensional action tokenization, sometimes causing the robot to halt.
- $\pi_0.5$ produced smoother actions but still exhibited overfitting.
- GR00T N1.5 achieved the highest average success rate (51%), attributed to extensive pretraining on large-scale humanoid datasets.
Failure Modes: The most challenging categories were Loco-Manipulation and High-Precision Manipulation. In the "Insert rose into vase" task, nearly all policies achieved a 0% success rate, highlighting a lack of fine-grained visuospatial perception in current models.

2. Pretraining Utility

The ablation study confirmed that pretraining on Humanoid Everyday prior to task-specific fine-tuning consistently improved the performance of VLA models. This suggests that exposure to diverse humanoid behaviors provides a valuable prior that enhances robustness and stability in downstream manipulation tasks.

Significance and Claims

The paper positions Humanoid Everyday as a foundational resource to advance research in general-purpose humanoid manipulation. Its significance is threefold:

Data Diversity: It bridges the gap in existing datasets by offering a large-scale, multimodal collection that spans diverse environments, full-body locomotion, dexterous manipulation, and human interaction.
Standardized Benchmarking: The accompanying cloud-based evaluation platform is the first designed specifically for humanoid robots, lowering the barrier to entry for researchers without physical hardware access and enabling fair, reproducible comparisons.
Insight into Limitations: The analysis of current policies reveals that while large VLA models show promise, significant challenges remain in high-dimensional control, fine-grained perception, and robustness to dynamic environments.

The authors conclude that while current imitation learning methods face substantial hurdles, the combination of this dataset, the evaluation platform, and the insights gained from policy analysis lays the groundwork for developing more capable and embodied robotic agents in real-world scenarios. They acknowledge limitations, such as the need for specialized model designs for high-dimensional spaces and the current inability of policies to autonomously reset environments, identifying these as directions for future work.