💻 computer science

HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

The paper introduces HABIT, a large-scale dataset of over 10,000 human-present robot manipulation episodes organized into Collaborator, Coworker, and Supervisor roles, which enables policies to learn essential human-aware behaviors like synchronization, yielding, and gesture grounding that are absent in traditional human-absent datasets.

Original authors: Jaehwi Song, Suchae Jeong, Byeongguk Jeon, Sungdong Kim, Minjoon Seo, Hyungmok Son, Kimin Lee

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaehwi Song, Suchae Jeong, Byeongguk Jeon, Sungdong Kim, Minjoon Seo, Hyungmok Son, Kimin Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to do chores. Most robots today are trained in a "ghost town" scenario: they practice moving boxes, wiping tables, and sorting trash in an empty room where no humans are around. They become very good at the physical motions, but they have no idea how to act when a real person is standing right next to them. They might bump into you, ignore your hand gestures, or try to grab a tool while you are still holding it.

The paper introduces HABIT (Human-Aware Behavior and Interaction Training), a new dataset designed to fix this. Think of HABIT not just as a library of robot moves, but as a library of dance steps where the robot and a human partner are learning to move together.

The Three "Dance Styles"

The researchers organized the tasks into three distinct roles, like different genres of dance, to teach the robot how to handle different social situations:

The Collaborator (The Tango):
- The Scenario: The human and robot must work together on the same object at the same time.
- The Analogy: Imagine two people trying to carry a heavy couch up a staircase. If one person moves too fast or too slow, they trip. In this role, the robot learns to wait for the human to lift a corner before it moves, or to hand over a tool exactly when the human reaches for it. It's all about synchronization.
- Example: A human places a napkin on a tray, and the robot lifts the tray to help.
The Coworker (The Conga Line):
- The Scenario: The human and robot are in the same room doing different tasks, but they share the same space.
- The Analogy: Imagine two people cooking in a small kitchen. One is chopping vegetables, and the other is stirring a pot. They aren't touching each other, but they have to dodge around each other so they don't collide. The robot learns to yield (step aside) if the human walks into its path, rather than plowing forward.
- Example: A human sorts trash on one side of a table while the robot packs boxes on the other side, constantly moving out of each other's way.
The Supervisor (The Conductor):
- The Scenario: The human gives directions, and the robot follows.
- The Analogy: Imagine a conductor waving a baton to tell an orchestra when to play. The robot learns to watch the human's hands and eyes. If the human points at a specific donut, the robot doesn't just grab the first one it sees; it looks at the finger and grabs that specific donut. It learns gesture grounding.
- Example: A human points to a specific container, and the robot puts bread inside it.

How They Collected the Data

To make sure the robot actually learned these social skills, the researchers didn't just let the human and robot play around randomly. They used a strict "script" (called a workflow) that forced the human and robot to react to each other in real-time.

No "Cheat Codes": The human operator couldn't just guess what the robot would do next. They had to wait for the robot to move before reacting, and vice versa. This ensured the robot learned to respond to actual human cues, not just memorized patterns.
Safety First: If the robot looked like it was about to bump into the human, the human operator would immediately pull the robot back. This taught the robot that "stopping" is a valid and important move.

What Happened When They Tested It?

The researchers took two existing robot brains (AI models) and trained them on this new HABIT data. They compared these "Habit-trained" robots against robots trained only on the old "ghost town" data.

The Results: The HABIT-trained robots were significantly better.
- In the Coworker role, they stopped crashing into humans.
- In the Supervisor role, they actually looked at where the human pointed instead of guessing.
- In the Collaborator role, they timed their moves perfectly with the human.
The "Superpower": The most interesting finding was sample efficiency. When they tried to teach the HABIT-trained robots a new task they hadn't seen before, they learned it much faster (using fewer examples) than the robots trained on the old data. It's as if HABIT taught the robot the concept of "being polite and aware," which made learning new specific chores much easier.

The Bottom Line

The paper argues that for robots to be useful in our homes and offices, they can't just be strong and precise; they have to be socially aware. By training robots on data where humans are actually present and interacting, we get robots that don't just work around us, but work with us.

Limitations: The authors admit this was done in a controlled lab with a specific type of robot and a limited number of human operators. While they varied the humans' clothes and body types to test adaptability, real-world environments are messier than their setup. However, the core message stands: robots need to see humans to learn how to be good partners.

Technical Summary: HABIT – Human-Aware Behavior and Interaction Training Dataset

1. Problem Statement

Recent progress in general-purpose robot manipulation has been driven by large-scale demonstration datasets. However, a critical gap exists: existing datasets are predominantly collected in human-absent settings, where the robot acts as the sole agent. Consequently, policies trained on such data, while competent in isolation, fail to exhibit human-aware behaviors necessary for shared environments (e.g., homes, factories).

Specifically, robots cannot learn to:

Yield to avoid collisions when a human enters their path.
Synchronize temporally and spatially when jointly manipulating objects.
Ground gestures to interpret human intent without explicit language instructions.

These behaviors are missing not because they are difficult to learn, but because they cannot be demonstrated without a human present in the scene. This paper addresses the need for a dataset that explicitly captures human-robot interaction (HRI) dynamics.

2. Methodology: The HABIT Dataset

2.1 Task Design and Roles

The dataset organizes 60 distinct tasks into three interaction roles, defined by the dependency between the human and the robot:

Collaborator: The human and robot jointly accomplish a shared goal through direct physical interaction (e.g., handing over tools, jointly holding objects). This requires tight spatiotemporal synchronization.
Coworker: The human and robot pursue separate tasks within a shared workspace without direct physical contact. The primary challenge is collision avoidance and yielding.
Supervisor: The human directs the robot via explicit cues (e.g., gestures, pointing) while the robot infers intent from visual input alone.

2.2 Data Collection Protocol

To ensure the data captures learnable human-aware behaviors, the authors implemented specific collection protocols:

Reactive Interaction: Operators (both human and robot) are prohibited from pre-executing subtasks based on memory. They must act only after directly observing their partner's behavior. This ensures that the cues triggering robot actions are visible in the recorded observations.
Behavior Elicitation:
- Safety-First Yielding: Robot operators are instructed to decisively retract arms if a collision is imminent, recording yielding as the default response.
- Temporal Adaptation: Human movement speeds are varied to prevent the robot from learning a fixed cadence.
- Gesture Grounding: Human operators sample wait times before pointing, forcing the policy to attend to the gesture rather than acting prematurely on language instructions.
Diversification: Conditions are varied across episodes (clothing colors, object order) and across operators (different body types) to support out-of-distribution (OOD) generalization.

2.3 Hardware and Setup

Robots: Two Franka Research 3 arms with Robotiq grippers, teleoperated via Meta Quest 3 controllers.
Sensors: Five synchronized RGB cameras:
- Three robot-side (wrist views and an egocentric view capturing the human and workspace).
- Two human-side (human egocentric view and an exocentric view of the full workspace).
Scale: The dataset comprises 10,563 episodes totaling 164.19 hours of bimanual manipulation.

3. Evaluation Framework

The authors evaluate policies using a subset of 6 representative tasks (2 per role). Success is defined by three binary criteria for every robot subtask:

Manipulation: Correct physical execution and object state.
Workflow Compliance: Adherence to the task workflow, including ordering, synchronization, and cue following.
Human Safety: No collisions or contact with human-held objects.

The primary metric is the Success Rate over independent trials.

4. Key Results

4.1 Task Success Rates

The authors fine-tuned two open-source Vision-Language-Action (VLA) models, $\pi_0.5$ and GR00T N1.6, on HABIT and compared them against a Robot-only baseline (collected without a co-present human).

Collaborator Tasks: Robot-only baselines are not applicable (N/A) as these tasks inherently require a human partner.
Coworker Tasks: HABIT-trained models showed the largest gains (e.g., $\pi_0.5$ improved from 40% to 95% on Waste Sorting). This highlights the necessity of learning reactive yielding to resolve path conflicts.
Supervisor Tasks: HABIT provided significant gains on complex tasks requiring gesture grounding (e.g., Food Storage), where language alone was insufficient to disambiguate targets. On simpler tasks (Donut Serving), performance was comparable to Robot-only.

4.2 Emergent Human-Aware Behaviors

Training on HABIT elicited specific behaviors absent in Robot-only models:

Spatiotemporal Synchronization: In Collaborator tasks, models learned to wait for human cues before acting.
Proactive Yielding: In Coworker tasks, models successfully avoided collisions by retracting when humans approached.
Gesture Grounding: In Supervisor tasks, models learned to interpret pointing gestures to identify targets.

4.3 Failure Analysis

The study identified role-specific failure modes in Robot-only models:

Collaborator: Precondition violations (acting before the human is ready).
Coworker: Collisions due to lack of yielding.
Supervisor: Failure to follow gestures.
HABIT-trained models substantially reduced these specific failure rates across both architectures.

4.4 Sample-Efficient Adaptation

The authors demonstrated that mid-training a VLA on HABIT enables rapid adaptation to new HRI tasks.

When fine-tuned on downstream tasks with limited data (50–200 demonstrations), models pre-trained on HABIT outperformed models trained directly on the downstream data.
For example, on Waste Sorting, HABIT mid-training with 50 demonstrations matched the performance of direct fine-tuning with 200 demonstrations.

5. Significance and Claims

The paper positions HABIT as a stepping stone toward robot foundation models that are not merely capable, but genuinely safe and socially compatible in human-inhabited environments.

New Axis of Diversity: The authors claim that introducing "human presence" as a dimension of dataset diversity is critical. They argue that standard robot learning pipelines, which rely on human-absent data, miss the signal required for safe co-existence.
Generalization: The results suggest that human-aware behaviors (yielding, synchronization, gesture grounding) are learnable from data and transfer across different VLA architectures.
Modesty: The authors acknowledge limitations, noting the dataset is collected in a single environment with a one-to-one human-robot configuration. They do not claim HABIT solves all multi-agent or complex social dynamics but assert it provides a necessary foundation for learning basic co-presence behaviors.

In conclusion, HABIT demonstrates that training on human-present data yields policies that are not only more successful in shared spaces but also exhibit emergent social behaviors that cannot be derived from robot-only demonstrations.