💻 computer science

HRIBench: Benchmarking Interaction-Centric Human-Robot Collaboration

This paper introduces HRIBench, a diagnostic benchmark featuring structured interaction scenarios and specialized metrics to evaluate and improve human-robot collaboration by addressing the critical gap in intent understanding, temporal synchronization, and protocol adherence that current vision-language-action models fail to capture.

Original authors: Chang Liu, Jiawei Zhang, Tao Zhang, Ye Wang, Hongyu Zhou, Qin Jin

Published 2026-07-16

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Chang Liu, Jiawei Zhang, Tao Zhang, Ye Wang, Hongyu Zhou, Qin Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just lonely workers doing tasks in empty rooms, but partners in a busy, chaotic dance with humans. For a long time, scientists have been teaching robots how to grab, lift, and move objects using "Vision-Language-Action" models. Think of these models as a robot's brain that can see an object, read a command like "pick up the cup," and figure out how to move its arm to do it. These robots are getting pretty good at solo acts, like a magician performing a trick alone on stage. But real life isn't a solo performance; it's a duet. In the real world, robots need to work with people, which means they have to understand what a human is thinking, wait for the right moment to move, and stop immediately if a human gets in the way. The big question researchers are asking is: Can a robot that is great at moving objects also be great at working with people, or does it freeze up when the human starts dancing?

This is exactly what a new study called HRIBench sets out to investigate. The researchers, a team from Renmin University of China and Beijing Normal University, realized that while we have plenty of tests to see if a robot can pick up a block, we don't have good ways to test if a robot can collaborate with a human. To fix this, they built a new "gym" for robots called HRIBench. Instead of just watching a robot try to move an object, HRIBench puts the robot in a script where a human plays a specific role. Sometimes the human is a Teacher (showing the robot what to do with a gesture), sometimes a Partner (helping hold an object together), and sometimes an Intruder (unexpectedly getting in the way to see if the robot panics or stays safe).

The team created 13 different scenarios with over 650 different "episodes" to test top-tier robot brains like GR00T, π0.5, and ACT. The results were a bit of a wake-up call. Even though these robots are incredibly strong at moving things on their own, they struggled mightily when asked to collaborate. When the human was just a teacher, the robots succeeded about 53% of the time. When they had to work as partners, that dropped to 50%. But when a human acted as an "Intruder" and disrupted the task, the robots' success rate plummeted to just 10%. They often kept trying to do the task even when it was unsafe or when the human had clearly changed the plan.

However, the story doesn't end with failure. The researchers found that by training the robots specifically on the data from their new HRIBench gym, the robots actually got much better at working with humans. In a real-world test using a physical robot arm, the success rate of a task jumped from 10% (when trained only on real human data) to 43% after the robot was first trained on the HRIBench simulations. This suggests that while today's robots are still clumsy dance partners, giving them the right kind of practice in a simulated world can help them learn to move in sync with us, turning a solo act into a successful collaboration.

Technical Summary: HRIBench – Benchmarking Interaction-Centric Human-Robot Collaboration

Problem Statement
Current Vision-Language-Action (VLA) benchmarks primarily evaluate isolated manipulation skills (e.g., pick-and-place, object operation) under the implicit assumption that robots operate alone. However, real-world human-robot collaboration (HRC) fundamentally requires coordination under shared agency. Existing evaluations fail to model the underlying interaction structure, leaving critical capabilities untested: intent understanding, temporal synchronization, protocol adherence, and safe interaction in dynamic environments. Consequently, a policy may demonstrate strong manipulation competence in isolation but fail to respond to a collaborator's reaching motion, violate coordination timing, or continue execution after disruptive human intervention. This gap highlights a disconnect between manipulation competence and collaborative intelligence.

Methodology
To address this, the authors introduce HRIBench, a diagnostic benchmark designed to evaluate intent-aware human-robot collaboration through executable interaction scenarios.

Scenario Scripts: The core abstraction is the "scenario script," a structured, executable representation of collaborative episodes. These scripts explicitly model:
- Agent roles and semantic goals.
- Temporal dependencies and causal constraints.
- Human motion distributions.
- Reward functions and success conditions.
  This abstraction separates what collaboration requires from how a policy executes it, allowing for interpretable semantic layers connecting high-level requirements to simulation environments.
Interaction Roles: The benchmark organizes behavior around three fundamental human-robot interaction patterns:
1. Instructor: Robots must infer intended actions from human demonstrations or gestures.
2. Collaborator: Robots must engage in synchronized multi-agent coordination (e.g., object handover, joint manipulation) requiring precise timing and responsiveness.
3. Intruder: Robots must maintain safety and adapt execution online under disruptive human intervention (e.g., blocking paths, invalid interference).
Generation Pipeline: HRIBench utilizes a four-stage pipeline:
1. Scripting: Language models generate scenario scripts from task metadata.
2. Scene Instantiation: Environments are compiled with diverse objects and layouts.
3. Motion Synthesis: Human trajectories are synthesized using HY-Motion-1.0, refined with task-aware constraints (e.g., hand stability, approach directions), and varied across 50+ trajectory variants per task.
4. Simulation Validation: Generated episodes are filtered for reachability, validity, and semantic correctness. The final benchmark comprises 13 role-conditioned tasks with over 650 executable evaluation episodes.
Evaluation Metrics: Beyond binary task success, HRIBench introduces interaction-centric metrics:
- Effectiveness: Collaboration Success Rate (CSR), Completion Time, Idle Ratio.
- Coordination: Temporal Synchronization, Response Latency, Order Compliance.
- Safety & Robustness: Collision-Free Rate, Human Contact Safety, Contradictory Instruction Recognition, Disruption Success Rate.

Key Contributions

HRIBench Benchmark: A diagnostic framework reformulating interaction evaluation as executable coordination modeling, featuring 13 tasks spanning instructor-guided execution, collaborative coordination, and robustness under intervention.
Scenario Scripts: A structured, executable representation that explicitly models agent roles, temporal dependencies, and human behavior distributions, enabling the generation of reproducible interaction episodes.
Interaction-Centric Evaluation Protocol: A multi-dimensional metric suite (synchronization, responsiveness, protocol compliance, safety) that allows for the disentangled analysis of collaborative failure modes across modern robot foundation models.

Experimental Results
The authors evaluated adapted policies based on GR00T N1.5, $\pi^0.5$ , and ACT under a unified protocol.

Simulation Performance: Current foundation policies struggle significantly in collaborative settings despite strong manipulation abilities.
- Instructor Tasks: Best CSR was 0.533.
- Collaborator Tasks: Best CSR dropped to 0.500.
- Intruder Tasks: Performance collapsed, with CSR dropping to only 0.100 across all models.
- Analysis: While models achieved high Order Compliance (often 1.000), they failed to yield, maintain safe clearance, or recover from disruption. VLA-based policies generally outperformed the imitation-learning baseline (ACT) in cue grounding, but all struggled with temporal coordination and human perturbation.
Real-World Adaptation: A small-scale study using a LeRobot SO-100 arm demonstrated the utility of HRIBench data for physical deployment.
- Real-only post-training (20 real teleoperation trajectories) achieved an average success rate of 0.10.
- Sim+Real post-training (pre-training on HRIBench simulation data, then fine-tuning on real data) improved the average success rate to 0.43.
- This improvement was consistent across all three interaction roles, suggesting that structured interaction data provides transferable priors for real-world collaborative behavior when real demonstrations are scarce.

Significance and Claims
The paper claims that HRIBench provides a necessary diagnostic tool to expose the limitations of current embodied foundation models in collaborative settings. The authors argue that binary task success is insufficient for evaluating collaborative intelligence; a robot may complete an object-level goal while violating temporal coordination or safety protocols.

By shifting the focus from isolated manipulation to "executable coordination," HRIBench reveals that current policies lack robust intent-aware behavior and temporal coordination. The benchmark demonstrates that while manipulation competence is a prerequisite, it is not sufficient for collaboration. The authors modestly claim that their work advances interaction-centric robot learning by providing a structured framework to diagnose specific coordination deficiencies (e.g., intent grounding, timing, safety recovery) and by generating data that effectively improves physical adaptation in data-scarce scenarios. They acknowledge limitations, noting that as a simulated benchmark, it cannot fully capture real-world human variability or serve as a standalone safety evaluation for physical deployment.

Technical Summary: HRIBench – Benchmarking Interaction-Centric Human-Robot Collaboration

More like this