🤖 AI

DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Target

This paper introduces DynaHOI-Gym, a unified online platform, and DynaHOI-10M, a large-scale benchmark with 10 million frames, to address the gap in hand-object interaction research by focusing on dynamic targets and time-critical coordination, alongside a baseline method that improves location success rates by 8.1%.

Original authors: BoCheng Hu, Zhonghan Zhao, Kaiyue Zhou, Hongwei Wang, Gaoang Wang

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: BoCheng Hu, Zhonghan Zhao, Kaiyue Zhou, Hongwei Wang, Gaoang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of catch with a friend. If your friend throws a ball that sits still in the air, catching it is easy. You just reach out and grab it. But what if your friend is throwing the ball while running, spinning, or bouncing it off a wall? To catch it, you can't just react; you have to predict where the ball will be a split second from now and move your hand there before the ball gets there.

This paper, "DynaHOI," is about teaching robots (specifically their robotic hands) to do exactly that: catch moving objects in real-time.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Static" Trap

For a long time, scientists have been teaching robots how to pick up things. But almost all their training was like teaching a robot to pick up a stationary apple sitting on a table.

The Reality: In the real world, things move. Balls roll, people walk by, and tools slide.
The Gap: Current robot brains are great at grabbing a still apple but terrible at catching a flying one. They are like a goalie who is amazing at stopping a ball that isn't moving, but freezes when the ball is flying toward them.

2. The Solution: A New Training Ground (DynaHOI-Gym)

To fix this, the authors built a virtual video game gym called DynaHOI-Gym.

Think of it like a flight simulator: Just as pilots practice in a simulator before flying a real plane, robots practice catching moving objects here.
The Features: In this gym, objects don't just sit there. They fly in circles, bounce like balls, swing like pendulums, or roll down ramps. The gym creates millions of these scenarios automatically, so the robot can practice thousands of hours in a short time.

3. The Big Dataset: The "10 Million Frame" Library

The authors didn't just build the gym; they filled it with a massive library of practice runs called DynaHOI-10M.

The Scale: Imagine a library with 10 million video frames and 180,000 different catching attempts.
The Variety: It covers 11 different types of objects (like apples, balls, bottles) and 22 different ways they can move (spinning, bouncing, rolling).
Why it matters: This is the "textbook" robots need to learn the physics of moving targets.

4. The New Strategy: "Look Before You Leap"

The paper introduces a new way for robots to think, called "ObAct" (Observe-then-Act).

The Old Way: Most robots look at the object right now and immediately try to grab it. This is like trying to catch a speeding car by looking at it for one second and jumping in front of it. You'll miss.
The New Way (ObAct): The robot is taught to watch the object for a few seconds first. It studies the pattern: "Ah, the apple is moving in a circle. It will be at the top in 0.5 seconds." Then, it moves its hand to that future spot.
The Result: This simple change of "watching first" made the robots significantly better at catching things, improving their success rate by about 8%.

5. The Test: How Did the Robots Do?

The authors tested the smartest robot brains available today (including models from big tech companies) in this new gym.

The Scorecard: They didn't just check if the robot caught the object. They checked:
- Did it get close enough? (Localization)
- Did it actually close its fingers? (Grasping)
- Was the movement smooth? (Did it jerk around or move gracefully?)
- How fast was it?
The Verdict: Even the best robots struggled. They could often get close, but they failed to actually grab the moving object. It's like a baseball player who runs to the right spot to catch a fly ball but drops it because their hands weren't ready.
The Winner: The new "ObAct" method (the one that watches first) beat all the other models, proving that anticipation is the key to success.

Summary

This paper is a wake-up call for robotics. It says, "Stop teaching robots to pick up static objects; the real world is dynamic."

They built a massive virtual playground with moving targets, created a huge dataset of practice runs, and showed that the secret to catching moving things isn't just having fast hands—it's having a brain that can predict the future by watching the present.

The Takeaway: To teach a robot to be a master catcher, you have to teach it to be a prophet first.

1. Problem Statement

Current benchmarks for Hand-Object Interaction (HOI) and hand motion generation predominantly focus on static objects. They rely on frame-by-frame alignment with ground truth (GT) trajectories, which fails to capture the critical challenges of dynamic scenarios where targets are moving.

The Gap: Real-world manipulation requires temporal prediction and motion anticipation (e.g., intercepting a moving ball). Existing models struggle to forecast where a target will be in the near future and execute precise motor commands to capture it.
Limitations of Current Evaluation: Direct physical testing is costly and hard to reproduce. Existing datasets lack parameterized motion control and online, closed-loop evaluation, making it difficult to assess a model's ability to reason about time-critical coordination.

2. Methodology & Framework

The authors introduce a comprehensive ecosystem consisting of a simulation platform, a large-scale dataset, and a novel baseline model.

A. DynaHOI-Gym (Simulation Platform)

A unified, online, closed-loop evaluation platform built on the Unity physics engine.

Function: It allows agents to interact directly with a simulated environment rather than just predicting static frames.
Key Features:
- Parameterized Motion Generators: Targets follow predefined continuous trajectories (e.g., circular, projectile, pendulum) with controllable physical parameters.
- Observe-Before-Act Protocol: Unlike static tasks, the hand first observes the moving target for a short window to infer motion dynamics before executing an interception action.
- Real-time Synchronization: Ensures strict temporal alignment between the model's inference and the simulator's physics updates.

B. DynaHOI-10M (Dataset)

A large-scale benchmark built on DynaHOI-Gym containing 10 million frames and 180,000 hand capture trajectories.

Motion Diversity: Organized into a 3-level hierarchy:
1. Basic Kinematic Primitives: Straight lines, circular arcs, simple harmonic motion.
2. Physics-Constrained: Projectile motion, pendulum, inclined rolling, impact response.
3. Stochastic & Complex: Hybrid trajectories.
Structure: Covers 8 major motion types and 22 fine-grained subcategories across 11 object categories.
Data Generation: Automated pipeline generating diverse object scales, backgrounds, and textures.

C. Evaluation Metrics

Moving beyond simple frame-wise alignment, the benchmark uses rollout-based metrics:

Overall Success:
- Localization Success ( $S_{loc}$ ): Did the hand get within a threshold distance of the target?
- Grasping Success ( $S_{gra}$ ): Did the hand successfully close fingers on the target?
Trajectory Quality:
- Spatial Linearity ( $Q_{line}$ ): How straight is the path?
- Temporal Smoothness ( $Q_{smooth}$ ): Consistency of step lengths (coefficient of variation).
Completion Speed ( $R_{time}$ ): Normalized time to complete the task.

D. ObAct Baseline

A new "Observe-Before-Act" baseline designed to address the lack of temporal reasoning in standard models.

Architecture: Integrates a short-term observation window (historical frames) with the current frame using spatiotemporal attention.
Mechanism: A Vision-Language Model (VLM) encodes the visual history and task text. The resulting motion-aware representation is fused with proprioceptive states and fed into a diffusion policy (reusing GR00T-N1.5 weights) to predict continuous hand actions.

3. Key Contributions

DynaHOI-Gym Platform: The first unified, online, closed-loop simulation environment specifically for dynamic hand-object capture, featuring parameterized motion generators and rollout-based evaluation.
DynaHOI-10M Benchmark: A massive dataset (10M frames, 180K trajectories) covering diverse motion patterns (8 categories, 22 subcategories) and object types, enabling systematic evaluation of motion anticipation.
ObAct Baseline: A novel architecture demonstrating that integrating short-horizon temporal observation significantly improves interception accuracy compared to single-frame prediction models.

4. Experimental Results

The authors evaluated 12 models, including 6 specialized policy models (VLA, Diffusion) and 6 generalist Vision-Language Models (VLMs).

Performance of Existing Models:
- Low Success Rates: Even the best policy model (GR00T-N1.5) achieved only 27.9% localization success ( $S_{loc}$ ) and a mere 3.5% grasping success ( $S_{gra}$ ).
- VLM Limitations: Generalist VLMs (e.g., GPT-5.1, Gemini-3 Pro) performed poorly, with $S_{loc} \leq 7\%$ , indicating a fundamental inability to perceive and predict motion dynamics.
- Trade-offs: Diffusion policies showed higher success but unstable trajectories (low $Q_{smooth}$ ), while autoregressive models had cleaner paths but lower success rates.
ObAct Baseline Performance:
- Significant Improvement: ObAct achieved a 36.0% localization success rate, an 8.1% absolute improvement over the strongest diffusion baseline.
- Robustness: It consistently outperformed all prior models across all 7 metrics, including trajectory quality and completion speed.
- Ablation: Increasing the number of observation frames consistently improved success rates and reduced errors, confirming that temporal context is the key factor for dynamic capture.
Detailed Analysis:
- Grasp vs. Localization: Successful localization does not guarantee successful grasping. Models often fail to synchronize finger closure with the moving target at the exact moment of interception.
- Data Scaling: Performance scales monotonically with training data size, suggesting DynaHOI-10M provides high-quality supervision.
- Periodicity Sensitivity: Models perform better on periodic motions (e.g., circular) than linear ones, suggesting they rely on recognizable temporal patterns.

5. Significance

Shift in Focus: DynaHOI moves the field from static pose recovery to dynamic motion anticipation, a critical capability for real-world robotics (e.g., catching, dodging, interacting with moving objects).
Diagnostic Tool: The benchmark reveals that current state-of-the-art models (including VLMs) lack the temporal reasoning required for dynamic manipulation, highlighting a major blind spot in embodied AI.
Future Direction: The results suggest that future models must integrate spatiotemporal attention and explicit motion prediction modules rather than relying solely on single-frame visual inputs. The platform provides a reproducible testbed for developing "motion-aware hand intelligence."