💻 computer science

Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach

This paper introduces a novel dataset of 23.7K gaze-primed motion sequences and a text-conditioned diffusion model that generates realistic full-body motions, successfully capturing the natural human behavior of spotting an object before reaching it.

Original authors: Masashi Hatano, Saptarshi Sinha, Jacob Chalk, Wei-Hong Li, Hideo Saito, Dima Damen

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Masashi Hatano, Saptarshi Sinha, Jacob Chalk, Wei-Hong Li, Hideo Saito, Dima Damen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a kitchen to grab a jar of peanut butter from a high shelf. Before you even lift your hand, your eyes have already locked onto that jar. You might even tilt your head slightly or shift your weight to get a better angle. This split-second moment of "looking before reaching" is called priming. It's your brain's way of saying, "I see the target, and I'm getting ready to grab it."

For a long time, computers trying to animate human movement (like in video games or robots) were terrible at this. They could make a character walk or run, but when it came to grabbing something, the character would often just teleport their hand to the object or reach blindly without looking first. It looked robotic and unnatural.

This paper, "Prime and Reach," is like a new recipe for teaching computers how to be more human. Here is the breakdown:

1. The Problem: The "Blind Grab"

Imagine a robot trying to pick up a cup. If you just tell it, "Go get the cup," the robot might spin around and slam its hand into the cup without ever looking at it. It lacks the anticipation that humans have. Humans don't just move; we plan our movement by looking at where we are going first.

2. The Solution: A Massive Library of "Looking"

The researchers realized they needed to teach the computer what "looking before grabbing" actually looks like. So, they went on a scavenger hunt through five different video datasets (collections of real people doing daily tasks).

The Hunt: They looked for moments where a person's eyes landed on an object before their hand touched it.
The Result: They curated a massive library of 23,700 of these specific "look-and-grab" sequences. Think of this as a giant cookbook of "how to look before you reach."

3. The Teacher: The "Diffusion" Model

To teach the computer, they used a type of AI called a Diffusion Model.

The Analogy: Imagine a sculpture made of clay that is currently just a messy pile of noise (like static on an old TV). The AI acts like a sculptor who slowly chips away the noise, step-by-step, until a clear, smooth statue emerges.
The Twist: In this case, the AI isn't just chipping away noise; it's being guided by a "goal." The researchers told the AI: "Start with this messy noise, but make sure the final statue looks like a person who looked at a cup before grabbing it."

4. The Two Ways to Give Instructions

The researchers tested two ways to tell the AI what to do:

The "Pose" Instruction: "Stand here, and end up in this exact pose holding the cup." (Like giving a dancer a specific final pose).
The "Location" Instruction: "The cup is over there. Go get it." (This is harder because the AI has to figure out how to stand and move to get there).

5. The Magic Metric: "Prime Success"

How do you know if the robot is doing it right? They invented a new test called Prime Success.

The Test: Did the character's eyes (or head direction) actually lock onto the object before the hand touched it?
The Result: Their new AI scored incredibly high on this test. It learned to "look" at the target, pause for a split second to orient itself, and then reach. It stopped doing the "blind grab."

6. Why This Matters

This isn't just about making better video games.

For Robots: If a robot is helping you in a kitchen or a hospital, it needs to look at what it's touching to be safe and efficient.
For Virtual Humans: If you are in a virtual reality meeting, you want the avatar to look at you when you speak, not stare at the ceiling while reaching for a coffee cup.

The Bottom Line

The researchers took a complex human behavior (the subtle art of looking before reaching), gathered thousands of real-life examples, and taught an AI to mimic it. The result is a digital human that doesn't just move; it anticipates. It looks at the destination before it moves, making the movement feel natural, fluid, and truly human.

In short: They taught the computer that you have to look at the cookie before you grab it.

1. Problem Statement

Human motion generation has advanced significantly in translating text to motion and navigating environments. However, existing methods often rely on synthetic datasets or controlled lab recordings, failing to capture natural, anticipatory behaviors found in real-world interactions. Specifically, the "Prime and Reach" (P&R) behavior—where a human visually spots (primes) an object via gaze before physically reaching for it—is largely unexplored in generative models.

Current challenges include:

Lack of Data: Most egocentric (first-person) datasets capture gaze but lack full-body 3D motion, while full-body datasets often lack gaze data.
Missing Anticipation: Generative models typically focus on the final goal (reaching) but ignore the preparatory phase (priming/looking), resulting in unnatural, robotic movements.
Control Limitations: Existing controllable motion generation methods struggle to synthesize the specific sequence of "looking first, then moving" conditioned on object locations.

2. Methodology

The authors propose a comprehensive framework involving data curation, a diffusion-based generation model, and novel evaluation metrics.

A. Data Curation: The Prime & Reach Dataset

The authors curated 23,728 P&R sequences from five publicly available datasets (HD-EPIC, MoGaze, HOT3D, ADT, and GIMO).

Extraction Logic: They identified "pick-up" and "put-down" events ( $t_e$ ) and analyzed a temporal window preceding the event to find the prime time ( $t_p$ ).
Gaze Priming Detection: Using eye-tracking data, they projected the user's gaze ray into the 3D environment. A sequence is labeled as "primed" if the gaze ray intersects the 3D bounding box of the target object (or the future location for put-downs) for a minimum duration before the reach occurs.
Full-Body Estimation: For datasets lacking full-body motion, they utilized EgoAllo to estimate full-body poses (SMPL-H/SMPL-X) conditioned on the egocentric camera pose and gaze.
Sequence Definition: Each sequence spans from $t_p - 2s$ to $t_e$ , capturing the anticipatory phase, the prime, and the reach.

B. Model Architecture: Goal-Conditioned Diffusion

The core model is a Transformer-based Diffusion Model adapted for motion generation.

Pre-training: The model is first pre-trained on the Nymeria dataset (large-scale egocentric motion with text) to learn a prior of natural human movement.
Conditioning Mechanism:
- Text: Describes the action type (e.g., "pick up" vs. "put down").
- Initial State: The starting pose and joint velocities (crucial as P&R sequences do not start from a neutral pose).
- Goal Condition: The model is trained under two conditions:
  1. Goal Pose: The final full-body pose at the end of the motion.
  2. Object Location: Only the 3D coordinates of the target object ( $o_{3D}$ ).
Condition Injection: Conditions are flattened, projected to a latent space, and added to the text embedding ( $\tilde{z}_t = z_t + p$ ) via residual addition before being injected into the Transformer decoder. This preserves the pre-trained text-motion prior without disrupting it with new attention pathways.
Inference Optimization: During inference, the model performs diffusion latent noise optimization. It treats the initial noise as a latent variable and optimizes it over $M$ iterations using a loss function that enforces the initial state, the goal (pose or location), and motion smoothness (jerk).

C. Evaluation Metrics

The authors introduce a new metric to evaluate the specific behavior of priming:

Prime Success: Measures if the generated motion exhibits intentional gaze priming. It checks if the head forward vector intersects the target location within a time window around the ground truth prime time, and if the head's angular velocity decelerates (indicating a deliberate stop to look).
Reach Success: Percentage of sequences where the wrist reaches within 10cm of the goal.
Standard Metrics: Goal MPJPE, Location Error, Foot Skating, and Mean Per Joint Position Error (MPJPE).

3. Key Contributions

First P&R Dataset: Creation of the first large-scale dataset (23.7K sequences) explicitly capturing the "gaze-priming then reaching" behavior from diverse real-world egocentric datasets.
Novel Generation Framework: A diffusion-based model capable of synthesizing full-body motion that naturally incorporates anticipatory gaze behavior, conditioned on either a final pose or a target object location.
New Metric: Introduction of the "Prime Success" metric to quantitatively evaluate the anticipatory nature of generated motion, a capability previously unmeasured in motion generation.
State-of-the-Art Performance: The model significantly outperforms baselines (including MDM, GMD, DNO, DartControl, and WANDR) in both priming and reaching capabilities.

4. Experimental Results

The model was evaluated on five datasets (HD-EPIC, MoGaze, HOT3D, ADT, GIMO).

Prime Success: The proposed model achieved the highest Prime Success across all datasets.
- Example: On the GIMO dataset, it achieved 50.0% Prime Success (Goal Pose condition), outperforming the best baseline (DNO) by 18.2%.
- On MoGaze, it improved Prime Success by 19.5% over baselines when conditioned on object location.
Reach Success: The model achieved near-perfect Reach Success (often 100% on several datasets) by leveraging latent noise optimization to ensure the hand reaches the target.
Comparison to Baselines:
- Text-only baselines (e.g., MDM) failed to prime or reach effectively due to lack of spatial guidance.
- Optimization-based baselines (DNO, DartControl) achieved high Reach Success but significantly lower Prime Success, as they focused on the endpoint rather than the trajectory's anticipatory nature.
- The proposed P&R model learned the strategy of priming from the data, not just the endpoint.
Ablation Studies:
- Pre-training: Pre-training on Nymeria (rich in natural interactions) yielded better results than HumanML3D.
- Conditions: Removing the initial state or velocity reduced Prime Success significantly.
- Architecture: A Decoder-based Transformer outperformed an Encoder-based approach.

5. Significance and Impact

Bridging the Gap: This work bridges the gap between computer vision (gaze tracking) and motion synthesis, enabling the creation of digital humans that exhibit human-like anticipatory behavior.
Robotics and VR: The ability to generate "look-then-reach" motions is critical for robotics (where robots must identify targets before grasping) and immersive VR/AR avatars that need to behave naturally.
Methodological Shift: It demonstrates that simply optimizing for the final goal is insufficient for natural motion; generative models must be trained on the process of interaction, including the preparatory phases.
Future Directions: The authors note that while body motion is synthesized, hand pose (grasping) is not yet modeled, pointing to future work in full hand-object interaction synthesis.

In summary, "Prime and Reach" establishes a new benchmark for natural human motion generation by explicitly modeling the cognitive-motor link between gaze and action, proving that anticipatory behavior can be learned and synthesized through data-driven diffusion models.