🤖 AI

EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting

The paper proposes EMPIRE, a two-stage framework that introduces explicit manipulation planning as a learnable intermediate representation to decouple manipulation reasoning from motion synthesis, achieving state-of-the-art accuracy in egocentric bimanual hand-motion forecasting on the newly constructed EMPIRE-651K dataset.

Original authors: Wen Wang, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wen Wang, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to make a sandwich. You could simply show it a video of a human doing it and ask the robot to copy the hand movements frame by frame. But this approach often fails because the robot sees the motion without understanding the intent behind it. It might grab the bread correctly but then try to spread the jam with a closed fist, or it might forget to hold the plate steady while lifting the jar. To build truly intelligent machines that can interact with the world, we need them to understand not just how a hand moves, but why it moves that way and what step comes next. This is the core challenge of "egocentric" vision, where a computer sees the world from a first-person perspective, much like a human wearing a camera on their head. The goal is to predict what a person's hands will do next, a skill essential for robots that work alongside us, for virtual reality avatars that feel real, and for machines that can learn from watching human videos.

For a long time, researchers have tried to solve this by feeding video and text instructions directly into a computer model and asking it to guess the future hand positions. These models, often built on large language and vision systems, are powerful, but they tend to skip the middle step. They try to jump straight from "I see a cup and a table" to "the hand will move here," without explicitly figuring out the sequence of actions required to get there. It is like asking someone to write a story by guessing the last sentence of every paragraph without thinking about the plot in between. This shortcut works for simple tasks, but when the task is complex or lasts a long time, the errors pile up, and the robot's hands start to drift away from what they should be doing. Furthermore, trying to teach the model to understand the scene and generate the motion at the same time creates a conflict; the pressure to get the motion right can actually blur the model's understanding of the scene, making it less reliable over time.

A new approach, called EMPIRE, changes the game by forcing the computer to pause and make a plan before it moves. The researchers behind this method realized that before a human reaches for an object, their brain has already mapped out a sequence of steps: reach, grasp, lift, and place. Instead of letting the computer guess the final hand position directly, EMPIRE asks the computer to first write down this sequence of steps in a structured format. This is the "planning" stage. The computer looks at the video, the text description of the task, and even a rough estimate of the distance to objects, and it generates a clear, step-by-step list of what each hand should do. For example, it might output a plan that says, "Left hand: lift the cup; Right hand: steady the plate." This plan acts as a bridge, translating the messy visual world into a clean, logical instruction set.

Once this plan is created, the computer freezes that part of its brain and moves to the second stage: "acting." In this phase, a different part of the system takes the written plan and the current position of the hands to generate the actual smooth, fluid motion. Because the planning part is frozen, it cannot be confused or overwritten by the pressure of generating the motion. This separation ensures that the computer's understanding of the task remains stable and clear, even as it figures out the complex physics of moving fingers and wrists. The researchers tested this two-step method on a massive new dataset they built, containing over 650,000 examples of people performing 111 different tasks, from folding clothes to handling delicate objects. They trained the system to first predict the plan and then the motion, and the results were striking.

The new system outperformed previous methods by a significant margin. When measuring how far off the predicted hand positions were from the real ones, the new method reduced the error by nearly 20 percent compared to the best existing models. More importantly, it was much better at predicting the fine details of finger movements, which are crucial for delicate tasks. While other models might get the general location of the hand right, they often fail to predict how the fingers should curl or rotate. The new method, by relying on the explicit plan, kept the fingers in the right place for much longer periods, making the predictions accurate even for tasks that lasted five seconds or more. The system also proved to be incredibly efficient, generating these predictions in just one second, whereas other powerful models took minutes to do the same job.

The study suggests that the key to better robot hands is not just making the models bigger or faster, but giving them a better way to think. By forcing the system to articulate a plan before it acts, the researchers found a way to keep the computer's understanding of the world clear and its actions precise. This approach works particularly well for difficult tasks where the steps are complex and the timing is critical. It shows that when we teach machines to think in steps, rather than just guessing the outcome, they become much more capable of interacting with the physical world in a way that feels natural and reliable. The work opens a path toward robots that can learn from human videos with a level of understanding that goes beyond simple imitation, allowing them to handle the intricate, multi-step challenges of daily life.

Technical Summary: EMPIRE

Problem Statement

Forecasting dexterous hand motions from egocentric (first-person) observations is a critical capability for intelligent interactive systems, robot learning, and human-robot collaboration. While recent advances in Vision-Language Models (VLMs) and large-scale egocentric datasets have driven progress, existing methods face two fundamental limitations:

Implicit Supervision of Manipulation Planning: Current approaches typically map multimodal observations directly to future hand motions, implicitly inferring the underlying manipulation process. This often leads to locally plausible motions that deviate from the intended interaction progression, particularly in long-horizon tasks where early planning errors accumulate.
Coupled Optimization: Existing methods jointly optimize the VLM backbone and the motion generation model. This coupling causes motion-generation gradients to continually update the semantic representations used for conditioning. Consequently, optimizing for spatial motion objectives can overwrite the manipulation-aware structures acquired during VLM pretraining, weakening the model's ability to understand scenes and anticipate interactions.

Methodology: EMPIRE

The authors propose EMPIRE (Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting), a two-stage framework that decouples manipulation planning from motion synthesis.

Stage I: Learn to Plan

In this stage, a VLM (specifically PaliGemma-2) is trained to generate an explicit manipulation plan from multimodal context.

Inputs: The model receives an egocentric RGB observation, inferred monocular depth features (to address VLM limitations in metric distance and 3D reasoning), a coarse action caption, and camera field-of-view (FoV) data.
Output: Instead of predicting raw motion, the model outputs a temporally ordered sequence of hand-specific sub-actions (e.g., "reach," "grasp," "lift," "place").
Training: The model is fine-tuned using autoregressive next-token prediction on the plan tokens, while visual and depth encoders remain frozen. The resulting hidden states of the plan tokens ( $H_{plan}$ ) serve as an interpretable intermediate representation.

Stage II: Learn to Act

In this stage, a motion generator synthesizes future bimanual hand motions conditioned on the plan.

Architecture: A flow-matching Diffusion Transformer (DiT) acts as the motion generator.
Conditioning: The entire Stage-I VLM is frozen. The DiT conditions on the current hand state and the hidden states ( $H_{plan}$ ) extracted from the Stage-I prediction.
Training Strategy: Crucially, the DiT is trained on plans predicted by Stage I rather than ground-truth plans. This eliminates the train-deployment mismatch. By freezing the planner, the motion-generation gradients cannot alter the learned manipulation-aware representations, ensuring stable semantic conditioning.

Dataset: EMPIRE-651K

To support this method, the authors constructed EMPIRE-651K, a large-scale dataset derived from EgoDex.

Scale: 650,910 five-second forecasting windows across 111 manipulation tasks.
Annotations: The dataset includes explicit per-hand manipulation plans generated via a two-stage VLM-assisted pipeline (using Qwen2.5-VL for captioning and plan generation, and Qwen3 for consistency filtering).
Processing: Original ARKit skeleton annotations were converted into temporally consistent MANO-based hand representations, preserving both global motion and fine-grained finger articulation.

Key Contributions

Explicit Intermediate Representation: The introduction of explicit manipulation planning as a learnable intermediate representation, bridging the gap between perception and motion generation.
Decoupled Optimization: A two-stage training framework that freezes the planner during motion generation, preventing motion gradients from disrupting semantic representations and providing stable conditioning.
EMPIRE-651K Dataset: A new benchmark dataset with explicit per-hand manipulation plans, facilitating research in long-horizon bimanual forecasting.
State-of-the-Art Performance: The method achieves superior accuracy in both global hand placement and fine-grained finger articulation compared to existing baselines.

Experimental Results

Evaluated on the held-out test set of EMPIRE-651K, EMPIRE demonstrates significant improvements:

Accuracy: EMPIRE achieves a Mean Per-Joint Position Error (MPJPE) of 84.53 mm and a finger-relative error of 38.97 mm. This represents a 19.8% reduction in MPJPE compared to the re-implemented VITRA baseline.
Efficiency: EMPIRE reduces training time by 38.8% (71 hours vs. 116 hours for VITRA) and achieves inference speeds of 1.0 second per window. This is 83.5× faster than the Being-H0-14B model while achieving comparable overall MPJPE and significantly better finger articulation.
Generalization: The benefits of explicit planning are most pronounced in complex, long-horizon tasks. The method improves performance on 89 out of 111 tasks, with the largest gains observed in "hard" tasks (average MPJPE reduction of 49.7 mm). It also generalizes well to unseen tasks, improving 74 out of 85 unseen scenarios.

Significance and Claims

The paper claims that EMPIRE validates explicit manipulation planning combined with decoupled motion learning as an effective paradigm for accurate and efficient egocentric dexterous-motion forecasting. By explicitly modeling the intermediate manipulation process, the method overcomes the limitations of direct observation-to-motion mapping. The decoupled design ensures that the semantic understanding of the scene remains stable and is not corrupted by the optimization of low-level motion objectives. The authors posit that this approach is particularly valuable for long-horizon and compositional manipulation tasks where structured temporal guidance is essential.