💻 computer science

LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model

LOME is an action-conditioned egocentric world model that generates realistic, physically consistent human-object manipulation videos by jointly estimating spatial actions and environmental contexts, thereby outperforming existing methods in motion control and generalization for applications like AR/VR and robotic training.

Original authors: Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue Wang

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to pour a glass of water, pick up a coffee mug, or stack cups.

In the past, teaching robots (or computers) to do this was like trying to build a car engine from scratch every time you wanted to drive. You had to manually program the physics of gravity, the friction of the table, and the exact shape of every cup. It was slow, rigid, and if you gave the robot a different cup, it would get confused.

LOME (Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model) is a new "magic recipe" that changes the game. Instead of building an engine from scratch, LOME is like a super-smart movie director who has watched millions of hours of human videos and learned how the world works.

Here is how LOME works, broken down into simple concepts:

1. The "First-Person" Perspective (Egocentric)

Most robots see the world like a security camera in the corner of the room. LOME sees the world like you do: through your own eyes. It's "egocentric," meaning it understands that when you reach out with your right hand, the cup moves because you moved it. This perspective is crucial for understanding how we interact with objects in our daily lives.

2. The Three Ingredients

To create a video of a human doing something (like pouring water), LOME needs three specific things, like a chef needing a recipe, ingredients, and a cooking style:

The Scene (The Photo): You give it a starting picture (e.g., a table with a green bottle and a mug).
The Script (The Text): You tell it what to do (e.g., "Pour the water from the bottle into the mug").
The Choreography (The Action Map): This is the secret sauce. Instead of just saying "pour," LOME looks at a map of exactly where your hands are moving at every single second. It's like giving the director a dance routine to follow.

3. The "Joint Dance" (The Secret Sauce)

Here is where LOME gets really clever.

In older methods, the computer would try to guess the video and the hand movements separately. It was like asking a painter to paint a dance while the dancer is standing still in a different room. The result was often messy: the hand might reach for the cup, but the cup wouldn't move, or the water would just appear out of thin air.

LOME does something different. It treats the hand movement and the video as a single, connected dance.

The Analogy: Imagine a puppet master (the hand) and a puppet (the object). Old methods tried to animate the puppet based on a description of the master. LOME learns the joint relationship. It understands that because the hand twists the bottle, the water must flow. It learns the physics of the interaction, not just the look of it.

4. What Makes It Special?

It Gets the Physics Right: If you ask LOME to pour water, it doesn't just draw a blue stream. It understands that the water level in the mug should rise, and the level in the bottle should fall. It simulates the "cause and effect" of the real world.
It's Flexible: If you show it a video of a human picking up a red apple, and then ask it to pick up a blue ball, it can do it. It hasn't memorized the specific apple; it has learned the concept of "picking up."
No 3D Modeling Needed: Usually, to make realistic robot movements, you need complex 3D blueprints of every object. LOME skips this. It learns directly from 2D videos, making it much faster and easier to use in the real world.

The Result

When you run LOME, it generates a video that looks like a real person doing the task.

The Hands: They move exactly where you told them to.
The Objects: They react realistically (liquid flows, cups stack, things fall if dropped).
The Consistency: The video doesn't glitch or flicker; it flows smoothly from start to finish.

Why Does This Matter?

Think of LOME as a universal simulator for reality.

For Robots: Instead of building a robot in a lab and spending months teaching it how to open a fridge, we can use LOME to generate thousands of "what-if" scenarios (videos) to train the robot. The robot learns by watching LOME's movies.
For Virtual Reality (VR/AR): Imagine putting on VR goggles and asking, "Show me how to fix this leaky faucet." LOME could instantly generate a photorealistic video of a human fixing it, right in front of your eyes, so you can learn by watching.

In short, LOME is a bridge between what we say ("pour the water"), what we do (the hand movements), and what happens (the water flows), creating a realistic movie of human interaction without needing a physical set or a 3D blueprint.

1. Problem Statement

Learning to generate realistic human-object manipulation videos is a critical challenge in computer vision and robotics. Existing approaches face several limitations:

Physics-based animation: Requires extensive manual modeling and does not generalize well to diverse object morphologies or real-world environments.
Reconstruction-based methods: (e.g., 3D/4D reconstruction) are limited in synthesizing novel interactions and often fail to generalize to unseen scenarios.
Text/Image-to-Video (I/T2V) models: While they capture rich motion priors, they struggle with precise motion control. Conditioning solely on text or images often results in poor motion quality and a lack of physical plausibility (e.g., liquids not flowing correctly).
Latent Action Spaces: Recent video world models often rely on discrete, fixed-size latent action spaces, which limits the ability to handle continuous, fine-grained human actions.
Optical Flow Limitations: Using optical flow as a control signal is problematic for manipulation because it prescribes object motion rather than letting it emerge naturally from human actions, and it suffers from occlusions.

The core problem is creating a system that can generate temporally consistent, photorealistic egocentric videos of human-object interactions that strictly adhere to specific human actions (poses and gestures) while simulating realistic physical consequences (e.g., liquid dynamics) without relying on explicit 3D/4D reconstruction or simulation.

2. Methodology: LOME

LOME is an action-conditioned egocentric world model built upon a pretrained video diffusion model (specifically Wan2.1-VACE-14B). It introduces a novel framework to learn the joint distribution of human actions and environmental contexts.

Key Components:

Inputs and Conditions:
- Reference Image ( $I$ ): Defines the scene and objects.
- Text Prompt ( $y$ ): Describes the intended manipulation.
- Per-frame Human Actions ( $A$ ): Continuous 3D human poses and hand gestures projected into 2D spatial maps.
- Camera Parameters: Extrinsics and intrinsics encoded as ray maps to maintain camera consistency.
Action Representation (Spatial Action Maps):
- Instead of using latent vectors, LOME represents actions as 2D rasterized spatial maps at full video resolution.
- 3D keypoints (skeleton and hands) are projected onto the image plane using camera parameters.
- Keypoints outside the camera frustum are masked out to prevent information leakage.
- This provides pixel-level guidance for where hands should appear, ensuring precise control.
Joint Action-Environment Modeling (Core Innovation):
- Naive conditioning fails: Simply concatenating action conditions with video latents often leads to poor interaction quality.
- Joint Denoising: Inspired by VideoJAM, LOME encodes the action maps into latent space ( $a$ ) and concatenates them with the video latents ( $x$ ) and image latents ( $x_I$ ) along the temporal dimension.
- The model learns to denoise both the video and the action maps simultaneously. This forces the model to learn the joint distribution $p(video, action | context)$, effectively decoupling action dynamics from environmental dynamics while enforcing consistency.
Modified Guidance Strategy:
- Standard Classifier-Free Guidance (CFG) assumes conditions are independent of the output. Since the action map is also being denoised by the model, it is not independent.
- LOME modifies the sampling distribution (inspired by Inner Guidance) to account for the dependency between the denoised action and the video.
- The inference guidance is adjusted to weigh the conditional probability of the action given the video, ensuring the generated video strictly follows the provided action trajectory.
Architecture Details:
- Base Model: Wan2.1-VACE-14B.
- Fine-tuning: Uses LoRA (Low-Rank Adaptation) on the VACE module while freezing the main Diffusion Transformer (DiT) blocks to preserve pre-trained motion priors.
- Camera Adapter: A lightweight module encodes camera extrinsics into features added to the video latents.

3. Key Contributions

LOME Framework: An egocentric world model capable of generating contact-rich, fine-grained human-object manipulation videos conditioned on images, text, and continuous per-frame human actions.
Joint Distribution Learning: A novel training objective that denoises action maps and video latents jointly, enabling the model to learn the causal relationship between human actions and physical object responses.
Action-Conditioned Guidance: A modified inference strategy that handles the non-independence of denoised action signals, significantly improving action adherence.
Realistic Physics without 3D: The system synthesizes realistic physical consequences (e.g., liquid pouring, stacking) purely from 2D video data, eliminating the need for explicit 3D/4D modeling or simulation.

4. Experimental Results

The authors evaluated LOME on the EgoDex dataset and in-the-wild lab captures, comparing it against state-of-the-art baselines: CoSHAND (image-based), Wan-I2V (text/image-based), and Go-with-the-Flow (GwtF) (optical flow-based).

Quantitative Performance:

Action Following (PCK@20): LOME achieved 66.85%, significantly outperforming the best baseline (CoSHAND at 51.33% and GwtF at 47.06%).
Motion Consistency (FVD): LOME achieved an FVD of 39.58, a substantial improvement over CoSHAND (59.83) and Wan-I2V (68.77), indicating smoother and more realistic motion.
Temporal Consistency (tLPIPS): LOME showed the lowest temporal jitter (0.051).

Qualitative & User Study:

Physical Realism: In "pouring" tasks, only LOME successfully generated coherent liquid dynamics where the liquid level rose progressively. Other methods failed to simulate the fluid or the interaction correctly.
User Preference: In a study with 30 participants, LOME received 97% preference for action adherence and 94% for visual quality.
Generalization: LOME successfully handled unseen objects and environments, including scenarios where objects were occluded in the input image (e.g., behind a fridge door), generating plausible interactions.

Ablation Studies:

Removing joint modeling degraded performance significantly.
Using temporal concatenation (vs. channel concatenation) proved superior, allowing bidirectional attention between video and action frames.
The camera adapter was crucial for maintaining spatial consistency.

5. Significance and Impact

Robotics Training: LOME provides a scalable pipeline for generating diverse, realistic training data for robots to learn manipulation skills without the cost and limitations of physical simulation or 3D modeling.
AR/VR: The ability to generate photorealistic, action-consistent videos from simple inputs paves the way for immersive Augmented Reality and Virtual Reality experiences.
Paradigm Shift: The paper demonstrates that joint modeling of actions and environment within a video diffusion framework is a more effective approach for manipulation than treating actions as simple external controls. It bridges the gap between generative video models and physical world simulation.

Limitations: The method relies on the accuracy of 3D pose estimation and camera calibration; errors in these inputs can lead to projection misalignments. Additionally, coordinating complex multi-object interactions (e.g., scooping ice into a cup while holding the cup) remains challenging. Future work aims to improve efficiency via distillation.