💻 computer science

Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction

This paper presents a real-time human-centric world model that synthesizes coherent upper-body interactions by combining a unified multi-scale implicit representation for continuous human motion with language-encoded discrete states for precise object contact control, achieving 25 FPS inference on H100 GPUs.

Original authors: Chaonan Ji, Jinwei Qi, Peng Zhang, Bang Zhang

Published 2026-07-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaonan Ji, Jinwei Qi, Peng Zhang, Bang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie where the main character can reach out, grab a coffee cup, take a sip, and set it down, all while talking to you. For a long time, computer scientists have been great at making the character's body move realistically, but they often struggled with the "grabbing" part. Sometimes the hand would float through the cup like a ghost, or the cup would magically change shape when touched. This is the challenge of "human-object interaction": teaching a computer to understand not just how a person moves, but how they physically touch and change the world around them.

To do this, researchers usually rely on two main tools. First, there's video generation, which is like a digital artist that can paint moving pictures based on a description. Second, there's world modeling, which is a fancy way of saying a computer tries to predict how a scene will change over time. The big question has been: Can we combine these to make a character that doesn't just dance in place, but actually interacts with objects in real-time, like a video game character that feels real enough to touch?

This is exactly what a team of researchers from Tongyi Lab has been working on. They built a new system that acts like a "real-time human-centric world model." Think of it as a digital puppet master that doesn't just control the puppet's limbs, but also decides if the puppet is holding a hammer or just waving its hand in the air. Their goal was to create a system that can take a live video of a person, a picture of an object (like a cup), and a simple command (like "grasp" or "no contact"), and instantly generate a video where the person interacts with that object perfectly.

The team found that by splitting the control into two distinct parts, they could solve the problem much better than previous methods. The first part is the continuous human state, which handles the smooth, flowing movements of the body, hands, and face. Imagine this as the puppeteer's hand guiding the puppet's muscles. The second part is the discrete interaction state, which is like a simple switch that tells the system, "Is the hand touching the object right now?" or "Is it holding it?" By using a specific, short command for this switch (like the words "grasp" or "no contact") instead of a long, complicated sentence, the computer can switch between these states instantly and accurately.

The result is a system that can generate these interactions at a speed of 25 frames per second with a delay of only about 1000 milliseconds. This means it's fast enough to feel like a live conversation or a real-time video game. The researchers showed that their method creates more realistic hand movements and better contact with objects compared to older systems, which often produced weird glitches like floating objects or hands that didn't quite close around what they were supposed to hold. They also created a special dataset of over 30,000 examples to teach their system how these interactions should look. While they haven't claimed this solves every problem in the universe, their experiments suggest that this "continuous-discrete" approach is a significant step forward in making digital humans that can truly interact with their world in real-time.

Technical Summary: Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction

Problem Statement

Recent advances in human video generation have improved the quality of upper-body synthesis for applications like telepresence and interactive agents. However, existing methods primarily focus on person-centric motion-driven generation (e.g., reenactment or avatar animation) without explicitly modeling human-object interaction or its effects on the surrounding local scene. While text-conditioned generation models can depict interactions, they often rely on free-form prompts that entangle motion and interaction semantics, leading to weak motion control and unstable contact outcomes. Furthermore, most current approaches are computationally expensive and limited to offline generation, lacking the unified framework necessary for real-time, interactive deployment where a person's motion and their specific contact state with objects must be controlled simultaneously and with low latency.

Methodology

The authors propose a real-time human-centric world model for upper-body interactive generation. The framework synthesizes coherent local world dynamics centered on a person, where coordinated body, hand, and facial motions evolve jointly with controllable human-object discrete interactions. The core of the method is a continuous-discrete joint control scheme comprising two complementary components:

1. Continuous Human-State Control

To model coordinated upper-body, hand, and facial dynamics, the authors introduce a unified implicit representation based on multi-scale motion encoding.

Multi-Scale Decomposition: The driving input is decomposed into four regions: the upper body, left hand, right hand, and face.
Implicit Encoding: Region-specific motion inputs are encoded into latent vectors using ViT-based encoders. These latents are fused into a shared latent space.
Implicit Retargeting: The fused motion latents are concatenated with linearly projected reference image patches and fed into a decoder. This design allows for expressive, fine-grained control of body-hand-face dynamics without explicit skeletal retargeting, preserving identity while capturing subtle motion details.

2. Discrete Interaction-State Control

To model the contact relation between the person and surrounding objects, the authors represent interaction using language-encoded discrete interaction-state commands rather than open-ended semantic prompts.

Discrete States: The interaction state ( $s_o$ ) is defined as a small set of discrete variables, specifically "no contact" and "grasp".
Explicit Control: Text serves as an explicit command to switch between these states, complementing the continuous human-state control. This separation allows the model to generate different local visual outcomes (e.g., grasping vs. non-contact) under similar human motion conditions.
Data Pipeline: A dedicated rendering pipeline was constructed to generate a high-quality dataset of over 30,000 samples. This pipeline uses Z-Image for reference generation, Qwen-VL for object detection, and Wan2.2-14B-I2V for video synthesis, filtered by Qwen-VL and manual inspection to ensure interaction plausibility.

3. Real-Time Optimization

To achieve streaming real-time inference, the model is optimized for efficiency:

Distillation: The model is distilled using CausVid, reducing sampling steps from 20 to 4 while introducing causal and sliding-window attention for autoregressive generation.
Self-Forcing: The Self-Forcing technique is employed to mitigate temporal error accumulation during iterative generation.
Performance: These optimizations enable streaming generation at 25 FPS on two H100 GPUs with an end-to-end latency of approximately 1000 ms.

4. Training Strategy

The model undergoes a two-stage training strategy:

Stage I (Human-State Pretraining): The model learns a unified implicit human representation from general upper-body motion videos (without object references) using Flow Matching, augmented with representation alignment loss (DINOv3 features) and keypoint supervision loss.
Stage II (Interaction-State Learning): The model is initialized from Stage I, the motion encoder is frozen, and the remaining modules are trained on the synthetic interaction dataset using only the Flow Matching objective to learn discrete interaction control.

Key Contributions

Human-Centric World Model: A framework that formulates person-centered scene evolution through the joint modeling of continuous human states and discrete interaction states, moving beyond simple motion reproduction to interactive local world modeling.
Unified Control Scheme: The introduction of a unified implicit multi-scale representation for human-state control combined with language-conditioned discrete interaction-state commands, enabling precise and controllable human-object interaction generation.
Real-Time System: The development of a streaming system capable of 25 FPS inference on H100 GPUs, supporting online interaction-state switching with low latency.

Results

The paper presents both qualitative and quantitative evaluations comparing their method against state-of-the-art baselines (Wan2.2, Humo, VACE, MAI, and Wan-Animate).

Qualitative: The method demonstrates superior identity preservation, more stable hand-object interactions, and accurate motion following compared to baselines. It successfully handles complex hand movements and allows explicit switching between "no contact" and "grasp" states under similar driving motions.
Quantitative: The method achieves favorable performance on objective metrics, including lower FVD (154.1) and tLPIPS (0.178), higher CLIP-I (0.851), and improved Object Quality (OQ: 0.151). It also shows lower HandSIM and FaceSIM errors, indicating better fine-grained local motion accuracy. User studies confirm improvements in text-following, action-following, motion consistency, and visual quality.

Significance

The authors claim this work represents a practical step beyond motion imitation toward real-time human-centric world modeling. By decoupling continuous human motion from discrete interaction states and optimizing for real-time performance, the framework addresses the limitations of existing methods that either lack explicit interaction control or are too slow for interactive applications. The system enables the synthesis of coherent, controllable, and interactive local world states centered on a person, facilitating applications in telepresence and interactive agents where a person not only moves but also interacts with and alters their immediate environment.