🤖 AI

Robust Motion Generation using Part-level Reliable Data from Videos

This paper addresses the challenge of occlusions and off-screen captures in video-based motion data by proposing a robust part-aware masked autoregression model that leverages only credible part-level information to generate high-quality human motion, accompanied by the introduction of a new large-scale benchmark called K700-M.

Original authors: Boyuan Li, Sipeng Zheng, Bin Cao, Ruihua Song, Zongqing Lu

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Boyuan Li, Sipeng Zheng, Bin Cao, Ruihua Song, Zongqing Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a computer to move a digital human just by reading a sentence. You might type, "The person is dancing," and expect the screen to show a fluid, realistic performance. This field, known as text-to-motion generation, has made remarkable progress, but it has hit a wall. The best results so far come from training computers on special, high-quality recordings of real people moving, captured with expensive sensors in controlled studios. While these recordings are perfect, they are rare, limited in variety, and often miss the messy, complex ways people actually move in the real world. To break through this barrier, researchers have turned to the internet, hoping to learn from the billions of ordinary videos available online. The idea is simple: if a computer can learn from the vast library of the web, it could understand a much wider range of actions. But there is a catch. When a computer tries to figure out how a person is moving in a regular video, it often cannot see the whole body. A person might be partially hidden by a tree, cut off by the edge of the camera frame, or blocked by another person. In these moments, the computer has to guess where the missing limbs are.

This guessing creates a fundamental problem for learning. If a computer is shown a video where only the upper body is visible, and it is forced to learn from a full-body guess that includes invisible legs, it might learn the wrong things. It could start believing that the invisible legs are real evidence, rather than just a guess. For a long time, researchers faced a difficult choice: either throw away every video clip that wasn't perfectly clear, losing most of the data, or keep all the clips and hope the computer could ignore the bad guesses. Both approaches had serious flaws. Throwing away clips meant losing valuable information about specific movements, like how a person moves their arms while sitting. Keeping everything meant teaching the computer with false information. A team of researchers has now found a way to solve this by changing how the computer looks at the data. Instead of treating the whole video clip as either good or bad, they taught the computer to look at the body part by part.

The researchers developed a new system that acts like a careful editor. When the system analyzes a video, it first checks which parts of the body are actually visible and which parts are hidden. If a person's legs are blocked by a table, the system knows to trust the movement of the arms and torso but to ignore the computer's guess about the legs. It treats the visible parts as solid facts and the hidden parts as blanks to be filled in later. This approach allows the system to learn from thousands of videos that were previously considered too messy to use. By focusing only on the parts of the body that the camera actually sees, the system builds a reliable understanding of movement without being confused by its own guesses. It is like learning to recognize a bird by studying only the wings when the tail is hidden, rather than trying to memorize a drawing of the whole bird that includes a made-up tail.

To test this idea, the team created a massive new collection of data from the internet, containing nearly two hundred thousand pairs of video clips and text descriptions. They analyzed these clips and found that in more than three-quarters of the frames, at least one part of the body was not fully visible. This confirmed that the problem of missing information is not rare; it is the norm for web videos. Using this dataset, they trained their new system and compared it against older methods that either threw away the messy clips or tried to learn from all of them without filtering. The results were clear. The new system, which pays attention to which parts are visible, produced much more accurate and realistic movements than the older methods. It was able to generate motions that were not only more faithful to the text description but also smoother and more natural.

The study showed that keeping the messy clips and teaching the computer to ignore the invisible parts was far better than throwing the clips away. When the researchers tested their system on a large pool of two hundred thousand clips, it significantly outperformed the best existing methods. The older methods that kept all the data made many mistakes because they tried to learn from the invisible parts, while the methods that threw away the bad data lost too much variety. The new system managed to keep the variety of the large dataset while avoiding the mistakes. It learned that a person sitting and playing drums might have their legs hidden, but their arm movements are still real and useful to learn from. By focusing on the reliable parts, the system could piece together a complete picture of the movement without being misled by the missing pieces.

This work suggests a new way to teach computers about the physical world using imperfect data. It shows that we do not need perfect, studio-quality recordings to teach a machine how to move. Instead, we can use the vast, messy collection of videos from the internet if we teach the machine to be smart about what it trusts. The system does not need to see the whole body to learn how a person moves; it just needs to see enough of the body to understand the action. This approach opens the door to training much larger and more capable systems in the future, capable of understanding a wider range of human behaviors than ever before. The researchers found that by respecting the limits of what the camera can see, the computer can learn more, not less. This shift in strategy, from demanding perfect data to learning from partial evidence, marks a significant step forward in teaching machines to understand human motion.

Technical Summary: Learning Text-to-Motion from Partially Observed Web Motions via Part-Aware Masked Modeling

Problem Definition

Text-to-motion (T2M) generation models require large-scale motion-text corpora, yet high-quality motion capture (MoCap) data remains limited in scale and diversity. While internet videos offer a promising alternative with open-vocabulary actions and diverse viewpoints, recovering 3D motion from these videos using Human Mesh Recovery (HMR) pipelines introduces a critical challenge: partial observability.

In web videos, bodies are frequently cropped, self-occluded, or blocked by objects. HMR systems typically fill in missing joints using learned priors to output a complete motion sequence. However, this creates a mismatch between the supervision signal and the visual evidence: some joints are directly supported by image data, while others are hallucinations. Existing pipelines handle this via two suboptimal clip-level policies:

Keep-All: Treats the entire recovered motion (including hallucinated parts) as ground truth, introducing label noise.
Discard: Removes any clip containing non-visible body parts, drastically reducing the available training data (often discarding >70% of clips) and losing valuable local motion dynamics.

The paper posits that web-recovered motion should be treated as a part-level credible supervision problem, where only observed body parts provide reliable learning targets, while unobserved parts should be inferred rather than directly supervised.

Methodology: RoPAR

The authors propose RoPAR (Part-Aware Text-to-Motion Framework), a system designed to learn from partially observed web motions by restricting supervision to image-supported body parts. The framework operates in three stages:

1. Visibility-Aware Part Decomposition

The human skeleton is decomposed into five kinematic parts: Torso, Left/Right Arms, and Left/Right Legs. For each frame, a 2D pose estimator (ViTPose) provides confidence scores for joints. An observability mask ( $O^p_i$ ) is generated for each part $p$ at frame $i$ :

If the average confidence of joints in a part exceeds a threshold $\tau$ , the part is deemed credible.
If not, the part is non-credible.
Crucially, non-credible parts are retained in the sequence for context but are excluded from direct supervision.

2. Part-Aware Latent Modeling (P-VAE)

A part-aware Variational Autoencoder encodes motion into a latent space.

Input: Local joint positions, rotations, velocities, and global root velocities.
Constraint: The reconstruction loss and KL divergence are applied only to credible parts. Non-credible parts contribute zero gradient.
Outcome: The latent space is shaped solely by observed dynamics, preventing HMR completions from distorting the representation. The output is a latent grid $Z$ with one token per part per frame.

3. Visibility-Guided Masked Generation

A text-conditioned transformer generates the full motion sequence using a masked modeling approach:

Input Masking: Non-credible tokens are always masked in the input (replaced with a learnable mask token $[M]$ ). Additionally, a subset of credible tokens is randomly masked to ensure the model learns to predict from context.
Supervision: The model is trained to predict the masked credible tokens using the text prompt and the remaining visible context.
Exclusion: Non-credible tokens are never used as prediction targets. The model learns to infer these missing parts from the text and the credible parts, rather than imitating potentially noisy HMR outputs.
Refinement: A lightweight latent diffusion head refines the coarse sequence generated by the transformer to improve temporal smoothness and cross-part coordination.

Key Contributions

Problem Formulation: The paper formally defines web-mined T2M under partial observability as a part-level credible supervision problem, demonstrating that standard clip-level policies (Keep-All and Discard) are mismatched to the heterogeneous reliability of web motion data.
RoPAR Framework: A novel architecture that integrates visibility estimation, part-aware latent modeling, and masked generation. It selectively trusts image-supported parts while inferring unsupported parts from text and context.
K700-M Dataset & Protocol: The authors introduce K700-M, a dataset of 198k motion-text pairs mined from Kinetics-700, specifically designed to evaluate T2M under partial observability. They establish matched evaluation protocols for both "Keep-All" and "Discard" baselines.

Experimental Results

Experiments were conducted on K700-M and HumanML3D (as a fully credible control).

Performance on K700-M: RoPAR significantly outperforms strong baselines (MDM, MotionGPT, MoMask, MotionStreamer) under both standard web-data policies.
- In the Keep-All setting (scale-preserving), RoPAR reduced the best baseline FID by 19.1% (from 24.33 to 19.68) and improved R@1 by 20.3% (from 0.591 to 0.711).
- In the Discard setting (cleaner but smaller data), RoPAR still achieved the best results (FID 21.38), demonstrating that even with cleaner targets, the part-aware approach is superior.
Scaling Behavior: As the training pool size increased (25k to 200k clips), RoPAR consistently improved, whereas baselines either suffered from noise (Keep-All) or stagnated due to data scarcity (Discard).
Ablations: Removing the part-wise decomposition or the credible-only supervision rule caused significant performance drops, confirming that the architecture's ability to separate credible from non-credible supervision is the primary driver of success.
Qualitative Analysis: RoPAR produces more stable and physically plausible motions, particularly for asymmetric actions or occluded scenarios, avoiding the artifacts (e.g., drifting limbs, implausible poses) seen in baselines that treat hallucinated joints as ground truth.

Significance and Claims

The paper argues that the bottleneck in web-scale T2M is not merely the quality of motion recovery, but the mismatch between visual evidence and the supervision signal. By treating partial observability as "structured missing supervision" rather than a binary data-cleaning problem, RoPAR enables models to utilize difficult, partially visible clips without learning model-based completions as ground truth.

The authors claim that selective part-level supervision is more effective than clip-level filtering. This approach allows the model to aggregate complementary partial observations (e.g., legs visible in one clip, arms in another) into a shared text-conditioned motion prior. The framework is presented as a solution specifically for the supervision problem within web-scale T2M, showing that keeping partially useful clips and restricting supervision to image-supported parts yields superior results compared to discarding data or trusting all recovered joints.