🤖 AI

MoPO: Incorporating Motion Prior for Occluded Human Mesh Recovery

MoPO is a novel framework for occluded human mesh recovery that leverages motion priors from pose sequences to detect occlusions, predict missing joint positions, and refine the final pose, thereby achieving state-of-the-art accuracy and temporal consistency on both occlusion-specific and standard benchmarks.

Original authors: Tao Tang, Hong Liu, Xinshun Wang, Wanruo Zhang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tao Tang, Hong Liu, Xinshun Wang, Wanruo Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a dancer looks like while they are performing a complex routine. Now, imagine that every few seconds, a large tree or another dancer steps in front of them, hiding parts of their body. If you only look at the single frame where they are hidden, you might guess their arm is in the wrong place, or that their leg is floating in mid-air. This is the problem computer scientists face when trying to create 3D models of people from videos where the person is partially blocked.

The paper introduces a new system called MoPO (Motion Prior for Occluded Human Mesh Recovery). Think of MoPO as a "super-smart guesser" that doesn't just look at the current picture, but remembers the dancer's recent movements to fill in the blanks.

Here is how it works, broken down into simple steps:

1. The Problem: The "Blind Spot"

Current technology is good at figuring out a person's pose when they are fully visible. But when a person is blocked (occluded) by an object or another person, the computer gets confused. It tries to guess where the hidden parts are based only on the tiny bits of the body it can see. This often leads to two bad results:

Wrong Poses: The computer might guess the hidden arm is twisted in an impossible way.
Jittery Motion: In a video, the computer might guess the arm is here in one frame and there in the next, making the 3D model look like it's shaking or vibrating uncontrollably.

2. The Solution: MoPO's "Memory Lane"

MoPO solves this by realizing that movement has a pattern. If you see someone's arm moving up in the last three frames, you can be pretty sure where it will be in the next frame, even if a tree blocks your view. MoPO uses this "motion memory" to fix the blind spots.

It does this in two main stages:

Stage A: The "Fill-in-the-Blanks" Detective

First, MoPO acts like a detective checking a security camera.

Spotting the Hiding: It looks at the video and checks: "Is this part of the body visible right now?" It uses a special detector that looks at both the current image and the recent past to decide if a joint (like a knee or elbow) is hidden.
Predicting the Missing: If a joint is hidden, MoPO doesn't guess randomly. It uses a lightweight "motion predictor" (think of it as a short-term memory bank) to look at where that joint was a moment ago and where it is going. It then completes the missing part of the skeleton with a highly probable guess.
- Analogy: Imagine you are watching a magician pull a rabbit out of a hat, but a curtain blocks your view for a split second. A normal observer might think the rabbit vanished. MoPO, however, remembers the rabbit was moving upward just before the curtain dropped, so it "fills in" the rabbit's position behind the curtain so the trick looks continuous.

Stage B: The "Blending and Polishing" Chef

Once MoPO has a "completed" skeleton (one where the missing parts are filled in), it needs to turn that skeleton into a full 3D body (skin and all).

Mixing the Ingredients: It takes the "completed skeleton" (the motion memory) and mixes it with the actual visual details from the video (like the color of the shirt or the shape of the torso).
Refining the Pose: Sometimes, just guessing the position isn't enough; the joints need to rotate correctly. MoPO uses a technique called "Inverse Kinematics" (like a puppeteer adjusting strings) to make sure the 3D body moves naturally and doesn't look like a broken robot. It separates the "swing" of the limb (which is easy to see) from the "twist" (which is harder to see) to get the rotation just right.

3. The Results: A Smooth, Accurate Dance

The authors tested MoPO on many different videos where people were blocked by objects or other people.

Accuracy: MoPO was significantly more accurate than previous state-of-the-art methods. It reduced errors in predicting where joints are by about 8.5% to 12% compared to the best existing tools.
Smoothness: The biggest win was in video stability. Because MoPO uses the "motion memory," the 3D models don't jitter or shake when the person is hidden. The movement flows smoothly, just like a real human.

Summary

In short, MoPO is a system that recovers 3D human models from videos by saying, "I can't see your arm right now, but I know exactly where it is because I remember how you were moving a second ago." By combining this memory of movement with the visual clues it can see, it creates a much more accurate and stable 3D reconstruction, even in messy, crowded, or blocked scenes.

Technical Summary: MoPO – Incorporating Motion Prior for Occluded Human Mesh Recovery

1. Problem Statement

Recovering 3D human meshes from monocular video is a critical task for applications in animation, virtual reality, and robotics. While recent advancements have improved performance on standard benchmarks, existing methods struggle significantly with occlusions (self-occlusion, object-person, and person-person).
Current state-of-the-art (SOTA) occluded Human Mesh Recovery (HMR) methods face two primary challenges:

Inaccurate Poses under Heavy Occlusion: Relying solely on insufficient spatial features from visible body parts leads to inaccurate and unnatural pose estimations. Existing methods often depend on 2D evidence (masks, keypoints) or implicit priors from diffusion models, which fail under severe occlusion.
Temporal Inconsistency and Motion Jitter: When applied to video, frame-by-frame estimation without robust temporal modeling results in severe motion jitter. Existing video-based methods often lack the ability to model long-term dependencies or reason about occluded joints effectively, leading to unstable outputs.

The core hypothesis of this work is that pose sequences inherently contain reliable motion priors for estimating occluded body parts, which are more robust than relying solely on insufficient image features or general scene priors.

2. Methodology: MoPO Framework

The proposed method, MoPO (Motion Prior for Occluded human mesh recovery), is a model-based approach that integrates motion prediction into the HMR pipeline. It operates in two main stages:

A. Motion De-occlusion Module

This stage aims to reconstruct a complete 3D joint sequence by predicting occluded joints based on historical motion.

Spatial-Temporal Occlusion Detector: Unlike methods relying only on current-frame confidence, MoPO computes a weighted sum of confidence scores from the current and previous frames. It further utilizes an RNN to estimate confidence offsets based on historical trends, grouping joints into visible ( $V$ ) and occluded ($OccJ$) sets. This ensures temporally consistent occlusion detection.
Occluded Parts Complement: For detected occluded joints, a lightweight MLP-based motion predictor is employed.
- A lifting network first estimates 3D poses ( $P^{3D}$ ) from 2D detections.
- The predictor uses a Discrete Cosine Transform (DCT) to convert pose sequences into the frequency domain.
- An MLP-Mixer network processes the spatial-temporal information in the frequency domain to predict future poses.
- An Inverse DCT (IDCT) converts the predictions back to 3D positions.
- The system iteratively replaces occluded joints in the current frame with predicted joints from the history, generating a completed pose sequence ( $P^{3D}_{com}$ ).

B. Motion-Aware Fusion and Refinement Module

This stage integrates the completed motion prior with visual data to estimate the final SMPL parameters.

Occlusion-Alleviated Fusion: The completed pose sequence is encoded into a motion context ( $C_M$ ) using an RNN. This context is fused with image features (extracted via a ViT backbone) using Multi-Head Cross-Attention (MCA). The motion context acts as the query, while image features serve as keys and values, allowing the network to extract human-relevant features (shape and initial pose) guided by reliable motion priors.
Motion-Aware Refinement: To address the difficulty of regressing SMPL 6D rotation parameters directly from fused features, the method employs Inverse Kinematics (IK) principles:
- The SMPL pose is decomposed into swing (rotation between neighboring joints, derivable from joint positions) and twist (rotation along the bone axis, not derivable from joints).
- The swing is calculated directly from the completed joint sequence ( $P^{3D}_{com}$ ), while the twist is regressed from the fused features.
- These components are combined to refine the initial pose ( $\theta_{init}$ ) into a final refined pose ( $\theta_{refined}$ ), which is then fed to a pre-trained SMPL regressor to generate the final mesh.

3. Key Contributions

Explicit Motion De-occlusion: MoPO is the first to introduce motion complement-based de-occlusion for HMR. It explicitly completes occluded joints from past sequences, providing an occlusion-free motion prior that is more reliable than implicit priors from general diffusion models.
Motion-Aware Fusion and Refinement: The framework integrates completed poses with image features via cross-attention and refines SMPL parameters using a swing-twist decomposition strategy. This effectively leverages motion priors to overcome the ambiguity of regressing rotations under occlusion.
State-of-the-Art Performance: The method achieves superior results on both occlusion-specific and standard benchmarks, significantly reducing error metrics (MPJPE and MPVPE) compared to existing SOTA methods.

4. Experimental Results

The authors evaluated MoPO on several benchmarks, including 3DPW-OC, 3DPW-PC, 3DOH, 3DPW-Crowd, CMU-Panoptic, and the standard 3DPW.

Occlusion-Specific Benchmarks:
- On 3DPW-OC (object-person occlusion), MoPO reduced MPJPE by 8.5% (from 70.9mm to 64.9mm) and MPVPE by 12.0% (from 88.0mm to 77.4mm) compared to the previous SOTA (DPMesh).
- On 3DPW-PC (person-person occlusion), MoPO achieved 76.0mm MPJPE, outperforming previous methods.
- On CMU-Panoptic (multi-person scenes), MoPO showed superior robustness, achieving an average improvement of 2.0mm over DPMesh.
Standard Benchmarks:
- On the standard 3DPW dataset, MoPO achieved 60.6mm MPJPE, 37.1mm PA-MPJPE, and 71.2mm MPVPE. This represents a 10.2% reduction in MPJPE compared to MEGA (67.5mm), demonstrating that the motion prior benefits even in less occluded scenarios.
Efficiency: MoPO uses significantly fewer parameters (85.0M) and FLOPs (1.48G) compared to DPMesh (2607.4M parameters, 168.51G FLOPs), which relies on a heavy pre-trained diffusion model.
Qualitative Analysis: Visualizations confirm that MoPO produces more accurate and temporally consistent meshes under heavy occlusions, with significantly lower acceleration error (indicating less motion jitter) compared to competitors.

5. Significance and Claims

The paper claims that MoPO addresses the fundamental ill-posed nature of occluded HMR by shifting the reliance from insufficient spatial image features to reliable motion priors derived from temporal sequences.

Robustness: By explicitly completing occluded joints, the method mitigates the "unrealistic poses" and "severe motion jitter" that plague existing video-based HMR methods.
Generalization: The method demonstrates strong generalization on "in-the-wild" internet videos and diverse real-world scenarios, suggesting that the explicit motion prior is a more stable foundation for HMR than implicit scene priors.
Efficiency: The lightweight design (MLP-Mixer based predictor) offers a computationally efficient alternative to diffusion-based approaches while achieving higher accuracy.

The authors conclude that leveraging human motion prior is a critical direction for future research in occluded human mesh recovery, offering a more reliable solution for complex, real-world scenarios.