💻 computer science

ViPS: Video-informed Pose Spaces for Auto-Rigged Meshes

ViPS is a feed-forward framework that learns a valid, controllable pose space for auto-rigged 3D meshes by distilling motion priors from pretrained video diffusion models, enabling zero-shot generalization to diverse assets without requiring scarce artist-authored 4D training data.

Original authors: Honglin Chen, Karran Pandey, Rundi Wu, Matheus Gadelha, Yannick Hold-Geoffroy, Ayush Tewari, Niloy J. Mitra, Changxi Zheng, Paul Guerrero

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Honglin Chen, Karran Pandey, Rundi Wu, Matheus Gadelha, Yannick Hold-Geoffroy, Ayush Tewari, Niloy J. Mitra, Changxi Zheng, Paul Guerrero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital puppet (a 3D character) that you want to animate. In the old days, animators had to build a "skeleton" inside the puppet and then spend hours manually figuring out which joints could bend, how far they could twist, and what combinations of movements looked natural. If they bent a knee too far backward, the leg would snap like a broken twig. If they twisted a neck too much, the head would clip through the body. It was a tedious, manual process of trial and error.

ViPS (Video-informed Pose Spaces) is like giving that digital puppet a "muscle memory" and a "common sense" brain, so it knows how to move naturally without needing a human to micromanage every joint.

Here is how it works, broken down into simple analogies:

1. The Problem: The "Clunky" Puppet

Think of a standard 3D rig like a marionette with loose strings. You can pull the strings to make it move, but if you pull too hard, the puppet falls apart or looks like a contortionist in a nightmare.

The Issue: Computers don't inherently know that a human knee shouldn't bend backward, or that a dog's tail shouldn't snap off. Without a guide, random movements look broken and unnatural.

2. The Solution: Learning from a Movie

Instead of teaching the computer rules like "knees bend forward only," ViPS teaches the computer by showing it movies.

The Analogy: Imagine you want to teach a robot how to dance. Instead of giving it a manual on anatomy, you show it 10,000 hours of videos of people dancing, running, and jumping. The robot watches and learns, "Oh, when a person lifts their leg, their hip moves this way, and their arm swings that way."
The Magic: ViPS uses a powerful AI video generator (like the ones that make AI movies) as a teacher. It watches videos of animals and objects moving and asks, "What does a realistic pose look like?"

3. The "Pose Space": A Safe Playground

Once the AI has watched all those videos, it builds a "Pose Space."

The Analogy: Imagine a giant, invisible playground for your puppet.
- The Ground: This is the "safe zone." If your puppet stands anywhere on this ground, it looks natural and anatomically correct.
- The Cliff: If you try to move the puppet into a position that is impossible (like a head spinning 360 degrees), it falls off the cliff into the "nonsense zone."
How ViPS Helps: ViPS is the map of this playground. It tells you exactly where the safe ground is for any character you give it, whether it's a human, a dragon, or a weird alien you've never seen before.

4. How It Works (The "Distillation" Process)

The paper calls this "distilling video priors."

The Analogy: Think of a master chef (the video AI) who makes a perfect soup. You can't just give the soup to a student; you have to teach them the recipe. ViPS takes the "flavor" of the video AI's knowledge and distills it into a simple, compact recipe (a mathematical model) that fits inside a standard 3D rig.
The Result: You don't need a massive database of 3D animations (which are hard to make). You just need the video AI's "common sense" about how things move.

5. Why It's a Big Deal

Zero-Shot Learning (The "Chameleon" Effect): Usually, if you train a robot to walk, it can only walk. If you give it a new animal (like a giraffe), it fails. ViPS is like a chameleon. You can feed it a 3D model of a creature it has never seen in its training data (like a fantasy dragon), and it will instantly figure out how that dragon should move based on the video patterns it learned.
The "Edit" Button: Imagine you want to move a character's hand to wave. In old systems, moving the hand might make the elbow twist weirdly. With ViPS, you just move the hand, and the AI automatically adjusts the shoulder and torso to make the whole movement look smooth and natural, like a human doing it. It's like having a smart assistant that fixes your mistakes instantly.
Closing the Loop: It also works backward. You can tell the AI, "Make a video of a bear running," and ViPS will generate the perfect 3D skeleton movements first, then use those movements to guide the video generator. It ensures the video looks physically real.

Summary

ViPS is a tool that takes the "common sense" of how things move from watching millions of videos and packs it into a simple guide for 3D characters. It allows animators and creators to make characters move naturally, fix broken poses automatically, and animate creatures they've never seen before, all without needing to manually draw every single frame or write complex rules. It turns a rigid, clunky puppet into a living, breathing digital actor.

1. Problem Statement

Kinematic rigs (skeletons with skinning weights) provide a low-dimensional interface for animating 3D meshes but lack an inherent representation of the plausible manifold of joint configurations.

The Challenge: Naively sampling or manipulating raw rig parameters often results in semantic and geometric violations, such as anatomical hyperextension, unnatural bone twisting, or non-physical self-intersections.
The Gap: Existing methods to define a "pose space" (the distribution of valid poses for a specific asset) rely on scarce, expensive, artist-authored 4D datasets (rigged mesh animations). These datasets are typically category-specific (e.g., only humans) and do not scale to the long tail of diverse shapes and species.
The Goal: Can we automatically discover a plausible, controllable pose space for any auto-rigged 3D mesh without requiring curated 3D/4D motion data?

2. Methodology: Video-informed Pose Spaces (ViPS)

ViPS is a feed-forward framework that distills motion priors from a pretrained video diffusion model into a universal, mesh-conditioned pose space. It operates in three main stages:

A. Data Generation Pipeline (Video-to-Pose)

Instead of using artist-created 3D animations, ViPS synthesizes its own training data using generative video models:

Video Generation: Uses a VLM (InternVL) to generate prompts for an image generator (Flux 2) to create clean, full-body object images. These are then expanded into diverse motion videos using a video generator (Wan 2.2) accelerated by TurboDiffusion.
4D Reconstruction:
- Selects a "rest pose" frame ( $\alpha$ ) and reconstructs a textured mesh ( $M_\alpha$ ) using SAM3D.
- Uses ActionMesh (a feed-forward 4D reconstruction model) to generate a sequence of meshes with shared topology for all video frames.
- Extracts a skeleton ( $S$ ) from the rest pose mesh using RigAnything.
Pose Optimization: For each frame in the video, the system optimizes the skeleton parameters ( $P'$ $P^{'}$ ) to match the reconstructed 3D mesh of that frame. This involves minimizing a vertex distance loss ( $L_{recon}$ $L_{r eco n}$ ) and a skeleton edge length regularization ( $L_{edge}$ $L_{e d g e}$ ) to ensure the skeleton deforms the mesh plausibly while maintaining bone lengths.
- Result: A dataset of tuples $(M, S, P')$ representing valid poses for specific meshes and rigs, derived purely from 2D video priors.

B. Learning the Pose Space (The Model)

ViPS trains a diffusion model (specifically a Diffusion Transformer, or DiT) to learn the conditional distribution $p(P' | M, S)$ .

Input: A noisy version of the skeleton node positions ( $P^t$ ), the rest pose ( $P$ ), skeleton edges ( $E$ ), and semantic node features ( $F$ ).
Semantic Features: To handle unseen species without manual labels, the model aggregates per-vertex semantic features (derived from DINO and diffusion features of the mesh texture) into per-node features using skinning weights. This allows the model to understand what part of the body a joint controls (e.g., "paw" vs. "wing") without explicit text labels.
Architecture: The DiT processes tokens representing skeleton nodes. It uses a Skeletal Attention mechanism to ensure the attention between nodes respects the topological connectivity of the skeleton.
Training: Standard diffusion training (MSE loss) to predict the denoised pose $P'$ from noise.

C. Inference and Control

Once trained, the model supports several operations:

Sampling: Generating diverse, plausible poses for a given mesh/rig.
Manifold Projection (Inverse Kinematics): Using guided sampling, user-specified sparse constraints (e.g., moving a paw) are projected back onto the learned manifold. The model automatically adjusts other joints to maintain anatomical validity.
Pose Space Walk: Traversing the latent space to generate smooth, temporally coherent trajectories for animation.
Video Guidance: The generated 3D pose proxies can serve as structural guidance for video diffusion models, closing the loop between 2D generation and 3D control.

3. Key Contributions

Universal Pose Space Discovery: Formulates pose space discovery as learning a universal, mesh-conditioned generative distribution over rig parameters, capturing both validity and plausibility without manual 4D data.
Video-to-Pose Distillation: Introduces a novel pipeline to transfer motion priors from video diffusion models into rig-parameter space, bypassing the need for curated 3D/4D datasets.
High-Quality 4D Dataset: Constructs a dataset of 127k poses spanning 100+ species and 200+ individuals, generated entirely via the proposed video-to-pose pipeline.
Zero-Shot Generalization: Demonstrates that the model generalizes robustly to unseen species and skeletal topologies (e.g., fantasy creatures, insects) without fine-tuning.
Closed-Loop Control: Shows how the discovered 3D pose space can act as a precise, differentiable proxy to steer video generation, enabling semantically aligned 3D-to-2D synthesis.

4. Results and Evaluation

The authors evaluated ViPS against state-of-the-art baselines, including AnyTop (trained on artist data) and Puppeteer (an optimization-based upper bound).

Quantitative Metrics:
- Plausibility (LSR): ViPS achieved a Luce Spectral Ranking score of -0.04 (cross-species) and 0.65 (single-species), significantly outperforming AnyTop variants (which scored -0.21 and -2.78 respectively) and approaching the optimization-based Puppeteer (1.90).
- Diversity (FSD): ViPS showed a Fréchet Skeleton Distance of 0.31 to the Puppeteer distribution, indicating high diversity and alignment with high-quality reconstructions, compared to 1.49 for AnyTop-motion.
Qualitative Results:
- ViPS generates diverse, anatomically correct poses for unseen species (e.g., aliens, bears, dragons) where baselines often collapse into conservative poses or produce geometric artifacts.
- Semantic Editing: The model successfully performs automatic Inverse Kinematics (IK), redistributing motion across the skeleton to satisfy user edits while preventing impossible joint angles.
Efficiency: ViPS generates poses in ~4 seconds (feed-forward), whereas Puppeteer requires ~30 minutes (optimization-based) per sequence.

5. Significance and Impact

Democratization of Animation: ViPS removes the barrier of needing expensive, artist-authored 4D motion data. It enables the creation of plausible animation rigs for any 3D asset, including rare species or fantasy creatures, using only 2D video priors.
Bridging 2D and 3D: It establishes a robust link between the rich, unstructured priors of video diffusion models and the structured, controllable nature of 3D kinematics.
Foundation for Next-Gen Synthesis: By providing a "steerable" 3D proxy, ViPS enables precise control over video generation, allowing users to dictate the structure of a video (via 3D poses) while the generative model handles the appearance.
Zero-Shot Capability: The ability to generalize to unseen topologies makes this a foundational step toward a truly universal character animation system.

In summary, ViPS redefines how we approach character animation by treating the pose space not as a manually curated asset, but as a latent distribution distilled from the vast motion knowledge embedded in modern video foundation models.