💻 computer science

Feed-forward Motion In-betweening for Any 4D

This paper proposes a novel feed-forward in-betweening framework that leverages a frame-wise mesh VAE and a keyframe-conditioned rectified flow model to efficiently generate long-horizon 4D mesh sequences with arbitrary shapes and enhanced spatiotemporal controllability, overcoming the limitations of slow inference and error accumulation in existing methods.

Original authors: Hiroki Nishizawa, Hubert P. H. Shum, Yoshihiro Fukuhara, Hirokatsu Kataoka, Shigeo Morishima

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hiroki Nishizawa, Hubert P. H. Shum, Yoshihiro Fukuhara, Hirokatsu Kataoka, Shigeo Morishima

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an animator trying to create a movie. Usually, you have to draw every single frame by hand, or at least draw the key moments (like a character jumping) and let the computer fill in the rest. This paper introduces a new tool called CompletionAny4D that acts like a super-smart, instant "fill-in-the-blanks" machine for 3D movies.

Here is how it works, broken down with simple analogies:

The Big Problem: The "Slow and Clunky" Computer

Before this new tool, making 3D movies (called 4D, which is 3D objects moving over time) was like trying to bake a cake by hand-kneading the dough for every single second of the movie.

Old Way: Computers had to do a massive amount of math for each specific scene just to figure out how the object should move. This took hours (sometimes 20 hours for a few seconds of video) and was very hard to control. If you wanted the character to raise their left arm, the computer often guessed wrong or took forever to get it right.
The Limitation: Existing fast tools were like a "random motion generator." They could make things move quickly, but you couldn't tell them exactly what to do. If you said "jump," it might jump, but it might also twist in a weird way you didn't want.

The Solution: CompletionAny4D

The authors built a system that is like a professional animator's assistant that works in seconds, not hours. It can take a 3D object (like a robot, a cat, or a tree) and a few "keyframes" (snapshots of the start and end of a movement, plus maybe a few in the middle) and instantly fill in all the smooth motion between them.

Here are the three "secret ingredients" that make it work:

1. The "Skeleton vs. The Dance" (Frame-wise VAE)

Imagine you have a puppet. The shape of the puppet (its head, arms, legs) is one thing, but the dance it does is another.

Previous tools tried to memorize the whole dance at once, which made it hard to stop and change a specific move.
CompletionAny4D separates the puppet's shape from its movement. It looks at the puppet's shape once (the "anchor") and then treats every single frame of the dance as its own separate instruction. This allows the computer to say, "Okay, at this exact second, the arm must be here," without messing up the rest of the body.

2. The "GPS Navigation" (Keyframe Conditioning)

Think of the motion as a road trip.

Old Fast Tools: They would just drive in a general direction. You might end up in the right city, but you might take a weird detour or miss the specific turn you wanted.
CompletionAny4D: You give it a map with specific checkpoints (Keyframes). "Start here, stop at this tree, and end at the mountain." The computer is forced to drive exactly through those checkpoints. It doesn't just guess; it calculates the smoothest, most natural path that must pass through your specific points.

3. The "Straight-Line Magic" (Rectified Flow)

This is the engine under the hood. Imagine you want to get from Point A to Point B.

Old methods: They might take a winding, confusing path, trying to figure out the best way step-by-step, which causes errors to pile up (like a drunk walk).
CompletionAny4D: It uses a technique called "Rectified Flow." Think of this as drawing a perfectly straight line between your start and end points and then filling in the dots along that line. It's much faster and much more accurate because it doesn't get confused or drift off course.

What It Can Do (and What It Can't)

The Wins:

Speed: It generates 16 seconds of animation in about 4 seconds on a powerful computer. That's instant compared to the 20 hours it used to take.
Control: If you give it a text prompt like "jump and spin" and a few keyframes, it follows your instructions much better than previous tools.
Long Stories: Even though it was trained on short clips, it can stitch these short clips together to make longer movies without the character getting "drunk" or losing its shape.

The Limits (What the paper admits):

No "Partial" Control: You can't tell it to "only move the left hand" while keeping the rest perfectly still. It controls the whole object at once.
No Shape-Shifting: If the object is supposed to change its fundamental shape (like a caterpillar turning into a butterfly), this tool can't do that. It keeps the object's "skeleton" the same throughout the movie.
One Shot: It generates a fixed-length movie in one go. It can't keep generating forever in a single pass (though you can stitch them together).

The Bottom Line

CompletionAny4D is a new way to make 3D animations that is fast, controllable, and precise. Instead of guessing how a 3D object should move, it listens to your specific instructions (keyframes) and fills in the gaps instantly, making it possible to create complex 3D animations in seconds rather than hours.

Technical Summary: CompletionAny4D

Problem Statement

4D dynamics, representing 3D geometry evolving over time, are fundamental to world modeling in animation and games. However, generating controllable 4D mesh sequences remains challenging due to the scarcity of large-scale, long-horizon 4D datasets with arbitrary topologies. Existing approaches face two primary limitations:

Inference Efficiency vs. Controllability: Early text-to-4D methods rely on distilling video diffusion priors or test-time optimization (e.g., Score Distillation Sampling). While these avoid the need for curated 4D training data, they require prohibitively expensive per-instance optimization (e.g., ~20 hours for 41 frames), rendering them impractical for interactive use.
Limitations of Feed-Forward Generators: Recent feed-forward models offer fast inference but lack precise spatiotemporal controllability. They often struggle to generate specific desired motions based on text prompts alone. Furthermore, extending these models to long-horizon sequences via autoregressive generation leads to error accumulation and motion drift, as they lack explicit constraints to maintain consistency over time.

The core problem addressed is how to achieve fast, feed-forward 4D mesh generation that supports precise spatiotemporal control via sparse keyframes and text, while maintaining consistency over long horizons, all without requiring per-scene optimization.

Methodology: CompletionAny4D

The authors propose CompletionAny4D, a text-and-keyframe-conditioned feed-forward framework designed for motion in-betweening on arbitrary 4D meshes. The framework decouples topology/shape information from motion information and operates in a latent space.

1. Frame-wise Mesh VAE

To enable direct keyframe conditioning, the authors design a frame-wise mesh VAE, contrasting with existing sequence-wise VAEs (e.g., DyMeshVAE) that compress temporal information.

Architecture: The encoder processes each frame independently. It extracts a shared shape structure latent ( $\mathbf{V}_0^n$ ) anchored by a reference mesh ( $M_0$ ) and generates per-frame trajectory latents ( $\mathbf{V}_t^n$ ) for each timestep.
Mechanism: By encoding frames independently while preserving a shared shape anchor, the model creates a frame-aligned latent space. This allows for the direct "clamping" or replacement of latent tokens at specific keyframe indices without implicitly modifying non-keyframe timesteps, a capability hindered by temporal compression in sequence-wise models.

2. Keyframe-Conditioned Rectified Flow

The generation of non-keyframe frames is modeled as a motion in-betweening task using a Rectified Flow model with an MMDiT (Masked Multi-Modal Diffusion Transformer) backbone.

Conditioning Strategy: Instead of simple latent replacement at sampling time (which yields weak control), the model employs a masked latent replacement scheme during training and inference.
- A binary mask $m$ identifies observed keyframes.
- The conditioning input to the flow model consists of the text prompt, the shared shape latent, and a concatenated sequence where keyframe latents are "clean" (ground truth) and non-keyframe latents are "noised."
Training Objective: The model learns a velocity field $\mathbf{v}_\theta$ to transport noise to data along straight lines. The loss function minimizes the regression error between the predicted velocity and the target velocity, conditioned on the masked sequence.
Inference: Given a text prompt, a reference mesh, and sparse keyframes, the model solves the learned Ordinary Differential Equation (ODE) to synthesize the intermediate trajectory latents, which are then decoded into the full 4D mesh sequence.

Key Contributions

CompletionAny4D Framework: The first feed-forward framework for spatiotemporally controllable 4D mesh motion in-betweening on arbitrary topologies. It enables precise control via sparse keyframes and text, trained on short-horizon data but capable of long-horizon generation.
Frame-wise Mesh VAE: A novel VAE design that disentangles shape structure from motion and encodes frames independently. This architecture is essential for enabling direct keyframe latent injection and masked conditioning, overcoming the limitations of temporally compressed latent spaces.
Keyframe-Conditioned Rectified Flow: A rectified flow model that integrates sparse keyframe constraints directly into the conditioning mechanism via masked latent replacement. This approach learns to respect keyframe constraints during training, resulting in superior controllability compared to inference-time replacement baselines.
Long-Horizon Generation via In-betweening: The demonstration that sparse keyframe conditioning can stabilize generation beyond the training horizon. By chaining in-betweening tasks (e.g., generating 16 frames, then using the last frame as a new start keyframe), the method produces consistent long sequences without the drift typical of autoregressive generation.

Experimental Results

The method was evaluated on DyMesh16 (16-frame) and DyMesh32 (31-frame) benchmarks.

Short-Horizon Performance (DyMesh16):
- Accuracy: CompletionAny4D significantly outperformed both the released AnimateAnyMesh model and a sampling-time replacement baseline in vertex accuracy metrics (RMSE, Chamfer distance, DTW) on unconstrained frames. For instance, with 3 keyframes, the proposed method achieved an RMSE of 1.852 (scaled $\times 100$ ) compared to 6.438 for the baseline and 6.424 for AnimateAnyMesh.
- Controllability: User studies (20 participants) showed the proposed method achieved the highest scores in "Control" (4.12/5) and "Text" alignment (3.37/5), surpassing both the baseline and the open-source AnimateAnyMesh model.
- Quality: While AnimateAnyMesh achieved slightly higher VBench scores in open settings (no keyframes), CompletionAny4D maintained competitive motion naturalness and aesthetic quality while offering superior controllability.
Long-Horizon Performance (DyMesh32):
- Evaluated via a two-stage generation protocol (16 frames + 16 frames with overlap).
- Stability: AnimateAnyMesh exhibited significant rotational error accumulation and drift in the second stage. In contrast, CompletionAny4D maintained spatiotemporal consistency across stages, achieving the lowest RMSE (2.132) and DTW (0.7272) among all methods.
- User Preference: Users rated CompletionAny4D highest in naturalness, shape preservation, and controllability for long sequences.
Efficiency: The framework demonstrated fast inference speeds (approx. 2.8 seconds for 16 frames on an H100 GPU), significantly faster than optimization-based methods and competitive with or faster than other feed-forward baselines.

Significance and Claims

The paper claims that CompletionAny4D represents a significant step forward in controllable 4D content creation. Its primary significance lies in demonstrating that explicit spatiotemporal conditioning (via sparse keyframes and text) is a viable and effective alternative to text-only generation or expensive test-time optimization.

The authors emphasize that:

Training-time conditioning is superior to inference-time replacement: Learning to respect keyframe constraints within the flow model yields much better results than simply clamping latents during sampling.
Frame-aligned latents are necessary: The proposed frame-wise VAE is a critical architectural component that enables the direct manipulation of specific frames, a requirement for robust in-betweening that sequence-wise models cannot easily satisfy.
Short-to-Long Generalization: The method proves that models trained on short sequences can generate long, consistent 4D motions if guided by sparse keyframes, effectively solving the error accumulation problem inherent in autoregressive approaches.

The paper concludes that while text prompts alone are often insufficient for specifying complex, precise motions, the combination of text and sparse keyframes provides a powerful, practical interface for generating controllable 4D mesh animations.