💻 computer science

Generative Motion In-betweening by Diffusion over Continuous Implicit Representations

This paper proposes a novel latent diffusion model pipeline based on motion implicit neural representations (INR) that effectively reconstructs smooth and diverse motion transitions from extremely sparse keyframes while strictly preserving keyframe accuracy and ensuring motion continuity.

Original authors: Shiyu Fan, Paul Henderson, Edmond S. L. Ho

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiyu Fan, Paul Henderson, Edmond S. L. Ho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an animator trying to create a movie of a character walking from point A to point B. In the old days, you had to draw every single frame by hand. Later, computers could "guess" the frames in between if you gave them a start and an end pose. This is called motion in-betweening.

However, current computer programs often struggle when you only give them a few "keyframes" (snapshots of the character's pose). They might make the character's feet slide across the floor like they are on ice, or their movements might look jittery and unnatural. They also sometimes forget exactly where the character was supposed to be at the start and end points.

This paper introduces a new tool called Generative Motion In-betweening by Diffusion over Continuous Implicit Representations. That's a mouthful, so let's break it down with some analogies.

1. The Problem: The "Pixelated" vs. "Smooth" Map

Most animation computers treat time like a beaded necklace. They only know about specific beads (frames) at specific times. If you give them beads that are far apart, they have to guess the string connecting them. Often, the guess is wrong, leading to jerky movements or "foot sliding."

The authors propose using a smooth, continuous map instead. They use something called an Implicit Neural Representation (INR).

The Analogy: Imagine the difference between a low-resolution digital photo (pixelated, blocky) and a high-resolution vector drawing (smooth lines that look perfect no matter how much you zoom in).
How it works: Instead of storing a list of poses, their system learns a "smooth function" that describes the entire motion as a single, unbroken curve. This means the computer understands the motion as a continuous flow, not a series of disconnected snapshots. This naturally prevents the "jitter" and makes the movement look much more natural.

2. The Engine: The "Diffusion" Artist

To fill in the gaps between your sparse keyframes, the authors use a Diffusion Model.

The Analogy: Think of diffusion like a sculptor starting with a block of noisy, chaotic clay. The sculptor slowly chips away the noise, refining the shape until a perfect statue emerges.
How it works: The computer starts with random noise and slowly "denoises" it to create a motion. The challenge is making sure this new motion matches your specific start and end points (the keyframes) without losing the natural variety of movement.

3. The Secret Sauce: "Implicit Manifold Guidance" (IMG)

Here is the biggest innovation. Usually, when a diffusion model tries to follow specific rules (like "be at this exact spot at this exact time"), it gets confused. It might follow the rule so strictly that the movement becomes stiff, or it might ignore the rule and drift away.

The authors invented a new steering mechanism called Implicit Manifold Guidance (IMG).

The Analogy: Imagine you are trying to walk a tightrope (the "manifold" or the smooth path of natural motion) while holding a heavy weight that pulls you toward a specific destination (your keyframes).
- If you just pull hard toward the destination, you might fall off the tightrope (the motion becomes unnatural).
- If you just stay on the tightrope, you might miss the destination.
- IMG is the balance. It constantly checks two things:
  1. Geometric Error: "Are we at the right spot?" (Did the foot land where it should?)
  2. Manifold Error: "Are we still walking naturally?" (Are we staying on the smooth, natural path?)
The Result: The system gently nudges the motion to hit your keyframes accurately without forcing it to break the laws of physics or look robotic. It corrects the path in real-time as the "sculptor" chips away the noise.

4. Why This Matters

The paper claims that by combining these three things (Smooth Continuous Maps + Diffusion Sculpting + Smart Steering):

Accuracy: The character hits the exact start and end poses you asked for, even if you only gave them two or three snapshots.
Quality: The movement is smooth, with no foot sliding or jittering.
Variety: Unlike older methods that might produce the exact same boring walk every time, this system can generate many different, realistic ways to get from A to B.
No Text Needed: You don't need to type a description like "a happy walk" for it to work; it just uses the poses you give it.

Summary

Think of this paper as teaching a computer to be a master animator who can look at just a few rough sketches and fill in the rest of the movie with a smooth, realistic, and varied performance, all while making sure the character lands exactly where you told them to. They did this by teaching the computer to see motion as a smooth, continuous river rather than a series of choppy steps, and then giving it a special compass to stay on course.

Technical Summary: Generative Motion In-betweening by Diffusion over Continuous Implicit Representations

Problem Statement
Motion in-betweening is a fundamental task in character animation, requiring the generation of smooth, natural transitions between sparse keyframes. While recent advances in generative models, particularly diffusion models, have improved motion synthesis, existing methods face significant limitations. Traditional regression-based approaches often struggle with diversity, while current diffusion-based methods exhibit noticeable keyframe errors, motion discontinuities (such as jittering and foot sliding), and slow training/generation speeds. Furthermore, many state-of-the-art models rely on text conditioning to ensure quality, which limits their practicality in scenarios where textual descriptions are unavailable. A critical challenge remains in generating high-quality, diverse motions from extremely sparse and ambiguous spatial-temporal information without introducing unnatural artifacts.

Methodology
The authors propose a novel pipeline that integrates Latent Diffusion Models (LDM) with Motion Implicit Neural Representations (INR). The core innovation lies in sampling INR parameters from a continuous latent manifold conditioned on sparse keyframes, rather than operating on discrete time steps.

Motion Representation (INR): The system represents motion sequences as continuous functions mapping time to poses, encoded into a compact latent vector $z$ . This is achieved using a Variational Autoencoder (VAE) based on NeMF, which decomposes motion into global (root orientation and translation) and local (joint positions, rotations, velocities) components. This continuous representation captures global motion patterns and smooths transitions, mitigating the noise and discontinuities often found in discrete-time modeling.
Latent Diffusion Sampler: The authors design a diffusion sampler based on a Diffusion Transformer (DiT). Unlike standard self-attention mechanisms, the network utilizes cross-attention to integrate information from randomly selected sparse keyframes into the denoising process. This allows the model to map sparse constraints to the compact latent space effectively, even with large temporal gaps.
Training Objective: The model is trained to predict the clean latent $z_0$ from noisy inputs. The loss function combines the standard denoising objective with geometric consistency terms (root translation, root orientation, joint position, and joint rotation errors) calculated at the keyframe locations. Classifier-free guidance (CFG) is employed to enable flexible conditioning.
Implicit Manifold Guidance (IMG): To further reduce keyframe error while preserving the diversity of the generative model, the authors introduce a sampling optimization strategy called IMG. During the inference process (specifically the final 300 steps of a 1000-step DDIM schedule), the sampling direction is refined using two gradient-based constraints:
- Geometric Loss: Minimizes the error between the generated motion and the ground-truth keyframes.
- Manifold Preservation Loss: Prevents the optimization from pushing the latent representation off the learned data manifold. This is achieved by interpolating the generated motion with a spherical linear interpolation (Slerp) of the keyframes, encoding this "inpainted" motion, and minimizing the distance between the original prediction and this inpainted latent. This ensures the generated motion remains realistic and smooth.

Key Contributions

First Adaptation of Motion INR to Generative In-betweening: The paper presents the first framework to adapt Motion INR to the generative motion in-betweening task using a latent diffusion model. This approach leverages compact, continuous-time representations to generate high-quality motions from extremely sparse keyframes without text conditioning.
Manifold Preservation Strategy: The authors introduce the concept of manifold preservation for sampling optimization. They design a novel loss function within the posterior sampling algorithm tailored for motion implicit manifolds, which effectively reduces keyframe error while maintaining motion quality and diversity.
Superior Performance in Sparse Scenarios: The proposed method achieves state-of-the-art performance in motion in-betweening tasks conditioned on extremely sparse keyframes (e.g., 1, 5, or 20 random frames) without requiring textual descriptions, outperforming existing diffusion-based and VAE-based baselines.

Results
Extensive experiments were conducted on the HumanML3D dataset (using the SMPL representation) with comparisons against baselines such as CondMDI, NeMF, MMM, and TST.

Quantitative Metrics: The proposed model consistently outperformed baselines in Fréchet Inception Distance (FID), Diversity, and Multi-Modality across various keyframe densities (K=1, 5, 20). Notably, it achieved significantly lower keyframe errors compared to CondMDI and NeMF, particularly in sparse scenarios (K=1 and K=5).
Qualitative Improvements: Visual comparisons demonstrated that the model generates smoother transitions with fewer artifacts like foot sliding or jittering. Unlike baselines that often fail to reach target positions or exhibit unnatural limb movements, the proposed method maintained global trajectory accuracy and natural stride lengths.
Ablation Studies: Experiments confirmed the effectiveness of the INR representation over other VAE types (MLD, MMM) and validated that the full IMG strategy (combining geometric and manifold losses) yields the best balance between low keyframe error and high motion diversity.
Generalization and Efficiency: The model demonstrated strong generalization when tested on the AIST++ dataset (trained only on HumanML3D). Additionally, the use of cross-attention on 1D INR parameters rather than full sequence self-attention reduced computational complexity, and the flexible IMG sampling allowed for efficient trade-offs between speed and quality.

Significance
The paper claims that by establishing a mapping between INR and sparse spatial-temporal information within a latent diffusion framework, the proposed method successfully reconstructs plausible and smooth motions from extremely sparse data. The significance of this work lies in its ability to decouple high-quality motion generation from the need for dense keyframes or text conditioning. By introducing manifold preservation during sampling, the authors address the common trade-off between adhering to strict constraints and maintaining generative diversity. This approach offers a practical solution for character animation scenarios where only limited input data is available, achieving state-of-the-art results in challenging sparse keyframe settings while ensuring both keyframe accuracy and the diversity of in-between motions.