Technical Summary: Sketch2Motion

Problem Statement

Animating 2D hand-drawn sketches into 3D sequences presents significant challenges due to the inherent ambiguity, sparsity, and lack of explicit depth cues in 2D representations. While 3D animation naturally handles occlusions and complex motion, inferring plausible 3D structure and motion from 2D inputs is difficult. Existing methods often rely on specific motion categories (e.g., bipedal walking), predefined templates, or large-scale paired datasets, limiting their applicability to diverse character types and stylized sketches. Furthermore, recent text-to-3D approaches often lack physics-based constraints and soft-body dynamics, leading to animations that may be semantically aligned but physically implausible or temporally inconsistent.

Methodology

The authors propose Sketch2Motion, a diffusion-guided framework that synthesizes skeleton-based 3D animations from 2D sketches using text prompts. The pipeline integrates classical character animation techniques with deep generative priors.

1. Pipeline Overview

The process begins by inflating a 2D sketch into a textured 3D mesh. A kinematic skeleton and joint hierarchy are automatically inferred from this geometry. The system then optimizes skeletal motion parameters (joint rotations, root translations, and local offsets) guided by a text prompt, without requiring paired motion data.

2. Motion Representation and Initialization

• Representation: Motion is parameterized using global root translation, joint rotations (in axis-angle form), and local translational offsets to model non-rigid secondary dynamics.
• Initialization: To avoid degenerate local minima common in optimizing from scratch, the method employs a Motion Initialization module. This module classifies the character's morphology (biped, quadruped, etc.) and the semantic action (walking, jumping) to generate a procedural base motion using sinusoidal primitives. These dense trajectories are projected onto a low-dimensional manifold using Non-Uniform Rational B-Splines (NURBS), ensuring temporal smoothness and providing a physically plausible starting point for optimization.

3. Diffusion-Guided Optimization (MoSDS)

The core of the method utilizes a pre-trained Text-to-Video (T2V) diffusion model as a semantic prior. Instead of generating video directly, the model guides the optimization of skeletal parameters via Motion-Aware Score Distillation Sampling (MoSDS).

• Decoupling: Unlike standard Score Distillation Sampling (SDS), which can cause temporal flickering, MoSDS decouples the noise residual into an appearance gradient (temporal mean) and a motion gradient (frame-wise deviation).
• Optimization: The system heavily weights the motion gradient, forcing the optimizer to treat the diffusion model as a kinematic critic. This updates skeletal parameters to align with the text prompt's motion dynamics while preserving the 3D geometry's structural integrity.

4. Physics-Inspired Regularization

To ensure physical plausibility and stability, the optimization objective includes several regularization terms:

• Temporal Smoothness: Penalizes high-frequency jitter in velocity and acceleration.
• Topological Constraints: Enforces joint limits (Range of Motion) and symmetry for bilateral characters.
• Cyclic Consistency: Ensures seamless looping for periodic motions.
• Environment Interaction: Includes ground penetration penalties and a foot-contact loss to prevent foot-skating and ensure proper weight transfer.

5. Secondary Motion Simulation

To enhance realism, the framework integrates a Spring-Mass Simulator. The mesh is automatically partitioned into rigid (skeletal) and deformable (soft body) regions based on skinning weights. Dynamic vertices (e.g., hair, cloth, tails) are simulated using spring forces, structural constraints, and damping, generating natural secondary motion effects like follow-through and overlapping action.

Key Contributions

1. Diffusion-Guided Skeleton Optimization: A framework that applies motion-aware score distillation from a T2V diffusion model to directly optimize skeletal motion parameters, enabling semantic motion synthesis without paired data across diverse articulated entities.
2. NURBS Conditioning: A novel kinematic conditioning framework using NURBS to initialize and smooth motion trajectories, stabilizing the diffusion-guided optimization process.
3. Physics-Inspired Constraints: A comprehensive set of regularization terms (smoothness, joint limits, contact, symmetry) essential for producing stable and realistic animations.
4. Secondary Motion Integration: A spring-mass simulation framework that estimates and applies secondary motion effects to object components, bridging the gap between kinematic generation and dynamic simulation.
5. Dataset: The release of a dataset containing 100 hand-drawn sketches spanning diverse object categories (bipeds, quadrupeds, non-living objects) and structural complexities.

Results and Evaluation

The authors evaluate Sketch2Motion against state-of-the-art methods, including AnimateAnyMesh and BiMotion, using both qualitative and quantitative metrics.

• Quantitative Metrics:
  • Text-to-Video Alignment (T2VA): Sketch2Motion achieves the highest score (0.2120), indicating superior semantic alignment with text prompts compared to baselines.
  • Mesh Laplacian Distortion (MLD): The method achieves the lowest distortion (0.3523), demonstrating better geometric stability and surface preservation than AnimateAnyMesh (0.6796).
  • Multi-View Feature Consistency (MVFC): The approach yields the highest consistency (0.8933), confirming coherent 3D structure across different viewpoints.
• Qualitative Analysis: Visual results show that while baselines often suffer from structural inconsistencies, foot-skating, or lack of secondary motion, Sketch2Motion produces temporally coherent, physically plausible animations that respect skeletal constraints and exhibit natural secondary dynamics.
• Ablation Studies: Removing motion initialization leads to convergence issues and artifacts (e.g., floating geometry). Removing physics regularization results in physically incoherent motion (e.g., broken joints, sliding feet). Removing secondary motion results in stiff, artificial animations lacking organic "follow-through."

Significance

The paper claims that Sketch2Motion provides a generalized, fully differentiable, and modular solution for controllable 3D animation. Its significance lies in successfully bridging the gap between the generative flexibility of diffusion models and the structural rigor of classical animation pipelines. By optimizing explicit skeletal parameters rather than relying on implicit deformations or paired datasets, the method generalizes to bipeds, quadrupeds, and non-living articulated objects. It addresses the critical limitations of existing text-to-3D methods by incorporating physics-inspired constraints and secondary dynamics, resulting in animations that are not only semantically aligned but also physically plausible and temporally stable. The authors note that the code and dataset will be made publicly available to facilitate further research in this domain.