Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models

Stroke3D is a novel framework that generates animatable, rigged 3D meshes directly from user-drawn 2D strokes and text prompts by employing a two-stage pipeline involving latent diffusion-based skeleton generation and enhanced mesh synthesis via a curated dataset and preference optimization.

The Gist

Imagine you want to create a 3D character for a video game or an animated movie. Usually, this is like trying to build a complex robot from scratch while blindfolded. You have to sculpt the body, then manually insert a skeleton inside it, and finally paint a skin over the whole thing. It takes weeks of work by highly skilled experts.

Stroke3D is like a magical "instant robot builder" that lets you do this in minutes, just by drawing a few lines on a piece of paper and typing a sentence.

Here is how it works, broken down into simple steps:

1. The Problem: The "Static Statue" vs. The "Living Puppet"

Most AI tools today are great at making 3D statues. You can ask them to "make a dragon," and they will give you a cool-looking dragon. But if you try to make that dragon run or fly, it falls apart because it has no bones inside. It's a solid block of clay, not a puppet.

Other tools try to add bones, but they are like a clumsy surgeon—they often put bones in the wrong places (like a tail bone in the head) or miss bones entirely. They don't really understand how the character should move.

2. The Solution: Drawing the "Blueprint" First

Stroke3D flips the script. Instead of building the body first and hoping the bones fit, it builds the skeleton first, based on your drawing.

Think of it like an architect. Before building a house, you draw the floor plan.

• You (The User): You open a digital canvas and draw a few stick-figure lines. Maybe you draw a line for a spine, a few for legs, and a curve for a tail. You also type, "A dinosaur is ready to jump."
• The AI (The Architect): It looks at your scribbles and your text, and it instantly understands: "Ah, they want a jumping dinosaur. I need to build a skeleton with strong legs and a balancing tail."

3. The Magic Trick: The "Ghost Skeleton"

The paper describes a two-stage process, which is like a two-step recipe:

Step A: The Skeleton Generator (The "Bone Builder")
The AI uses a special "translator" (called a Latent Diffusion Model) to turn your 2D scribbles into a 3D skeleton.

• The Analogy: Imagine your 2D drawing is a shadow on a wall. The AI is a wizard who can look at that shadow and instantly conjure the 3D object casting it.
• It doesn't just guess; it uses your drawing as a strict guide. If you drew a long neck, the AI builds a long neck. If you drew a text prompt saying "flying bird," it adds wings. It creates a perfect, animated-ready skeleton that matches your intent.

Step B: The Mesh Synthesizer (The "Skin & Clothes Maker")
Once the skeleton is built, the AI needs to put "skin" and "clothes" on it.

• The Problem: Usually, AI struggles to make skin that fits perfectly over moving bones (like skin bunching up at the elbow).
• The Fix: The researchers created a massive library called TextuRig. Think of this as a giant photo album of thousands of 3D characters that already have perfect skin and bones. They taught the AI to learn from this album.
• They also used a technique called SKA-DPO. Imagine a teacher grading two different drawings of a bird. The teacher says, "This one has wings that look like they can actually fly (High Score), but this one looks like a broken toy (Low Score)." The AI learns to prefer the "High Score" version, refining its work until the skin fits the bones perfectly.

4. The Result: Ready-to-Go Animation

The final output isn't just a pretty picture. It's a rigged 3D asset.

• Rigged means it has a digital skeleton inside it, just like a real puppet.
• You can take this character, plug it into animation software (like Blender), and immediately make it run, jump, or dance. The skin stretches and bends naturally because the AI built the bones and the skin to work together from the start.

Why is this a big deal?

• Democratization: You don't need to be a 3D artist. If you can draw a stick figure and type a sentence, you can make a movie character.
• Control: You aren't at the mercy of the AI's random guess. Your drawing dictates the structure. If you want a character with three legs, you draw three legs, and the AI builds three legs.
• Speed: What used to take days of manual work now happens in seconds.

In summary: Stroke3D is like a magic sketchbook. You draw the bones, describe the character, and the AI instantly builds a fully animated, 3D puppet that is ready to star in your next movie or game.

Technical Summary

1. Problem Statement

The paper addresses two critical limitations in current 3D asset generation:

1. Lack of Animatable Geometry: Most existing 3D generation methods produce static geometries (meshes) that lack the skeletal hierarchy required for animation. While some methods generate skeletons from meshes, they often lack fine-grained control over the skeleton's structure.
2. Limited Structural Control: Current skeleton generation methods typically rely on end-to-end mesh-to-skeleton paradigms. Without explicit structural constraints, these methods often produce skeletons with joints in incorrect locations or missing critical joints, leading to unpredictable results that are difficult to rig.

The authors propose Stroke3D, a framework that allows non-professional users to generate ready-to-animate (rigged) 3D meshes directly from intuitive inputs: 2D user-drawn strokes and text prompts.

2. Methodology

Stroke3D employs a novel two-stage pipeline that decouples skeleton generation from mesh synthesis, ensuring high structural fidelity and semantic alignment.

Stage 1: Controllable Skeleton Generation

This stage generates a 3D skeletal structure conditioned on 2D strokes and text.

• Representation: The skeleton is represented as a graph $G=(X, E)$, where $X$ are joint coordinates and $E$ are edges.
• Skeletal Graph VAE (Sk-VAE): A Variational Autoencoder encodes the skeletal graph structure into a continuous latent space. It uses GCN and TransformerConv layers to aggregate structural information between neighboring nodes.
• Skeletal Graph Diffusion Transformer (Sk-DiT): A latent diffusion model operates within the Sk-VAE's latent space to generate skeletal embeddings.
  • Inputs: It takes the latent noise, text embeddings (via CLIP), and 2D stroke features.
  • Structural Guidance: To handle the ambiguity of 3D topology from 2D inputs, the model concatenates features derived from user-drawn strokes (2D joint coordinates and edge topology) with the latent noise. During training, stroke-like data is simulated by perturbing 2D projections of ground-truth skeletons.
  • Architecture: The Sk-DiT replaces standard self-attention with TransformerConv to respect graph connectivity and uses cross-attention to integrate semantic text guidance.
• Output: The Sk-VAE decoder reconstructs the final 3D skeleton from the generated latent embedding.

Stage 2: Enhanced Mesh Synthesis

This stage synthesizes a textured mesh conditioned on the generated skeleton.

• TextuRig Dataset: The authors curate a new dataset called TextuRig from Objaverse-XL. Unlike previous datasets, TextuRig specifically filters for models that possess both high-quality rigging (skeletons) and textures. It also re-captions these models using Vision-Language Models (VLMs) to provide rich semantic descriptions.
• Data Augmentation: The TextuRig dataset is used to augment the training data for the existing SKDream model (a skeleton-to-mesh generator), improving its ability to handle texture-aware generation.
• SKA-DPO (Skeleton-Mesh Alignment Direct Preference Optimization): To further refine geometric fidelity, the authors apply a preference optimization strategy.
  • Process: A reference model generates multiple mesh candidates for a given skeleton.
  • Scoring: A fine-tuned Dinov2 model calculates a SKA Score (SKeleton Alignment Score) to measure the geometric correspondence between the mesh and the skeleton.
  • Optimization: Pairs of "winning" (high score) and "losing" (low score) meshes are constructed. The model is fine-tuned using Direct Preference Optimization (DPO) to maximize the probability of generating meshes that align better with the skeleton.

3. Key Contributions

1. First 2D Stroke-to-Rigged-3D Framework: To the best of the authors' knowledge, this is the first work to generate rigged 3D meshes directly from user-drawn 2D strokes and text prompts, pioneering a "skeleton-first" workflow.
2. Latent Graph Diffusion for Skeletons: The introduction of Sk-VAE and Sk-DiT allows for the generation of structurally coherent skeletons with explicit control via 2D strokes, overcoming the unpredictability of end-to-end mesh-to-skeleton methods.
3. TextuRig Dataset: The curation of a high-quality dataset of textured, rigged, and captioned 3D models, addressing the scarcity of paired skeleton-texture data.
4. SKA-DPO Strategy: A novel preference optimization technique guided by a skeleton-mesh alignment score, significantly improving the geometric fidelity and structural coherence of the final mesh.

4. Experimental Results

The authors evaluated Stroke3D on the MagicArticulate (skeleton) and SKDream (mesh) benchmarks.

• Skeleton Generation:

  • Stroke3D achieved the lowest Chamfer Distance (CD) across all metrics (Joint-to-Joint, Joint-to-Bone, Bone-to-Bone) compared to baselines like RigNet, SKDream, MagicArticulate, and UniRig.
  • It demonstrated superior performance across diverse categories (Characters, Animals, Plants), maintaining balanced improvements where other methods struggled with specific classes.
  • Qualitative results showed that Stroke3D produces skeletons that strictly adhere to the user's drawn strokes, eliminating redundant or missing bones common in other methods.

• Mesh Generation:

  • Integrating TextuRig improved the Mean Instance (MeanInst.) SKA score by nearly 1.9 points over the SKDream baseline.
  • Applying SKA-DPO further boosted the MeanInst. score to 87.83 (a ~2.4 point increase over the baseline), demonstrating significant gains in skeleton-mesh alignment.
  • The generated meshes were successfully processed by standard automatic skinning tools (e.g., in Blender) and maintained structural integrity during animation, avoiding the collapsing or distortion seen in baseline methods.

• Ablation Studies:

  • Structural Condition: Removing the 2D stroke condition led to significantly slower convergence and lower quality, proving the necessity of explicit structural guidance.
  • Preference Margin: An optimal preference score margin of 0.10 was found for SKA-DPO.
  • Robustness: The model remained stable even when input strokes were sparse or perturbed.

5. Significance

Stroke3D represents a significant leap in democratizing 3D content creation. By decoupling the complex tasks of rigging and mesh generation and providing explicit structural control via 2D strokes, it lowers the barrier to entry for creating animatable 3D assets.

• Workflow Efficiency: It enables a "draw-and-generate" workflow, bypassing the steep learning curve of professional software like Blender for rigging.
• Application Potential: The framework is directly applicable to AR/VR, robotics simulation, and film production, where ready-to-animate assets are crucial.
• Technical Advancement: The combination of latent graph diffusion for topology control and DPO for geometric refinement sets a new standard for controllable 3D generation.