💻 computer science

SMAL-pets: SMAL Based Avatars of Pets from Single Image

This paper introduces SMAL-pets, a hybrid framework that combines 3D Gaussian Splatting with the SMAL parametric model to generate high-fidelity, animatable, and text-editable 3D dog avatars from a single input image, addressing challenges in data scarcity, morphological diversity, and realistic fur reconstruction.

Original authors: Piotr Borycki, Joanna Waczyńska, Yizhe Zhu, Yongqiang Gao, Przemysław Spurek

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Piotr Borycki, Joanna Waczyńska, Yizhe Zhu, Yongqiang Gao, Przemysław Spurek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single, beautiful photograph of your dog. Now, imagine you could magically pull that dog out of the photo, turn it into a 3D character, and make it sit, walk, or even give you a high-five—all just by typing a sentence like "Make the dog sit down."

That is exactly what the SMAL-pets paper is about. It's a new tool that turns a flat picture of a pet into a fully animated, 3D digital twin.

Here is how it works, explained through a few simple analogies:

1. The Problem: The "Plastic" Dog

Before this paper, making 3D animals was like trying to sculpt a realistic dog out of Play-Doh.

The "Plastic" Look: Old methods could make a 3D dog, but it often looked like a shiny, plastic toy. It lacked real fur, and if you tried to move its leg, the whole thing would stretch weirdly or fall apart.
The Missing Skeleton: To make a human 3D character move, you need a digital skeleton (rigging). Doing this for animals is a nightmare because every dog breed looks different. There isn't a "one-size-fits-all" skeleton for a Chihuahua and a Great Dane.

2. The Solution: The "Hybrid" Approach

The authors created SMAL-pets, which is like building a house using two different construction methods at the same time.

Part A: The Invisible Skeleton (SMAL)
Think of the SMAL model as a flexible, invisible wireframe skeleton. It knows the rules of how a dog's body is supposed to be built (where the legs attach, how the spine bends). It ensures the dog moves realistically, just like a real animal.
Part B: The Cloud of Dust (3D Gaussians)
Think of 3D Gaussian Splatting as a cloud of millions of tiny, glowing dust particles. These particles are incredibly good at capturing fine details like fluffy fur, shiny noses, and soft shadows. They make the dog look photorealistic.

The Magic Trick: SMAL-pets glues the "dust cloud" onto the "wireframe skeleton."

The skeleton makes sure the dog can bend its knees and wag its tail without breaking.
The dust cloud makes sure the fur looks soft and real, not like plastic.

3. The Two-Step Construction Process

The paper describes a two-stage building process:

Stage 1: The Rough Draft (Bound Optimization)
Imagine you are trying to wrap a blanket (the dust cloud) tightly around a mannequin (the skeleton). At first, the blanket is stuck to the mannequin's surface. This helps the computer figure out the basic shape and size of the dog.
Stage 2: The Fine Details (Unbound Optimization)
Once the shape is right, the computer lets the blanket "float" slightly away from the mannequin. This allows the dust particles to pop out and create fluffy fur, whiskers, and other tiny details that a tight blanket couldn't capture. It's like letting the fur stand up naturally instead of being pressed flat against the skin.

4. Making it Move: The "Remote Control"

Once the 3D dog is built, how do you make it move?

Direct Control: Because the dog is built on that flexible wireframe skeleton, you can just twist the skeleton's joints (like a puppeteer), and the fur and body follow perfectly.
Text Commands: This is the coolest part. You can type, "The dog is playing fetch," and the system uses AI to figure out what that movement looks like. It then translates that idea into movements for your 3D dog's skeleton. It's like having a remote control where you just speak the command, and the dog obeys.

5. Why This Matters

Before this, if you wanted a 3D dog for a video game or a movie, you needed a team of experts to spend weeks manually sculpting and rigging the model.

SMAL-pets changes the game by:

Speed: It takes a single photo and does the heavy lifting automatically.
Realism: It captures the "fluffiness" of real fur, not just smooth plastic shapes.
Ease: You don't need to be a 3D artist. You just need a photo and a text prompt.

In a nutshell: SMAL-pets is like a magic translator. It takes a 2D photo, understands the 3D bones inside the dog, covers it with realistic fur, and gives you a remote control to make it dance, run, or sit—all with a single click (or a typed sentence).

1. Problem Statement

The creation of high-fidelity, animatable 3D animal avatars (specifically dogs) from a single input image remains a significant challenge in computer vision and graphics. Existing methods face several critical limitations:

Data Scarcity: Unlike human avatars (which benefit from large datasets like SMPL), animal reconstruction suffers from a lack of large-scale, annotated datasets.
Morphological Diversity: Animals vary wildly in size, proportions, and features across breeds, making generalization difficult.
Texture and Detail: Current methods often fail to capture realistic fur textures, resulting in "plastic-like" or cartoonish appearances.
Animation and Rigging: Generating avatars that are fully editable and capable of complex, naturalistic movements typically requires labor-intensive manual mesh manipulation and expert rigging.
Single-Image Limitations: Reconstructing 3D geometry from a single view is an ill-posed problem, often leading to artifacts or static models that cannot be posed.

2. Methodology: SMAL-pets

The authors propose SMAL-pets, a hybrid framework that bridges generative modeling (3D Gaussian Splatting) and parametric reconstruction (SMAL model). The pipeline consists of four main components:

A. Hybrid Representation

The avatar is represented as a combination of:

SMAL Parametric Mesh: A skinned, multi-animal linear model providing the anatomical skeleton and structural priors ( $V = M(\beta, \theta, t, d)$ ).
3D Gaussian Splatting (3DGS): A set of Gaussian primitives ( $G$ ) representing high-fidelity visual appearance, including fine details like fur.

B. Training Pipeline (Two-Stage Optimization)

Since a single image lacks 3D information, the method first synthesizes a dense multi-view dataset using a feed-forward image-to-3D model (e.g., TripoSG) as "pseudo-ground truth." The optimization occurs in two stages:

Stage 1: Bound Optimization
- Gaussian primitives are constrained to the surface of the SMAL mesh using learnable barycentric coordinates relative to mesh faces.
- Goal: Align the SMAL topology with the generated geometry.
- Loss Function: Includes photometric loss (L1 + SSIM), edge length and Laplacian smoothing for mesh stability, pose regularization, and an opacity term to ensure surface coverage.
Stage 2: Unbound Optimization
- Gaussians are decoupled from the mesh surface to capture high-frequency details (e.g., fur) that the low-poly mesh cannot represent.
- Goal: Refine volumetric details while maintaining structural integrity.
- Constraints: A point-to-mesh distance term ( $L_{dist}$ ) is added to penalize Gaussians drifting too far from the underlying anatomy, ensuring the mesh still guides the shape. Scale regularization prevents large, spiky artifacts.
- Loss Function: Includes photometric loss, mesh regularization, distance constraints, scale regularization, and entropy-based opacity regularization.

C. Animation via Face-Based Skinning

To enable animation, the optimized Gaussians are re-associated with the SMAL mesh:

Skinning Weights: Calculated based on the inverse distance to the nearest mesh faces.
Local Coordinate Systems: Each Gaussian is baked into a local frame defined by its associated mesh face.
Deformation: During inference, modifying the SMAL pose parameters ( $\theta$ ) updates the mesh. The Gaussians deform using a Linear Blend Skinning (LBS) analogue, where their position, rotation, and scale are updated based on the movement of the underlying skeletal structure.

D. Editing and Motion Synthesis

The framework supports two modes of interaction:

Direct Pose Manipulation: Users can directly modify SMAL parameters to animate the dog into any skeletal configuration (e.g., walking, sitting).
Text/Image-Driven Modification:
- Appearance: Uses Direct Gaussian Editor (DGE) with natural language prompts (e.g., "High quality dog") to refine textures and remove "plastic" artifacts.
- Motion: Uses FramePack (generative video) to interpolate between a start image and a target pose (defined by text or image), and ActionMesh to map the resulting video back onto the 3D Gaussian representation, enabling complex, naturalistic movements.

3. Key Contributions

Novel Hybrid Framework: The first method to effectively integrate 3D Gaussian Splatting with the SMAL parametric model for single-image animal reconstruction, balancing visual fidelity with anatomical correctness.
Multimodal Editing Pipeline: Introduces a workflow allowing users to refine surface textures (fur) and execute complex animations using natural language prompts, bypassing manual rigging.
Automated Motion Synthesis: Develops a pipeline that translates generative video priors into actionable mesh deformations, enabling static reconstructions to perform dynamic behaviors.

4. Results and Evaluation

Visual Quality: SMAL-pets outperforms state-of-the-art methods (like DogRecon, GART, AniAv.) in preserving fine details, particularly fur, and reducing the "plastic" look common in generative 3D models.
Quantitative Metrics:
- CLIP-Score: The method achieves higher CLIP scores (similarity between input image and rendered output) compared to baselines, indicating better semantic alignment.
- PSNR: Significant improvements in Peak Signal-to-Noise Ratio are observed when moving from Stage 1 to Stage 2 and the Final (DGE) stage.
Animation: The method successfully transfers motion sequences (e.g., walking, sitting) across different breeds due to the shared SMAL topology. It demonstrates robustness in adapting to "in-the-wild" poses.
Backbone Analysis: The framework is agnostic to the initial image-to-3D generator but performs best with TripoSG, which offers consistent textures and fewer topological artifacts compared to SAM3D or Trellis.

5. Significance and Limitations

Significance:
SMAL-pets represents a major step forward in democratizing 3D animal content creation. By automating the rigging and animation process through a hybrid parametric-generative approach, it provides a flexible tool for VR, gaming, and digital film production. It solves the "plastic look" problem and enables text-driven control over both appearance and behavior.

Limitations:

Dependency on Initial Generation: The quality of the final avatar is limited by the initial image-to-3D model. Topological errors (e.g., extra limbs, three ears) in the synthetic pseudo-ground truth are inherited.
Tracking Failures: In extreme camera rotations during the video-driven motion synthesis phase, the tracker (ActionMesh) may lose alignment, leading to incorrect mesh deformations.
Data Constraints: While it improves upon existing methods, the lack of large-scale real-world annotated animal datasets remains a fundamental bottleneck for training robust priors.