Snapmoji: Instant Generation of Animatable Dual-Stylized Avatars

Snapmoji is an innovative avatar generation system that instantly creates animatable 3D avatars from user selfies by employing Gaussian Domain Adaptation for identity preservation and a novel dual-stylization process to apply secondary styles, all while overcoming the limitations of existing platforms through efficient mobile rendering.

The Gist

Imagine you have a favorite cartoon character that looks exactly like you (like a Bitmoji). Now, imagine you could instantly turn that character into a LEGO figure, a skeleton, a yarn doll, or a toy, all while keeping your face recognizable. And the best part? You can make that toy skeleton blink, smile, and talk in real-time on your phone, all in less than a second.

That is Snapmoji.

Here is a simple breakdown of how this magic trick works, using everyday analogies:

1. The Problem: The "Toy Store" Limitation

Right now, apps like Snapchat or Apple let you build an avatar by picking from a pre-made menu: "Do you want blue eyes? A red hat? A beard?"

• The Analogy: It's like going to a toy store where you can only buy the toys that are already on the shelf. If you want a "Spiderman" version of your avatar, the store doesn't have it. The artists have to hand-sculpt a new Spiderman toy from scratch, which takes weeks.
• The Goal: Snapmoji wants to be a 3D printer that can instantly print any style you can imagine, without needing a factory full of artists.

2. The Solution: The Two-Step Magic Trick

Snapmoji does this in two lightning-fast steps:

Step 1: The "Identity Translator" (Gaussian Domain Adaptation)

First, the system takes your selfie and turns it into a standard cartoon avatar (the "Base Avatar").

• The Analogy: Think of this as a universal translator. You speak "Real Human," and it instantly translates you into "Cartoon Language."
• The Secret Sauce: They use a technique called Gaussian Domain Adaptation (GDA). Imagine a master artist who has studied millions of 3D objects (like a library of every toy ever made). Instead of just copying your face, this artist understands the structure of 3D space. This allows them to turn your photo into a cartoon in 0.08 seconds (faster than a blink) while keeping your unique features (like your specific eye color or glasses) perfectly intact.

Step 2: The "Style Chameleon" (Dual-Stylization)

Once you have your base cartoon, you type in a prompt like "Make me a Yoda" or "Make me a LEGO minifigure."

• The Analogy: This is like putting a magic filter on your photo, but instead of just changing colors, it completely rebuilds the texture and shape.
• How it works: It uses a powerful AI (a diffusion model) that acts like a creative director. It looks at your base cartoon and says, "Okay, let's turn the skin into plastic bricks for LEGO," or "Let's turn the hair into yarn." It does this in less than a second, ensuring you still look like you, just in a different costume.

3. Bringing It to Life: The "Puppet Master"

So far, we have a static picture. How do we make it move?

• The Old Way: Most 3D avatars rely on complex math models designed for real human faces. If you try to put a cartoon head with giant eyes on that model, it breaks or looks weird.
• The Snapmoji Way: They use a hybrid approach. They combine 3DMM (for the realistic parts of the face) with FACS (a system used by animators to control cartoon features like "squint eyes" or "open mouth").
• The Result: It's like having a puppet master. When you smile on your phone camera, the system instantly maps your smile to the cartoon avatar. Because the math is so efficient, it runs smoothly on your phone at 30–40 frames per second (like a smooth video game), even in Augmented Reality (AR).

Why is this a big deal?

1. Speed: It used to take artists days to make a new avatar style. Snapmoji does it in 0.9 seconds.
2. Creativity: You aren't limited to a menu. You can be a "Cyberpunk Cat" or a "Medieval Knight" instantly.
3. Real-time: You can use it in video calls or AR filters right now, without lagging your phone.

In a Nutshell

Snapmoji is like a magic mirror. You look into it, and it doesn't just show you a reflection; it shows you a LEGO version of yourself that can dance, talk, and react to your real-life expressions, all generated instantly by AI. It bridges the gap between the boring "menu of options" we have today and a world where your digital self can be anything you can imagine.

Technical Summary

Here is a detailed technical summary of the paper "Snapmoji: Instant Generation of Animatable Dual-Stylized Avatars."

1. Problem Statement

Existing avatar systems (e.g., Snapchat Bitmojis, Apple Memoji) face three critical limitations:

1. Limited Assets: They rely on a finite set of predefined 3D assets. Creating new styles (e.g., turning a user's avatar into a "skeleton" or "LEGO" character) requires manual artist intervention to build new 3D models, which is slow and unsustainable.
2. Tedious Customization: Users must manually select traits from a list rather than generating entirely new visual concepts via text prompts.
3. Inefficiency: Current research methods for generating 3D avatars (e.g., StyleAvatar3D, DATID-3D) are often computationally expensive, slow to render, and lack real-time animatability on mobile devices, making them unsuitable for Augmented Reality (AR) applications.

The paper addresses the need for Dual-Stylization: the ability to instantly generate a primary avatar (e.g., standard Bitmoji style) from a selfie, and then re-style it into a secondary, user-specified theme (e.g., "Yoda," "Toy," "Yarn") while preserving the user's identity, all within a production-ready, real-time pipeline.

2. Methodology

The Snapmoji system employs a two-stage pipeline designed for speed, identity preservation, and mobile efficiency. The core representation uses 3D Gaussian Splatting (3DGS).

Stage 1: Image-to-2D Avatar Generation

This stage converts a realistic selfie into a stylized 2D avatar in two steps:

• Gaussian Domain Adaptation (GDA):
  • Goal: Map a realistic face to a primary avatar style (e.g., Bitmoji) without paired training data.
  • Technique: The authors leverage Large Multi-view Gaussian (LGM) models pre-trained on the Objaverse dataset. They fine-tune the U-Net component of LGM to translate real faces into the target avatar domain.
  • Mechanism: The network takes a reference image ($I_{ref}$) and outputs pixel-aligned 3D Gaussian parameters (scaling, position, rotation, color, opacity). These are rendered to produce the primary 2D avatar.
  • Data Synthesis: Since paired real-face/Bitmoji data is scarce, they use GAN inversion to synthesize 13,000 pairs by mapping Bitmojis into a source GAN's latent space to generate corresponding realistic faces.
• Dual-Stylization (Diffusion):
  • Goal: Apply a secondary style (e.g., "LEGO") based on a text prompt while preserving the primary structure and identity.
  • Technique: A Stable Diffusion pipeline is used with three specific conditioning mechanisms:
    1. SDEdit: Starts denoising from a noisy version of the GDA output to preserve coarse structure.
    2. ControlNet: Uses Canny edges of the GDA output to maintain the avatar's geometry.
    3. IP Adapter: Uses the original user selfie to condition the generation on facial similarity embeddings, ensuring identity preservation.
  • Speed: The process uses only 10 denoising steps, taking <1 second.

Stage 2: 2D-to-3D Lifting and Animation

• 3D Generation: The dual-stylized 2D avatar is lifted to 3D using an Asymmetric U-Net.
  • The network takes the unposed 2D avatar and driving features.
  • It outputs a set of 3D Gaussians representing the avatar in a rest pose.
  • Loss Functions: Includes rendering loss (RGB + Alpha), perceptual loss (LPIPS), normal consistency loss (to reduce noise), and depth distortion loss (to cluster Gaussians along camera rays).
• Expression Animation:
  • Hybrid Encoding: Instead of relying solely on 3D Morphable Models (3DMM), which fail on cartoon geometries, the system combines 3DMM features (for realistic base geometry) with FACS (Facial Action Coding System) blendshape weights (for cartoon-specific exaggerations like eye size or mouth shape).
  • Driving Signal: Expression codes are concatenated with identity features (from DINOv2) and positional maps.
  • Real-time Rendering: On mobile devices, a face tracker (e.g., Mediapipe) predicts blendshape weights. The system animates the avatar via linear interpolation between the rest pose and target expression Gaussians. This is rendered in WebGL at 30–40 FPS.

3. Key Contributions

1. Dual-Stylization Concept: Introduced a framework to generate a base avatar and instantly re-style it into diverse themes (e.g., "Toy," "Skeleton") without manual 3D asset creation.
2. Gaussian Domain Adaptation (GDA): A novel domain translation method that repurposes LGMs (pre-trained on Objaverse) to map real images to predefined avatar styles, achieving high fidelity and speed.
3. Efficient Mobile Pipeline: Developed a full-stack solution (including a JavaScript framework) that generates animatable 3D avatars in 0.9 seconds and renders them at 30–40 FPS on mobile devices, enabling real-time AR puppeting.
4. Hybrid Animation Control: Successfully integrated 3DMM and FACS blendshapes to animate cartoon avatars, overcoming the geometric limitations of standard 3DMMs.

4. Experimental Results

The system was evaluated against state-of-the-art methods (GAN Inversion, Diffusion, EG3D, LGM, DATID-3D, TextToon).

• Image-to-2D Quality (Table 2): GDA significantly outperformed GAN Inversion and Diffusion baselines in FID (72.94 vs >93), KID, and Identity Preservation (0.25 vs <0.20). Crucially, it was 100x faster than diffusion (0.08s vs 3.54s) and 1000x faster than GAN inversion.
• 2D-to-3D Generation (Table 3): Snapmoji achieved superior PSNR (18.73) and SSIM (0.81) compared to EG3D and LGM, with a generation speed of 0.091s (vs 2.82s for LGM and 95s for EG3D).
• User Study (Table 4): In a study with 27 participants, 92% preferred Snapmoji's identity preservation over DATID-3D, and 96% preferred its style diversity and aesthetic fit for a Bitmoji-style game.
• Mobile Performance (Table 5): The system runs at 30–40 FPS on an iPhone 13 Pro and 90–100 FPS on a laptop, whereas competitors like TextToon struggle to reach 15–18 FPS on similar hardware due to heavy 3DMM dependencies.

5. Significance

Snapmoji bridges the gap between academic research and production-grade avatar systems.

• Scalability: By automating the creation of new styles via text prompts, it eliminates the need for artists to manually model thousands of new assets for marketing campaigns or events.
• Real-Time AR: It is the first system to combine high-fidelity dual-stylization with real-time mobile rendering, enabling dynamic, personalized AR filters and social media effects.
• Generalizability: The pipeline is not limited to Bitmojis; the authors demonstrate its potential for other avatar ecosystems and even non-human domains (e.g., cats), suggesting a broad applicability in gaming, virtual meetings, and education.

In summary, Snapmoji delivers a 0.9-second selfie-to-animatable-avatar conversion that preserves identity, supports infinite stylistic variations, and runs efficiently on mobile hardware, solving the bottleneck of manual asset creation in the avatar industry.