Instant Expressive Gaussian Head Avatars at Over 100 FPS

Technical Summary: Instant Expressive Gaussian Head Avatars at Over 100 FPS

Problem Statement
The paper addresses the "portrait animation trilemma," a challenge where existing methods fail to simultaneously achieve high speed, 3D consistency, and expressive detail in facial animation.

• 2D Methods: Approaches based on video diffusion models or GANs offer high expressiveness but suffer from 3D inconsistency (e.g., identity drift, hallucinated features) and slow inference speeds due to expensive denoising processes, making them unsuitable for real-time applications like telepresence or digital twins.
• 3D Methods: Methods utilizing 3D representations (NeRFs, 3D Gaussians) ensure 3D consistency and faster speeds but often rely on parametric models (3DMMs) that limit expressiveness, or use computationally expensive global fusion mechanisms (e.g., dense attention layers) to combine structure and motion, resulting in inferior expression details and lower frame rates.

Methodology
The authors propose a feed-forward pipeline that instantly converts a single "in-the-wild" image into a 3D-consistent, animatable avatar using 3D Gaussian Splatting. The core components are:

1. 3D Gaussian Representation & Encoder:

   • The system adapts a state-of-the-art 2D-to-3D lifting encoder (originally designed for neural triplanes) to encode a single input image into a set of 3D Gaussians.
   • Instead of volumetric fields, the method samples 3D Gaussians from encoded triplanes. A lightweight MLP decoder maps feature vectors to Gaussian attributes (position, scaling, rotation, opacity, color).
   • Sampling is performed via a two-pass importance sampling strategy on rays, generating approximately 200K Gaussians sufficient for 512×512 resolution rendering.

2. Feature-Space Deformation for Animation:

   • Unlike prior works that deform Gaussians directly in 3D spatial space (updating position/scale vectors), this method deforms Gaussians in high-dimensional feature space.
   • Motion Basis Vectors: Each sampled Gaussian location is associated with a learned "motion basis vector" ($m_i$) that encodes spatially varying deformation capabilities (analogous to personalized blendshapes).
   • Local Fusion: A motion encoder extracts 1D motion coefficients from both the source and driving images. A lightweight motion decoder (using AdaLN) predicts a residual feature vector ($\delta f_i$) for each Gaussian based on the motion coefficients and the specific motion basis vector.
   • The final Gaussian attributes are decoded from the sum of the original feature vector and the predicted residual ($f_i + \delta f_i$). This local fusion strategy avoids computationally expensive global attention mechanisms, enabling high-speed inference.

3. Training Strategy with Diffusion Guidance:

   • To overcome the lack of real-world 3D-consistent expression data, the authors construct a synthetic dataset using a state-of-the-art 2D diffusion model (X-NeMo).
   • Frontalization: To prevent hallucinations and viewpoint inconsistencies common in diffusion models, source and driving images are first frontalized. Expressions are synthesized only on these frontal views.
   • Multi-view Supervision: A pre-trained 2D-to-3D network generates multi-view images from the frontalized synthetic portraits during training, enforcing cross-view geometric consistency.
   • The model is trained via a self-reenactment objective, minimizing reconstruction loss, perceptual loss, identity loss, and adversarial loss between the rendered output and the target images.

Key Contributions

• Efficient Animation Representation: A novel representation that decouples the 3D structure from the animation representation. By deforming features in a high-dimensional space using learned motion basis vectors, the method captures asymmetric expressions, wrinkles, and shadow changes more effectively than spatial deformation methods.
• Lightweight Local Fusion: The design replaces heavy global fusion mechanisms with an efficient local fusion strategy, achieving high expressivity without the computational cost of repeated attention layers.
• Synthetic Training Pipeline: A practical strategy to leverage 2D diffusion models for 3D animation training while mitigating their inherent 3D inconsistencies through frontalization and multi-view supervision.
• Performance: The method is the first to simultaneously achieve top-tier 3D consistency, detailed expression transfer, and real-time inference speeds.

Results

• Speed: The method runs at 107.31 FPS for animation and pose control on an NVIDIA 6000 Ada GPU. This represents a 3–4 order of magnitude speedup compared to diffusion-based methods and significantly outperforms existing 3D-aware methods (e.g., Portrait4D-v2 at ~14.5 FPS).
• Quality: In quantitative benchmarks (VOODOO-XP test set), the method achieves the best 3D consistency (MEt3R) and 3D shape accuracy (IoU) among all baselines. It also achieves state-of-the-art expression transfer accuracy (AED) and emotion similarity (EMO), surpassing even the diffusion-based teacher model (X-NeMo) in cross-reenactment tasks.
• Visual Fidelity: Qualitative results show faithful transfer of extreme expressions, including nasal wrinkles and eye movements, without the identity distortion or hallucinated features (e.g., fake mustaches) seen in other methods.

Significance
The paper claims to solve the portrait animation trilemma, enabling the creation of 4D digital twins from a single image with real-time performance. By achieving high expressiveness without sacrificing 3D consistency or speed, the method facilitates real-world applications such as AR/VR, video conferencing, and agentic AI, where controllability and real-time responsiveness are critical. The authors note that while the method is currently image-driven, the architecture supports encoding other conditions (audio, text) into motion coefficients for future expansion.