FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision

1. Problem Statement

Creating high-quality, animatable 3D head avatars from a single image is a significant challenge in computer vision due to two main constraints:

1. Under-constrained Reconstruction: A single 2D image lacks information about unobserved regions (e.g., the back of the head), making accurate 3D reconstruction difficult.
2. View-Expression Entanglement: Existing methods trained on monocular (single-view) video datasets often fail to generalize to novel viewpoints. The root cause identified by the authors is the entanglement between the driving signal (facial expression) and the target viewpoint. In monocular self-reenactment, the expression and viewpoint are perfectly correlated in the training data. Consequently, models learn to "guess" the viewpoint based on the expression, resulting in incomplete 3D heads that collapse or produce artifacts when rendered from new angles.

While multi-view datasets solve the completeness issue, they are scarce and lack the broad identity coverage of monocular datasets. Existing approaches often rely heavily on 3D Morphable Models (3DMMs) like FLAME to enforce geometry, which limits expressiveness and flexibility.

2. Methodology: FlexAvatar

FlexAvatar is an encoder-decoder framework designed to create complete 3D head avatars represented by 3D Gaussians. The core innovation lies in its ability to unify training on both monocular and multi-view data while preventing the aforementioned entanglement.

A. Architecture Overview

The pipeline consists of three main stages:

1. Encoder ($E$): Takes a single input image $I$ and projects it onto a 2D latent avatar code $A$ living in the UV space of a template head mesh. It uses a pre-trained DINOv2 backbone and a shallow ViT to extract features, which are then mapped to the UV space via cross-attention.
2. Decoder ($D$): Takes the avatar code $A$ and an expression code $z_{exp}$ (e.g., FLAME coefficients) to generate articulated 3D Gaussians.
   • Expression Modeling: Uses cross-attention to integrate expression codes directly, avoiding rigid 3DMM constraints.
   • Upsampling: Employs a novel StyleGAN-PixelShuffle upsampler. This combines PixelShuffle operations with StyleGAN2-style CNN blocks to upsample the latent features from a low-resolution map to a high-resolution feature map, enabling the generation of ~58k 3D Gaussians with fine details (e.g., eyes, mouth interior).
3. Renderer ($R$): Uses a tile-based differentiable rasterizer (from 3D Gaussian Splatting) to render the Gaussians from arbitrary viewpoints $\pi$.

B. Key Innovation: Learnable Bias Sinks

To address the entanglement of driving signals and viewpoints, the authors introduce Bias Sinks.

• Mechanism: Two learnable tokens, $z_{2D}$ and $z_{3D}$, are concatenated to the expression sequence before decoding.
  • $z_{2D}$ is used for samples from monocular datasets.
  • $z_{3D}$ is used for samples from multi-view datasets.
• Training Behavior: The model learns to associate $z_{2D}$ with incomplete, viewpoint-dependent reconstructions (absorbing the bias of monocular data) and $z_{3D}$ with complete, viewpoint-independent reconstructions.
• Inference Strategy: During inference, only the $z_{3D}$ token is used. This forces the model to utilize the "complete 3D" behavior learned from multi-view data while still benefiting from the strong generalization capabilities (identity coverage) learned from the vast monocular datasets. This effectively disentangles the expression from the viewpoint.

C. Fitting and Latent Space

The training process yields a smooth latent space. This allows for:

• Identity Interpolation: Blending avatar codes between two different people.
• Few-Shot/Monocular Fitting: If multiple images or a video of a person are available, the initial avatar code can be optimized (fitted) against all observations to refine the identity, while the decoder weights remain frozen to prevent overfitting.

3. Key Contributions

1. Novel Pipeline: An efficient method to generate high-quality, complete 3D head avatars from a single image within minutes.
2. Bias Sinks: A learnable token mechanism that successfully combines the generalization of monocular data with the geometric completeness of multi-view supervision, solving the view-expression entanglement problem without requiring restrictive 3DMM priors.
3. High-Quality Upsampler: A hybrid StyleGAN-PixelShuffle architecture that significantly improves visual fidelity, particularly in sensitive facial regions.
4. Flexible Application: The method supports single-image creation, few-shot creation, and monocular video avatar generation, all within a unified framework.

4. Experimental Results

The authors evaluated FlexAvatar on four tasks across multiple datasets (VFHQ, Ava256, NeRSemble, Cafca):

• 3D Portrait Animation: Outperformed state-of-the-art (SOTA) methods (e.g., GAGAvatar, LAM, Portrait4D-v2) on the VFHQ dataset in PSNR, SSIM, and LPIPS for both self- and cross-reenactment.
• Single-Image Avatar Creation: On the Ava256 dataset, FlexAvatar significantly outperformed competitors in generating complete 3D heads with realistic novel-view rendering, whereas other methods produced incomplete or distorted heads.
• Few-Shot Creation: With only 4 input images, FlexAvatar achieved superior identity preservation (CSIM) and sharpness compared to Avat3r and InvertAvatar.
• Monocular Video Creation: On the NeRSemble benchmark, FlexAvatar surpassed SOTA methods (including CAP4D and RGBAvatar) in rendering novel views and expressions, achieving higher sharpness (LPIPS) and identity preservation with significantly faster fitting times (~10 mins vs. hours).

Ablation Studies confirmed that:

• Training on monocular data alone leads to incomplete heads.
• Training on multi-view data alone leads to poor generalization.
• Simply mixing data without bias sinks fails to resolve entanglement.
• The StyleGAN upsampler is crucial for visual quality.

5. Significance

FlexAvatar represents a significant leap forward in 3D avatar generation by demonstrating that partial supervision (monocular data) can be effectively combined with sparse 3D supervision (multi-view data) to achieve both scalability and geometric completeness.

• Practical Impact: It enables the creation of high-fidelity, animatable avatars from a single smartphone photo or a short video clip, removing the need for expensive multi-camera capture rigs.
• Theoretical Insight: The concept of "Bias Sinks" offers a new paradigm for handling heterogeneous datasets in 3D reconstruction, potentially applicable to other domains where 3D data is scarce but 2D data is abundant.
• Expressiveness: By moving away from rigid 3DMM constraints, the method allows for more natural and flexible facial animations directly learned from data.