💻 computer science

FiCA: Feed-forward instant Gaussian Codec Avatars from a Single Portrait Image

FiCA is a feed-forward pipeline that generates photorealistic, real-time drivable 3D Gaussian avatars from a single portrait image by combining vision foundation models, a diffusion-based mesh reconstruction network, and a refinement module, eliminating the need for person-specific test-time optimization.

Original authors: Kim Youwang, Zhengyu Yang, Liuhao Ge, Yu Rong, Timur Bagautdinov, Su Zhaoen, Nir Sopher, Jovan Popović, Teng Deng, Tae-Hyun Oh, Chen Cao

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kim Youwang, Zhengyu Yang, Liuhao Ge, Yu Rong, Timur Bagautdinov, Su Zhaoen, Nir Sopher, Jovan Popović, Teng Deng, Tae-Hyun Oh, Chen Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single, casual selfie of yourself. Now, imagine a magic machine that can take that one flat photo and instantly build a living, breathing, 3D version of your head that you can make smile, frown, or turn around in real-time. That is essentially what FiCA does.

Here is how the paper explains this technology, broken down into simple concepts and analogies:

The Big Problem: The "One-Photo" Puzzle

Creating a realistic 3D human head usually requires a massive studio with dozens of cameras, expensive lights, and hours of scanning. It's like trying to build a full 3D model of a house just by looking at one photo of the front door. You are missing all the information about the back, the inside, and the sides.

Previous methods tried to solve this by using "guessing games" (optimization) that took a long time to figure out the missing pieces, or they required you to record a video of yourself moving around first. FiCA wants to skip the long wait and the video recording. It wants to go from one photo to a 3D avatar in about 5 seconds.

The FiCA Solution: A Three-Step Assembly Line

The authors built a "feed-forward" system, which is like a factory assembly line where the product moves through three distinct stations without stopping to be re-evaluated.

Station 1: The "Detective" (Vision Foundation Models)

First, the system looks at your single photo and acts like a super-smart detective. It uses pre-trained "vision foundation models" (AI that has already learned what human faces look like) to extract clues.

What it does: It doesn't just see pixels; it guesses your skin texture, the shape of your nose, and even where your eyes are, even if parts of your face are hidden or in shadow.
The Analogy: Think of this as an artist looking at a blurry sketch and mentally filling in the missing lines to see the whole picture.

Station 2: The "Dreamer" (The Diffusion Model)

This is the core magic. The system takes the "clues" from Station 1 and feeds them into a Diffusion Model.

What it does: A diffusion model is like an artist who starts with a canvas full of static (noise) and slowly paints a clear image. Here, the AI starts with a blank, noisy 3D mesh and "dreams" up the complete texture and geometry of your face, including parts you can't see in the photo (like the inside of your mouth or the back of your head).
The Analogy: Imagine you give a sculptor a single photo of a face and a bag of clay. The sculptor uses their knowledge of how faces work to sculpt the entire head, not just the side facing the camera.

Station 3: The "Polisher" (Feed-Forward Refinement)

Sometimes, the "Dreamer" gets the general shape right but misses the tiny details, like the exact shade of your skin or a specific wrinkle.

What it does: A refinement network looks at the original photo and the newly created 3D model side-by-side. It acts like a photo editor, tweaking the 3D model to make it look exactly like the person in the photo.
The Analogy: This is like a tailor taking a suit that fits well but needs a few extra stitches to make it look perfect. Crucially, this happens instantly; the system doesn't stop to "think" or "optimize" for hours.

The Final Product: The "3D Gaussian" Avatar

Once the 3D mesh is perfect, the system converts it into a 3D Gaussian Avatar.

What is that? Instead of building the face out of solid triangles (like a video game character), it builds the face out of millions of tiny, fuzzy, colored dots (Gaussians).
Why is it cool? These dots are incredibly efficient. They allow the avatar to look photorealistic and, most importantly, drivable. You can feed it new expressions (like a smile or a wink), and the dots rearrange themselves instantly to show that expression in real-time.

What the Paper Claims (and What It Doesn't)

Speed: It generates the avatar in about 5 seconds.
Input: It only needs one single portrait image. No videos, no 3D scanners.
Quality: The paper claims these avatars look more realistic and preserve your identity better than other recent methods that try to do the same thing.
No "Fine-Tuning": Unlike older methods that had to spend time "learning" your specific face after the initial scan, FiCA does everything in one go.

What the paper does NOT claim:
The paper does not claim this works for full-body avatars (only heads), nor does it claim it works perfectly with extreme lighting or motion blur (it admits these are current limitations). It also doesn't claim to be a medical tool or a diagnostic device; it is strictly for creating digital entertainment avatars.

In short, FiCA is a fast, one-click factory that turns a single selfie into a high-quality, animated 3D character, skipping the long, expensive, and complicated steps usually required to do so.

Technical Summary: FiCA - Feed-forward Instant Gaussian Codec Avatars

Problem Statement

Creating photorealistic, drivable 3D human avatars from a single portrait image is a highly ill-posed problem due to the scarcity of visual information required to infer 3D geometry and appearance. Existing solutions face a trade-off between data abundance and computational burden:

High-fidelity methods rely on dense, multi-view capture systems and offline optimization, which are time-consuming and resource-intensive.
Recent single-image methods often require cumbersome test-time optimization (e.g., GAN inversion or fine-tuning) to align a universal prior with a specific identity, limiting accessibility and speed.
Current feed-forward approaches often suffer from visual artifacts, lack of identity preservation, or limited expressivity when animating avatars.

The goal is to develop a system that generates authentic, real-time drivable avatars from a single casual portrait image in seconds, without person-specific test-time optimization.

Methodology: FiCA Pipeline

FiCA (Feed-forward Instant Gaussian Codec Avatar) is a modular, feed-forward pipeline that converts a single portrait image into a set of 3D Gaussians capable of real-time animation. The process consists of three primary stages:

1. Conditioning via Vision Foundation Models

To address the lack of 3D information in a single image, the system leverages human-centric vision foundation models to extract rich features:

CLIP Embedding: Extracts high-level visual semantic information ( $f_{CLIP}$ ) from the input image.
Sapiens Models: Fine-tuned versions of Sapiens predict per-pixel UV coordinates, mesh vertex coordinates, and surface normals.
Partial Observation: These predictions are unwrapped into four partial UV maps: RGB texture, visibility mask, normal map, and vertex coordinates ( $UV_{partial}$ ). These serve as the conditioning signals for the generative model.

2. Diffusion-Based Texture and Geometry Generation

A latent diffusion model (based on the DiT architecture) generates complete UV texture and geometry maps from the partial observations.

Input: The model takes the CLIP embedding, partial UV maps, a domain switcher (to distinguish between texture and geometry tasks), diffusion timesteps, and random noise.
Training Objective: The model is trained using a conditional flow matching objective to learn a generative mapping from imperfect partial observations to complete, authentic 3D mesh representations. Unlike inpainting models, this diffusion model is trained to "imagine" complete geometry and texture (including self-occluded regions like the mouth interior) while preserving identity.
Output: A complete canonical textured mesh ( $\tilde{T}, \tilde{G}$ ).

3. Feed-forward Mesh Refinement

To enhance fidelity and identity preservation without test-time optimization, a learned feed-forward refinement network ( $R_\phi$ ) is introduced.

Mechanism: The network takes the initial diffusion-generated texture and geometry, along with dense visual features extracted from both the original input image and the rendered diffusion mesh (via Sapiens ViT).
Architecture: A U-Net with cross-attention layers allows the network to reference image features to refine the UV maps.
Loss Function: Training utilizes a weighted sum of photometric loss, 2D keypoint loss, mask loss, and regularization terms to ensure the refined mesh aligns with the input image while preventing overfitting to visible parts.
Result: A refined canonical mesh ( $T, G$ ) with corrected skin tones and geometric details.

4. Decoding to 3D Gaussians via Universal Prior Model (UPM)

The final step converts the refined canonical mesh into a real-time drivable avatar using a Universal Prior Model.

Identity Encoder ( $E_{\psi}^{id}$ ): A CNN-based hypernetwork encodes the refined UV texture and geometry into identity-specific bias maps ( $\Psi_{id}$ ).
3D Gaussian Decoder ( $D_{\psi}^{dec}$ ): This decoder takes the bias maps and driving signals (expression codes, view vectors, gaze vectors) to generate a set of 3D Gaussians.
Output: A "Codec Avatar" represented by 3D Gaussians that can be animated in real-time with arbitrary expressions.

Key Contributions

FiCA System: A fully feed-forward pipeline that generates authentic, drivable Gaussian Codec Avatars from a single casual portrait image in approximately 5 seconds, eliminating the need for person-specific test-time optimization.
Diffusion Model for Avatar Generation: A novel diffusion model that learns a generative mapping from partial visual observations (derived from foundation models) to complete 3D mesh texture and geometry, effectively handling self-occluded regions.
Feed-forward Refinement Network: A learned module that enhances avatar fidelity and identity preservation by correcting subtle details (e.g., skin tone, cloth) through cross-attention with image features, avoiding the computational cost of iterative optimization.

Experimental Results

The authors evaluated FiCA on unseen identities with diverse attributes (race, gender, age, hairstyles) and novel expressions.

Qualitative Performance: FiCA produces avatars with high visual fidelity, preserving input details (e.g., tattoos, accessories) and generating plausible unseen regions (e.g., mouth interior). It outperforms PanoHead in view consistency and artifact reduction (e.g., no "ghost faces" in side views).
Quantitative Performance: Compared to recent competing methods (GPAvatar, VOODOO 3D, Portrait4D, GAGAvatar), FiCA achieves superior photometric quality (higher PSNR and SSIM) and significantly better identity preservation (higher ID-CSIM scores).
- PSNR: FiCA (Mesh) achieved 24.281 vs. the next best (GAGAvatar) at 22.157.
- ID-CSIM: FiCA achieved 0.5867 vs. the next best at 0.4339.
Efficiency: The pipeline generates a textured mesh in ~5 seconds, compared to ~80 seconds for PanoHead's per-subject GAN inversion.

Significance and Claims

The paper claims that FiCA represents a significant step toward the democratization of high-fidelity avatar creation. By combining human-centric foundation models with generative diffusion and a feed-forward refinement strategy, the system bridges the gap between the limited information of a single image and the need for complete, animatable 3D geometry.

The authors emphasize that their approach:

Removes the barrier of complex capture setups and hours-long optimization processes.
Enables real-time animation with arbitrary expressions, unlike static reconstruction methods.
Demonstrates that a universal prior, when conditioned correctly via diffusion and refinement, can generalize to diverse identities without per-user fine-tuning.

The paper also notes limitations, such as vulnerability to extreme lighting or motion blur in input images, and suggests future work in light normalization, blur correction, and the joint generation of accessories like glasses.