💻 computer science

From Blurry to Believable: Enhancing Low-quality Talking Heads with 3D Generative Priors

This paper introduces SuperHead, a novel framework that leverages dynamics-aware 3D inversion of pre-trained generative models to synthesize high-fidelity, animatable 3D talking head avatars with fine-grained details from low-resolution inputs, ensuring superior visual quality and temporal consistency compared to existing methods.

Original authors: Ding-Jiun Huang, Yuanhao Wang, Shao-Ji Yuan, Albert Mosella-Montoro, Francisco Vicente Carrasco, Cheng Zhang, Fernando De la Torre

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ding-Jiun Huang, Yuanhao Wang, Shao-Ji Yuan, Albert Mosella-Montoro, Francisco Vicente Carrasco, Cheng Zhang, Fernando De la Torre

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a low-quality, blurry 3D video game character's head. Maybe it was scanned from a cheap webcam or reconstructed from a grainy smartphone video. When you try to make this character talk, smile, or turn their head, the face looks like a melted wax figure: the skin is smudged, the teeth are invisible blobs, and the eyes lack detail.

SuperHead is a new tool designed to fix this mess. It takes that blurry, low-resolution 3D head and turns it into a crisp, high-definition, photorealistic avatar that can still move and talk naturally.

Here is how it works, using some simple analogies:

1. The Problem: The "Average" Blur

When you try to fix a blurry 3D head using old methods, it's like trying to guess the details of a face by looking at a dozen different people who are all slightly out of focus. The computer tries to find a "middle ground" between all the blurry images. The result? A face that looks like a smooth, featureless mask. It loses the unique details (like a specific scar or the shape of the nose) and looks fake.

2. The Secret Ingredient: The "Master Sculptor"

SuperHead uses a trick called 3D GAN Inversion. Think of a pre-trained 3D Generative AI (like GSGAN) as a Master Sculptor who has spent years studying millions of high-quality 3D heads. This sculptor knows exactly what a realistic nose, eyelash, or tooth looks like in 3D space.

Instead of trying to guess the details from the blurry input, SuperHead asks the Master Sculptor: "Show me a 3D head that looks like this blurry input, but make it perfect."

3. The Process: How SuperHead Fixes the Head

Step A: The Group Photo (Multi-View Inversion)
To make sure the 3D head looks real from every angle, SuperHead doesn't just look at one picture. It takes a "group photo" of the blurry head from many different angles (front, side, top).

The Analogy: Imagine trying to fix a statue by looking at it from one side. You might miss a crack on the back. SuperHead looks at the blurry head from all sides simultaneously and asks the Master Sculptor to create a perfect 3D model that matches all those views at once. This ensures the nose doesn't look weird when you turn the head.

Step B: The Skeleton Check (Rigging to FLAME)
The blurry input usually has a hidden "skeleton" (called a FLAME model) that controls how the face moves. However, the blurry skin might be attached to the wrong bones (e.g., the "teeth" might be stuck to the "lips").

The Analogy: Before painting the final details, SuperHead checks the skeleton. It adjusts the underlying wireframe so that the new, high-definition skin fits perfectly over the bones. This ensures that when the character smiles, the teeth actually show up in the right place.

Step C: The Motion Test (Dynamics-Aware Refinement)
This is the most important part. A static 3D head is easy to fix, but a talking head is hard. If you only fix the face when it's neutral, it might look weird when the character opens their mouth wide or raises an eyebrow.

The Analogy: Imagine a mannequin. If you dress it perfectly while it's standing still, the clothes might rip or look strange when the mannequin starts dancing. SuperHead tests the new high-definition head while it is "dancing" (making different expressions). It looks at the blurry input while the character is smiling, frowning, and talking, and it tweaks the 3D model to make sure the details (like the inside of the mouth or eyelids) stay realistic and don't glitch during movement.

4. The Result

The final output is a Super-Resolved 3D Head.

Before: A blurry, smudged blob that looks like a low-poly video game character from the 90s.
After: A crisp, detailed 3D head with visible pores, sharp teeth, and realistic hair, which can be animated to talk and emote without looking broken.

Why is this special?

Most previous tools tried to fix the video after it was made (like sharpening a blurry photo). SuperHead fixes the 3D model itself before it is even animated. It uses the "knowledge" of a Master Sculptor (the AI) to fill in the missing details, ensuring that the character looks real whether you are looking at them from the front, the side, or watching them make a funny face.

The paper claims this method creates better-looking avatars than current state-of-the-art tools, and it does so relatively quickly, making it possible to turn low-quality scans into high-quality digital humans for things like virtual reality and video games.

Technical Summary: SuperHead

Problem Statement

The creation of high-fidelity, animatable 3D talking head avatars is essential for immersive applications like VR, telepresence, and gaming. However, current reconstruction pipelines often rely on low-quality inputs (consumer-grade videos or images) characterized by low resolution, poor lighting, and motion blur. These inputs yield 3D avatars with blurry textures, missing fine details, and geometric artifacts.

While significant progress has been made in 2D image/video super-resolution (SR) and static 3D SR, these existing techniques are ill-equipped for dynamic 3D head avatars. Frame-by-frame 2D SR leads to temporal flickering, and video SR often fails to maintain cross-view consistency. Furthermore, standard 3D reconstruction methods (like 3D Gaussian Splatting) tend to produce "averaged" results when faced with inconsistent multi-view inputs, failing to recover high-frequency details. The core challenge addressed by this work is synthesizing high-quality geometry and textures for animatable 3D heads while ensuring multi-view consistency, temporal consistency during animation, and faithful identity preservation across diverse facial expressions.

Methodology: SuperHead

The authors propose SuperHead, a framework that directly upsamples low-resolution (LR) animatable 3D head avatars into high-fidelity 3D Gaussian Splatting (3DGS) models. The method leverages rich priors from pre-trained 3D generative models (specifically 3D GANs) via a novel dynamics-aware 3D inversion scheme.

The pipeline consists of three main stages:

1. Multi-View 3D GAN Inversion (Static Reconstruction)

The process begins by reconstructing a static 3D Gaussian head in a canonical space (neutral expression).

Input: A low-resolution animatable avatar driven by a parametric model (e.g., FLAME).
Sampling: The system samples $n$ multi-view renderings of the neutral expression from the LR model. These views are uniformly sampled on a sphere covering the frontal region.
Upscaling: These rendered images are upscaled to high resolution using an off-the-shelf 2D SR model.
Inversion: A latent code $w^*$ is optimized for a pre-trained 3D GAN (GSGAN) to minimize the reconstruction loss between the GAN-generated 3DGS renderings and the high-resolution anchor images.
Supervision: The optimization is jointly supervised by:
- Image Loss: Pixel-wise and perceptual loss (LPIPS) against the upscaled 2D images.
- Depth Loss: Consistency with depth maps derived from the underlying FLAME mesh (excluding hair regions, which FLAME does not model).
Two-Stage Optimization: Following recent inversion techniques, the method first optimizes the latent code $w$ , then performs a second-stage fine-tuning of the GAN generator parameters $g$ to capture finer details.

2. 3D Gaussian Rigging and Geometry Refinement

Once the static high-fidelity 3DGS head is generated, it must be bound to the underlying parametric mesh (FLAME) for animation.

Geometry Refinement: The initial LR mesh geometry often misaligns with the high-fidelity details generated by the GAN (e.g., teeth appearing on lips). The authors optimize the global shape parameters ( $\beta$ ) of the FLAME mesh to align 3D landmarks with 2D landmarks detected in the anchor images.
Rigging: The refined 3D Gaussians are bound to the surface of the corrected FLAME mesh. Each Gaussian is associated with its nearest mesh face, and its local parameters (position, rotation, scale) are transformed to global coordinates based on the mesh deformation.

3. Dynamics-Aware 3D Refinement

To ensure the avatar remains realistic under dynamic motion (diverse expressions and poses), the static inversion is insufficient.

Multi-Expression Sampling: The system augments the anchor image set with renderings of various facial expressions. These expressions are selected to ensure all facial regions (e.g., eyelids, inner mouth) are visible in at least one view and to cover a wide range of deformations.
Joint Optimization: The 3D GAN inversion is jointly optimized using this expanded set of multi-view, multi-expression images.
Deformation Handling: The system applies deformation fields ( $T_j$ ) to the canonical Gaussian head to match specific expression poses ( $\theta_j, \psi_j$ ). The total loss function combines the multi-view inversion loss and the dynamics-aware refinement loss, ensuring the generated avatar maintains identity and detail consistency even when the mesh deforms significantly.

Key Contributions

First Framework for Dynamic 3D Avatar SR: The paper presents the first method specifically designed to super-resolve low-resolution animatable 3D head avatars, addressing the unique challenges of temporal and cross-view consistency.
Dynamics-Aware 3D Inversion: The authors introduce a novel inversion scheme that couples 3D GAN priors with the underlying parametric mesh structure. By conditioning the inversion on multi-view and multi-expression renderings, the method ensures robustness against complex facial motions.
Geometry-Aware Rigging: The framework includes a specific step for refining the underlying mesh geometry before rigging, preventing misalignment between high-frequency GAN-generated details (like teeth) and the parametric model.
Efficiency: Despite leveraging generative priors, the method remains computationally lightweight compared to training from scratch, enabling fast inference.

Experimental Results

The method was evaluated on the NeRSemble (multi-view) and INSTA (monocular) datasets, comparing against state-of-the-art baselines including Video-based SR, SuperGaussian, and SR+GPAvatar.

Quantitative Performance: SuperHead outperformed all baselines in PSNR, SSIM, and LPIPS metrics across both datasets and settings (Novel View Synthesis and Self-Reenactment). For instance, on the NeRSemble Self-Reenactment task, SuperHead achieved a PSNR of 22.21 compared to the next best (Video-based SR) at 21.91.
Qualitative Quality: Visual results demonstrate that SuperHead synthesizes sharp textures and fine-grained details (e.g., facial hair, teeth, eyebrows) that are often blurry or missing in baseline methods. In some cases, the visual quality of the generated avatar approached or surpassed the pseudo ground-truth high-resolution avatars.
Consistency: The method successfully maintained consistency across novel camera poses and facial expressions, avoiding the flickering and artifacts common in frame-by-frame SR approaches.
Efficiency: SuperHead achieved competitive inference speeds (5 minutes per sequence), significantly faster than video-based SR methods (30 minutes) while delivering superior quality.

Significance and Claims

The paper claims that SuperHead represents a significant step forward in digital human creation by directly addressing the limitations of current 3D reconstruction pipelines when dealing with low-quality inputs. By leveraging pre-trained 3D generative priors, the method bypasses the need for expensive, specialized capture equipment, making high-fidelity 3D avatars more accessible.

The authors emphasize that their approach is the first to directly tackle the super-resolution of animatable 3D avatars, successfully balancing the generation of high-frequency details with the strict requirements of geometric and temporal consistency. They conclude that this work opens new possibilities for realistic digital humans in VR, telepresence, and entertainment, though they acknowledge limitations in handling complex hair dynamics and full 360-degree reconstruction.