💻 computer science

GeoFace: Consistent Multi-View Face Generation with Geometry-Constrained Diffusion

GeoFace is a geometry-constrained dual-stream diffusion framework that generates consistent multi-view face images by jointly synthesizing RGB and 3D geometry through a shared attention mechanism guided by a geometry-alignment loss, thereby overcoming the cross-view inconsistency issues prevalent in existing methods.

Original authors: Yeji Choi, Jinhyeok Choi, Jaewon Min, Minkyung Kwon, Jin Hyeon Kim, Seungryong Kim

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yeji Choi, Jinhyeok Choi, Jaewon Min, Minkyung Kwon, Jin Hyeon Kim, Seungryong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single photograph of a friend's face. Now, imagine you want to see what that face looks like from the side, from above, or even from behind. This is a classic puzzle for computers: a single photo is flat, but a face is 3D. If you try to guess what the back of the head looks like just by looking at the front, you might end up with a weird, distorted result where the nose disappears or the jawline twists into something impossible.

GeoFace is a new computer program designed to solve this puzzle. Think of it as a "3D sculptor" that doesn't just paint a picture; it actually builds a consistent 3D model in its mind while it paints.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Two-Headed" Monster

Previous AI programs that tried to do this were like artists who only cared about the paint. They could make a beautiful side view, but they didn't have a strict rulebook for how the 3D shape should hold together.

The Result: If you asked them to turn the head, the nose might suddenly shift to the left ear, or the chin might vanish. They were "photorealistic" but geometrically broken. It's like drawing a face on a balloon; if you stretch the balloon, the features get warped and inconsistent.

2. The Solution: A Dual-Stream Team

GeoFace changes the game by using a two-person team working together in real-time:

The Painter (Appearance Stream): This part focuses on making the image look realistic—skin texture, lighting, and colors.
The Architect (Geometry Stream): This part focuses strictly on the 3D shape. It doesn't care about color; it cares about where the nose, jaw, and forehead actually sit in 3D space.

Instead of working separately, these two "streams" talk to each other constantly. The Architect tells the Painter, "Hey, the nose is here, so you can't paint an eye there." The Painter tells the Architect, "The lighting suggests the cheek is curved this way, so adjust the shape."

3. The Secret Sauce: The "Universal Blueprint"

To make sure they agree, GeoFace uses a special tool called a Canonical UV Position Map.

The Analogy: Imagine a standard, flat map of the world (like a Mercator projection). No matter where you are on Earth, "London" is always in the same spot on that map.
How it works: GeoFace creates a similar "flat map" for the face, based on a standard 3D face model (called FLAME). Every single view (front, side, back) is forced to align with this same flat map.
The Benefit: Because every view is anchored to this single, unchanging blueprint, the AI can't cheat. It can't make the nose disappear in the side view because the blueprint says the nose must be there. This ensures that the face looks like the same person from every angle.

4. The "Teacher" (The Alignment Loss)

The paper mentions a special "loss function" (a way to measure mistakes). Think of this as a strict teacher grading the team.

The teacher checks the conversation between the Painter and the Architect.
If the Painter is looking at the nose in the front view, the teacher checks: "Is the Architect looking at the nose in the 3D map?"
If they are looking at different things, the teacher gives them a "failing grade" and forces them to correct their focus. This ensures the 3D shape and the 2D image are perfectly locked together.

5. The Results: What Did They Find?

The researchers tested GeoFace on datasets of people with many different faces and angles.

Better Consistency: Unlike other methods that produced "glitchy" faces when the angle changed, GeoFace kept the face looking solid and consistent. The nose stayed on the nose, and the jawline stayed on the jaw.
Better 3D Reconstruction: Because GeoFace generates a perfect 3D shape along with the images, it makes it much easier for other programs to build a real 3D model of the face later. It's like giving a builder a perfect set of blueprints instead of just a pile of bricks.

Summary

GeoFace is like giving an AI a 3D skeleton to hold onto while it paints. By forcing the AI to build a consistent 3D shape at the same time it paints the picture, it prevents the face from warping or breaking when you look at it from different angles. It turns a flat photo into a reliable, rotatable 3D character.

Technical Summary: GeoFace

Problem Statement

Generating photorealistic multi-view facial images from a single input is a critical challenge for 3D reconstruction, digital avatars, and immersive content creation. While recent diffusion-based multi-view models achieve high visual fidelity at the per-view level, they often lack explicit mechanisms to enforce a shared 3D structure across different viewpoints. Consequently, these models frequently produce geometrically inconsistent results, particularly under large pose variations, leading to artifacts such as identity degradation and structural misalignment (e.g., inconsistent nose or jawline shapes). Existing approaches relying on parametric models (like 3DMMs or FLAME) fitted to single images are inherently ill-posed, often converging to statistically averaged structures rather than subject-specific geometry. Similarly, appearance-driven diffusion methods that incorporate geometric cues (like normal maps) still rely on view-specific signals rather than a unified, shared 3D representation, leaving global geometric alignment unresolved.

Methodology

GeoFace addresses these limitations by proposing a unified dual-stream diffusion framework that jointly generates multi-view RGB images and 3D face geometry from a single reference image.

1. Dual-Stream Architecture

GeoFace extends a multi-view diffusion backbone (based on CAT3D) to operate two parallel streams within a single U-Net:

Appearance Stream: Denoises target view latents conditioned on Plücker ray embeddings to generate $N-1$ novel view RGB images.
Geometry Stream: Denoises a shared geometry latent representing a canonical UV position map. This map encodes dense 3D surface coordinates on a FLAME mesh in a view-invariant parameterization.

Both streams interact through shared 3D attention layers, allowing the appearance and geometry streams to mutually constrain each other during the denoising process. This integration treats geometry not merely as an auxiliary signal but as an intrinsic representation shared across all views.

2. Geometry Representation and Conditioning

Canonical UV Position Map: The 3D geometry is represented as an image-like $H \times W \times 3$ grid where each texel stores the 3D canonical coordinate of the corresponding point on the FLAME mesh. This representation is topology-aware and view-invariant, making it directly compatible with 2D convolutional architectures without modification.
Learnable Camera Token: Since the canonical UV map does not correspond to a specific camera viewpoint, standard Plücker ray embeddings are replaced with a learnable token ( $e_{geo}$ ). This token is optimized end-to-end as an implicit positional embedding for canonical geometry, avoiding spurious viewpoint assumptions.

3. Geometry-Guided Attention Alignment Loss

To explicitly enforce 3D-consistent cross-modal attention between the geometry and appearance streams, the authors introduce a geometry-guided attention alignment loss.

This loss supervises the cross-attention between geometry tokens (UV map) and appearance tokens (face images) using 3D-consistent correspondences derived from the FLAME UV position map and calibrated camera poses.
The loss is defined as a bidirectional cross-entropy between the predicted attention maps and ground-truth correspondences (based on pairwise L2 distances of 3D positions).
This mechanism encourages the appearance stream to reference pose-invariant geometric cues for robust alignment, particularly in occluded or ambiguous regions, while grounding the geometry stream in the appearance context.

4. Dataset Construction

Since canonicalized multi-view geometry supervision is not natively available, the authors constructed a dataset from multi-view face video collections (RenderMe-360 and NeRSemble v2). They utilized the VGGTFace pipeline to aggregate point maps and UV coordinates, refining them via bundle adjustment and mapping them to a shared FLAME canonical space via Procrustes alignment to create view-invariant geometry supervision signals.

Key Contributions

Unified Dual-Stream Framework: A novel diffusion architecture where appearance and geometry streams interact via shared attention layers, enforcing cross-view geometric consistency as an intrinsic property of generation.
Geometry-Guided Attention Alignment: A loss function that supervises cross-modal attention with 3D-consistent correspondences, ensuring the two streams mutually constrain each other.
Canonical UV Position Map: A view-invariant geometry representation defined in FLAME space that is natively compatible with diffusion models, facilitating shared geometry-constrained generation.
Empirical Validation: Extensive experiments demonstrating that jointly generated geometry significantly improves both visual quality and geometric consistency compared to appearance-only baselines.

Experimental Results

GeoFace was evaluated on RenderMe-360 and NeRSemble v2 datasets across frontal-to-mid and profile view ranges.

Quantitative Performance: GeoFace consistently outperformed existing baselines (including NeRF-based methods like PanoHead and diffusion-based methods like CAP4D, DiffPortrait360, and SEVA) across all metrics: PSNR, SSIM, LPIPS, and Cosine Similarity (CSIM) for identity preservation. Notably, the performance gap widened in profile views, where appearance-only methods struggle with large pose variations.
Qualitative Improvements: The method produced geometrically consistent results with better preservation of facial structures (nose, jawline, forehead) under large pose changes. It effectively resolved depth ambiguities in self-occluded regions where appearance-only models failed.
Downstream Utility: The jointly generated geometry served as an effective initialization prior for 3D Gaussian Splatting. Compared to random or COLMAP-based initialization, the mesh-based initialization led to faster convergence and lower LPIPS during reconstruction, demonstrating the utility of the generated geometry for downstream 3D tasks.
Ablation Studies: Removing the geometry stream, using fixed camera tokens instead of learnable ones, or omitting the alignment loss all resulted in measurable performance drops, confirming the necessity of each component.

Significance and Claims

The paper claims that GeoFace represents a significant step forward by moving beyond appearance-driven generation to a framework where geometric consistency is enforced intrinsically during the generative process. By jointly modeling appearance and a canonical geometry representation, GeoFace achieves superior identity preservation and structural fidelity, particularly in challenging scenarios involving large viewpoint changes. The authors posit that this approach facilitates more efficient and accurate downstream 3D reconstruction by providing a strong structural prior.

The paper acknowledges limitations, noting that the reliance on FLAME as a geometric backbone may limit the representation of regions outside the facial surface (e.g., hair, ears, teeth), which could affect subjects with complex hairstyles or accessories. Future work is suggested to extend geometric representations beyond the face region and further validate generalization to in-the-wild images.