🤖 AI

IConFace: Identity-Structure Asymmetric Conditioning for Unified Reference-Aware Face Restoration

The paper proposes IConFace, a unified framework that employs identity-structure asymmetric conditioning to effectively balance reference-based identity preservation and no-reference structural recovery for high-quality blind face restoration under severe degradation.

Original authors: Axi Niu, Jinyang Zhang, Senyan Qing

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Axi Niu, Jinyang Zhang, Senyan Qing

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Fixing a Blurry Face Without Losing Who You Are

Imagine you have an old, blurry, or scratched photograph of your grandmother. You want to restore it so it looks crisp and clear again. This is called Blind Face Restoration.

The problem is that the photo is so damaged that the computer doesn't know exactly what your grandmother's nose looked like or the exact shape of her eyes. It has to "guess." If the computer guesses wrong, the restored photo might look like a perfect, high-quality face, but it won't look like your grandmother anymore. It might look like a stranger.

The Solution: Using a "Reference" Photo

To fix this, we can give the computer a second photo of your grandmother taken on a different day (a "reference"). This helps the computer remember her specific features.

But here's the catch: The reference photo might have her smiling, wearing heavy makeup, or looking at a different angle. If the computer blindly copies the reference photo, it might accidentally give your grandmother a smile she didn't have in the blurry photo, or change the angle of her head. It's like trying to fix a broken vase by gluing on pieces from a different vase that happens to be the same color.

Enter IConFace: The "Smart Architect"

The authors of this paper created a new system called IConFace. Think of IConFace as a very smart architect who knows how to balance two conflicting instructions:

"Keep the shape of the broken vase" (The blurry input).
"Use the details from the good vase" (The reference photo).

Most previous systems tried to do this symmetrically (treating both photos equally), which often led to confusion. IConFace uses Asymmetric Conditioning, which means it treats the two photos differently based on their specific job.

1. The Blurry Photo is the "Blueprint" (Structure Anchor)

The system looks at the blurry, damaged photo first and says, "This is the only thing that tells us the correct shape, pose, and expression."

The Metaphor: Imagine the blurry photo is a blueprint for a house. Even if the blueprint is smudged and hard to read, it's the only thing that tells you where the walls and doors go. The system uses this blueprint to ensure the final face has the right shape and doesn't accidentally change the person's pose or expression.

2. The Reference Photo is the "Interior Designer" (Identity Anchor)

The system looks at the clear reference photo and says, "This tells us what the person looks like, but not how they are standing right now."

The Metaphor: The reference photo is like a catalog of furniture and paint colors. It tells the architect, "The owner likes blue walls and a specific style of sofa." The system extracts only the identity details (the "blue paint" and "sofa style") and ignores the pose or lighting of the reference. It then applies these details to the blueprint.

How It Works (The "Secret Sauce")

The paper describes two main pathways that work together:

The Identity Pathway (The "Name Tag"): The system takes the reference photos and compresses them into a single, powerful "identity anchor." It's like taking a whole album of photos and distilling them down to a single ID card that says, "This is who the person is." It uses this ID card to gently nudge the restoration toward the right person without forcing the reference photo's pose onto the result.
The Structure Pathway (The "Memory Bank"): The system takes the blurry photo and breaks it down into two parts: the big picture (overall layout) and the fine details (skin texture, small moles). It stores these in a special memory bank. This ensures that even if the reference photo is used, the system never forgets the original shape of the face in the blurry image.

The Best Part: It Works With or Without a Reference

Usually, you need a reference photo to get a perfect result. But what if you don't have one?

IConFace is a unified system. It's like a Swiss Army knife that has two modes:

Reference Mode: If you give it a reference photo, it uses the "ID card" to make the face look exactly like the person in the reference.
No-Reference Mode: If you don't give it a reference photo, it switches gears. It stops trying to guess the identity from a reference and instead relies entirely on its "Memory Bank" of the blurry photo to do the best possible job on its own.

The Results: Why It Matters

The paper tested IConFace against other top methods. Here is what they found:

Better Identity: When using a reference photo, IConFace was much better at keeping the person looking like themselves compared to other methods, which often made them look like a generic version of the person.
Better Details: It recovered fine details (like the shape of eyes or a small mole) much better, especially when the original photo was very damaged.
No "Over-Copying": Unlike other systems that might accidentally copy the reference photo's smile or lighting, IConFace kept the original pose and expression from the blurry photo while just fixing the quality.
Versatility: It works just as well when you have a reference photo as it does when you don't. You don't need two different tools for two different jobs; one tool does it all.

Summary

IConFace is a new tool for fixing blurry faces. It solves the problem of "guessing" by using a clever trick: it treats the blurry photo as the structure (the shape) and the reference photo as the identity (the look). This allows it to restore faces that look sharp and realistic while staying true to who the person actually is, whether or not you have a second photo to help.

Technical Summary: IConFace

Problem Statement

Blind face restoration (BFR) is a highly ill-posed problem, particularly under severe degradation where identity-critical details and local facial cues are often lost. While modern generative priors (diffusion, codebooks, transformers) improve perceptual realism, they struggle to maintain identity consistency without explicit evidence. Reference-aware restoration attempts to solve this by using same-identity images to provide missing identity evidence. However, existing methods face a critical trade-off: references encode not only identity but also pose, expression, illumination, age, and makeup. Directly transferring these features often leads to "over-attention," where the restored face adopts the reference's specific appearance (e.g., pose or expression) rather than preserving the structure implied by the degraded input. Furthermore, practical systems must handle scenarios where references are absent, noisy, or limited, yet many reference-aware models fail to degrade gracefully to robust no-reference restoration.

Methodology: IConFace

The authors propose IConFace, a unified framework that supports both reference-aware and no-reference restoration within a single checkpoint. The core innovation is Identity–Structure Asymmetric Conditioning, which treats the degraded input and reference images differently based on their roles.

1. Asymmetric Conditioning Design

The framework is built on a FLUX.2-klein-base-4B hybrid concat backbone. The main sequence concatenates noisy scene tokens, degraded-image tokens, and optional reference tokens ( $x_{in} = [x_{scene}; x_{deg}; x_{ref}]$ ). The asymmetry is implemented through two distinct pathways:

Global Identity Pathway (Reference Role):
- References are not treated as local spatial inputs but are distilled into a global identity anchor.
- Using a frozen AdaFace encoder, raw embeddings ( $z_r$ ) are separated into identity direction ( $e_r$ ) and reliability/norm ( $q_r$ ).
- Multiple references are aggregated via norm-weighted softmax ( $w_r$ ) to produce a single effective identity anchor ( $e_{id}$ ).
- This anchor is projected into the backbone's hidden space to generate modulation deltas ( $\Delta$ ) for image blocks. Crucially, this modulation is image-only; the text stream remains untouched, ensuring references control identity globally without acting as a local structure-transfer branch.
- Fallback: If references are absent, the pathway uses the normalized AdaFace embedding of the degraded image itself as a weak self-conditioning anchor.
Degraded Structure Pathway (Input Role):
- The degraded image is treated as the spatial structure anchor, as it is the only observation spatially aligned with the target.
- Low-Rank Residual: Degraded tokens are resized and injected as a low-rank input residual to reinforce the scene structure.
- Two-Route Memory: Full degraded attention is compressed into a fixed memory budget using two learned-query poolers:
  - Base Route: Pools $x_{deg}$ to preserve global layout and illumination.
  - Detail Route: Pools high-frequency residuals ( $x_{deg} - \text{smooth}(x_{deg})$ ) to emphasize local facial cues.
- This allows each block to read both coarse structure and fine details without replacing the main concatenation route.

2. Training Objective

The model is trained in latent space using a flow-matching objective.

Restoration Loss: Regresses the flow field ( $L_{fm}$ ).
Identity Loss ( $L_{id}$ ): For reference-aware samples, a cosine similarity loss is computed between the restored image and the reference anchor.
- A stabilizer term ( $L_{hard}$ ) is introduced for imperfect references, weighted by the consistency between the reference and the clean target.
- The identity loss is sigma-weighted ( $\omega(\sigma)$ ), down-weighting high-noise steps rather than hard-skipping them.
No-Reference Mode: The identity loss is zero when references are absent, relying solely on the flow-matching loss and the weak self-conditioning from the degraded image.

Key Contributions

Unified Optional-Reference Formulation: A single checkpoint supports both reference-aware and no-reference restoration through asymmetric conditioning, eliminating the need for separate models.
Asymmetric Identity–Structure Conditioning: A novel design where references are compressed into a global identity controller (AdaFace modulation), while the degraded image is reinforced as the spatial structure anchor via low-rank residuals and two-route memory.
Lightweight Pathways: The use of norm-weighted AdaFace modulation and low-rank cross-attention ensures efficient integration of identity and structure without heavy computational overhead.
Robustness to Missing References: The model naturally falls back to degraded-only restoration when references are absent, maintaining strong perceptual quality.

Experimental Results

The authors evaluated IConFace on four reference-aware benchmarks (CelebA-Test-Ref, FFHQ-Ref Moderate, FFHQ-Ref Severe, CelebHQRef100) and five no-reference benchmarks (LFW, CelebChild, WebPhoto, Wider-Test, CelebA-Test).

Reference-Aware Performance: IConFace achieved state-of-the-art performance on Ref-ArcFace and Ref-AdaFace across all benchmarks. Notably, on the severe degradation splits (FFHQ-Ref Severe and CelebHQRef100), it showed significant gains (e.g., +0.104 Ref-AdaFace on FFHQ-Ref Severe over the best baseline). It also maintained top-tier performance on perceptual metrics (MUSIQ, CLIP-IQA, MANIQA), demonstrating that identity recovery did not come at the cost of visual fidelity.
No-Reference Performance: In the absence of references, IConFace ranked first on MUSIQ, CLIP-IQA, and MANIQA across all no-reference datasets, proving that reference-aware training did not over-specialize the model.
Ablation Studies: Experiments confirmed that the Global Identity Pathway was the primary driver for reference alignment, while the Degraded Structure Pathway provided consistent gains in stabilizing the output structure. The Two-Route Memory (Base + Detail) outperformed single-route variants, validating the separation of global layout and local detail.

Significance and Claims

The paper claims that IConFace successfully addresses the "overuse of reference appearance" problem by explicitly decoupling identity control from spatial structure. By assigning references a global identity role and the degraded input a structure-preserving role, the model recovers identity-consistent details (e.g., eye spacing, skin texture, moles) while strictly adhering to the pose and expression of the degraded input.

The authors emphasize that this approach is particularly valuable under severe degradation, where the degraded image lacks sufficient identity evidence, but the reference provides it without dictating the facial state. The unified nature of the model allows it to serve as a robust solution for both personal photo enhancement (where references are available) and general blind restoration (where they are not), bridging the gap between reference-aware and no-reference paradigms.