Technical Summary: HDRFace

Problem Statement

Face restoration under complex degradations (e.g., blur, noise, downsampling, JPEG compression) remains an ill-posed inverse problem due to severe information loss. While diffusion models have become a dominant paradigm for this task by leveraging strong generative priors, existing methods typically condition generation solely on low-quality (LQ) inputs. This reliance on LQ images alone makes it difficult to recover identity-critical details and fine-grained textures, as many facial features are irreversibly damaged or heavily blurred in the input. Consequently, reconstructed images often suffer from suboptimal identity consistency and perceptual realism.

Methodology

The authors propose HDRFace, a High-Dimensional Representation conditioned Face restoration framework. The core philosophy is to inject semantically rich, fine-grained priors into the conditional generation flow without modifying the generative backbone. The pipeline consists of three main stages:

1. Intermediate Restoration and Feature Extraction

Instead of relying directly on the LQ input for guidance, the framework first employs an off-the-shelf restorer to generate a structurally reliable intermediate restoration ($I_{mid}$).

• Dual Feature Extraction: A pretrained high-dimensional feature encoder, DINOv3, is used to extract fine-grained facial representations from two sources:
  • The original LQ input ($I_{LR}$), denoted as $\mathbf{F}_{lr}$.
  • The intermediate restoration result ($I_{mid}$), denoted as $\mathbf{F}_{face}$.
• Complementary Roles: The LQ features are expected to provide structural constraints, while the intermediate features offer refined, semantically rich details.

2. Structure-Detail aware adaptive Fusion Mechanism (SDFM)

To effectively integrate these multi-source features, the authors introduce SDFM, an architecture-independent module that adaptively fuses $\mathbf{F}_{lr}$ and $\mathbf{F}_{face}$.

• Discrepancy Modeling: The module computes the absolute difference between normalized feature sequences to explicitly characterize the gap between structural cues and fine-grained details.
• Adaptive Gating: SDFM learns fusion weights through two perspectives:
  • Channel-wise gating ($\alpha_c$): Derived from globally pooled statistics of facial features, LR features, and their discrepancy.
  • Token-wise gating ($\alpha_t$): Derived from token-level concatenated features to capture local importance.
• Fusion: The final fused representation ($\mathbf{F}_{fused}$) is a weighted combination that emphasizes global constraints during structure modeling and strengthens representation guidance during detail synthesis, balancing structural consistency with detail fidelity.

3. Conditional Generation

The fused high-dimensional representation is injected into the conditional branch of the generative model, replacing or augmenting traditional text embeddings.

• Backbone Agnostic: The framework is validated on two distinct generative architectures:
  • SD V2.1-base: A diffusion-denoising U-Net architecture.
  • Qwen-Image: A DiT (Diffusion Transformer) architecture based on Rectified Flow.
• Training Strategy: The models are trained as one-step restoration systems. The training objective combines reconstruction, perceptual, identity (ArcFace), and GAN losses. For SD V2.1-base, the LQ latent serves as the noisy input at a fixed timestep; for Qwen-Image, a linear noise perturbation process (Rectified Flow) is employed.

Key Contributions

1. High-Dimensional Representation Conditioning: The paper proposes a framework that injects DINOv3 semantic features into the conditional branch of generative models. This provides priors beyond the limited information in LQ inputs, alleviating the ill-posedness of the restoration task.
2. SDFM Module: The design of an architecture-independent fusion mechanism that adaptively balances structural consistency (from LQ inputs) and detail fidelity (from intermediate results) without altering the generative backbone.
3. Generalization and Robustness: The method demonstrates stable and consistent performance gains across different generative architectures (U-Net and DiT) and generation mechanisms, highlighting strong cross-architecture generalization.

Experimental Results

Experiments were conducted on synthetic (CelebA-Test) and real-world (LFW-Test, CelebChild) datasets, comparing HDRFace against non-diffusion, multi-step diffusion, and one-step diffusion baselines (e.g., CodeFormer, OSDFace, PGDiff).

• Quantitative Performance: HDRFace consistently outperforms state-of-the-art methods across multiple metrics.
  • Perceptual Quality: Achieves superior scores in LPIPS, DISTS, and TOPIQ.
  • Identity Consistency: Demonstrates lower ArcFace embedding angles (Deg.) and landmark distances (LMD), indicating better preservation of facial identity.
  • Image Quality: Shows improvements in no-reference IQA metrics like Q-Align and LIQE.
• Qualitative Results: Visual comparisons show that HDRFace restores fine-grained details (e.g., makeup glossiness, pupil shapes, hair textures, earrings) more accurately than competing methods, with results closer to the ground truth.
• Ablation Studies:
  • SDFM: Removing SDFM or using only text/DINO features results in performance drops, confirming the necessity of adaptive fusion.
  • Input Configuration: Using both SR (intermediate) and LR inputs yields better results than using either alone, validating the complementary nature of the features.
  • Model Choice: DINOv3-L (Large) and Qwen-Image as the intermediate restorer provided the best performance compared to smaller variants or other guidance models like CodeFormer.

Significance and Claims

The paper claims that HDRFace offers a robust solution for face restoration under severe degradations by shifting the conditioning paradigm from low-quality inputs to high-dimensional visual representations.

• Beyond Text Prompts: The authors argue that text prompts are unsuitable for face restoration because identity-specific and fine-grained details are difficult to describe linguistically and cannot be reliably inferred from LQ inputs. High-dimensional visual representations encode these details more directly.
• Trade-off: While the method introduces computational overhead due to the intermediate restorer and feature extraction, the authors assert this trade-off (quality over speed) is worthwhile for identity-critical tasks. The one-step inference mechanism still maintains competitive latency compared to multi-step diffusion models.
• Limitations: The authors modestly acknowledge that the method relies on the reliability of the intermediate restoration. In cases of extremely severe degradation where identity-critical cues are completely lost, the method may still produce hallucinated details or structural deviations, as the problem remains fundamentally ill-posed. They suggest future work could involve selecting from multiple candidate intermediate results to improve robustness.