You Only Landmark Once: Lightweight U-Net Face Super Resolution with YOLO-World Landmark Heatmaps

Technical Summary: You Only Landmark Once

Problem Statement
Face image super-resolution (SR) aims to reconstruct high-resolution facial images from severely degraded low-resolution inputs. Under extreme upscaling factors (e.g., 8×), fine facial details are often lost, making accurate reconstruction difficult. Existing state-of-the-art methods typically rely on computationally heavy architectures (such as diffusion models or large vision transformers), adversarial training schemes, or separate alignment networks to guide the reconstruction of critical regions like eyes, nose, and mouth. These approaches often result in prohibitive model complexity, extensive training requirements, and limited applicability in resource-constrained or real-time scenarios. Furthermore, methods relying on dedicated landmark detectors can struggle with generalization and robustness when landmarks are occluded.

Methodology
The authors propose a lightweight, U-Net-based architecture designed to reconstruct 128×128 facial images from 16×16 inputs without the need for auxiliary training networks or adversarial training.

• Architecture: The model follows an encoder-bottleneck-decoder design. The encoder uses four convolutional blocks with increasing channel depth (starting at 48 filters) and stride-2 convolutions for downsampling. The decoder mirrors this structure with progressive upsampling and skip connections. To avoid checkerboard artifacts, the upsampling process utilizes nearest-neighbor interpolation followed by convolutional layers rather than transposed convolutions. A refinement head with residual blocks further enhances spatial details at full resolution. The model includes a global skip connection that adds an upsampled version of the low-resolution input to the final prediction to improve color consistency and convergence. The total parameter count is approximately 7.3 million.
• Heatmap-Guided Supervision: A core innovation is a novel supervision strategy that leverages heatmaps generated by YOLO-World, an open-vocabulary object detector. Instead of training a dedicated landmark network, the authors use YOLO-World to detect facial components (eyes, nose, mouth, etc.) in the ground-truth high-resolution images.
  • Heatmap Generation: Detected regions are processed using Scharr and Canny edge detectors, normalized, and smoothed with Gaussian blur. Spatial fading functions (Gaussian or Inverse Gaussian) are applied to modulate importance based on the distance from the center of the feature. Class-specific weights are manually assigned to emphasize visually prominent regions (e.g., eyes receive a weight of 4.5).
  • Loss Function: These heatmaps are converted into spatial weights to form a "heatmap-guided loss." This loss reweights pixel errors, assigning higher penalties to errors in semantically important regions while down-weighting less relevant areas.
• Composite Loss: The network is trained using a composite loss function combining:
  1. Pixel Loss (MSE): For low-level fidelity.
  2. Perceptual Loss: Based on VGG16 feature activations.
  3. Heatmap Loss: The proposed weighted pixel loss.
  4. LPIPS Loss: To enhance perceptual realism and texture sharpness.

Key Contributions

1. Zero-Cost Priors: The introduction of a method to obtain face super-resolution priors from a general-purpose YOLO-World model at zero additional training cost. This eliminates the need for task-specific auxiliary networks or landmark detectors.
2. Lightweight Framework: The presentation of a stable, lightweight super-resolution framework based on a U-Net backbone that balances reconstruction quality with computational efficiency.
3. Robust Validation: Extensive validation on the aligned CelebA dataset under both bilinear and bicubic degradation settings, demonstrating the approach's robustness and general applicability.

Experimental Results
The method was evaluated on the aligned CelebA dataset (16×16 to 128×128, 8× scaling).

• Quantitative Performance: Under bilinear downsampling, the proposed method with heatmap loss achieved a PSNR of 25.47 dB and SSIM of 0.716, outperforming the baseline without heatmap loss (24.66 dB PSNR) and surpassing several complex prior-based architectures (e.g., ProgFSR, EIPNet). Under bicubic downsampling, the method achieved 24.77 PSNR and 0.6924 SSIM.
• Efficiency: The model demonstrates significant efficiency advantages. It requires only 4.0 GMACs (or 1.28 GMACs with a reduced refinement head) compared to 19–45 GMACs for attention-based or iterative methods like MAIC and DIC.
• Inference Speed: The approach achieves real-time performance, exceeding 200 FPS on GPU and maintaining practical inference times on CPU (approx. 4.20 ms latency), significantly faster than competing methods.
• Qualitative Results: Visual inspections indicate that the heatmap-guided model produces sharper and more coherent reconstructions in critical facial regions (eyes, mouth) with fewer artifacts compared to models without the heatmap loss.

Significance and Claims
The paper claims that detection-driven priors from general-purpose foundation models can effectively guide extreme face super-resolution without increasing architectural complexity or supervision costs. By directly reusing detector outputs as supervision, the approach maintains an efficient training and inference pipeline. The authors assert that their results demonstrate lightweight networks can exploit these priors to achieve perceptually convincing extreme upscaling, offering a robust alternative for scenarios where computational resources are limited or real-time performance is required. The work highlights the potential of repurposing foundation models as zero-cost priors for image restoration under severe degradation.