Multi-Attribute guided Thermal Face Image Translation based on Latent Diffusion Model

This paper proposes a novel multi-attribute guided latent diffusion model enhanced by a Self-attn Mamba module to generate high-quality visible face images from thermal inputs, effectively overcoming domain shifts and preserving identity features for improved infrared face recognition.

The Gist

Imagine you are a detective trying to identify a suspect, but the only photo you have is a blurry, black-and-white thermal image taken at night. You can see the heat signature of their face, but you can't tell their skin color, age, or gender. Most modern "face ID" systems are like detectives who only know how to read clear, color photos taken in daylight. When you show them a thermal image, they get confused and fail.

This paper introduces a new tool to help these detectives: a smart translator that turns those blurry thermal heat maps into clear, color photos, while making sure the person's identity stays exactly the same.

Here is how the authors built this translator, using some creative analogies:

The Problem: The "Blurry Thermal" Gap

Current methods try to turn thermal images into visible ones, but they often fail in two ways:

1. Old methods (GANs): Like a painter who is in a rush, they often produce distorted, weird-looking faces that look nothing like the real person.
2. Newer methods (Diffusion Models): These are like a very careful artist who takes forever to paint. They make high-quality images, but they often get the details wrong—painting a young man as an old woman, or giving someone the wrong skin tone. They struggle to keep the "identity" intact.

The Solution: A "Smart Translator" with Three Special Tools

The authors built a new system based on a Latent Diffusion Model (LDM). Think of this model as a master painter working in a "dream space" (latent space) where it can work faster and more efficiently than painting on a giant canvas.

To make sure the painting is perfect, they added three special tools:

1. The "Attribute Detective" (Multi-Attribute Classifier)
Before the painter starts, this tool acts like a detective who looks at the thermal image and asks: "Is this person a man or a woman? Are they young or old? What is their skin tone?"

• How it works: The system was trained to look at thermal images and guess these traits just as accurately as if it were looking at a normal color photo.
• The Analogy: It's like giving the painter a detailed note card that says, "Draw a 29-year-old man with tanned yellow skin." This ensures the final photo doesn't just look like a face, but the right face with the correct features.

2. The "Speedy Brain" (Self-Attn Mamba)
Standard AI models are like a librarian who has to read every single book in a library one by one to find a specific fact. This is slow.

• The Innovation: The authors replaced the standard "attention" mechanism with something called Mamba.
• The Analogy: Mamba is like a librarian who can scan the whole library instantly and grab the right book in one go. It allows the system to understand the whole face at once (global modeling) without getting bogged down, making the translation process much faster and lighter on computer memory.

3. The "Identity Guardian" (ID Loss)
Even with a good description, the painter might accidentally change the person's nose or jawline.

• The Fix: The system has a "guardian" that constantly checks the new photo against the original thermal image. If the new photo starts to look like a different person, the guardian says, "Stop! Fix the jawline!" This ensures the person's unique identity is preserved.

The Results: A Better Portrait

The authors tested this new translator on two large datasets of thermal and visible face images. They compared their method against the best existing tools (both the "rushed painters" and the "slow artists").

• Quality: Their photos were sharper, had better colors, and looked more realistic (higher scores in image quality metrics).
• Identity: Their photos were much better at keeping the person's true identity. If you ran these new photos through a standard face recognition system, it recognized the person much more often than it did with photos made by other methods.
• Speed: Thanks to the "Speedy Brain" (Mamba), their system was significantly faster and used less computer power than the competition.

In Summary

The paper presents a new way to turn night-vision thermal faces into clear, color portraits. By combining a detective to guess the traits, a speedy brain to process the image quickly, and a guardian to protect the identity, they created a system that produces clearer, more accurate, and faster results than current technology.

Technical Summary

Technical Summary: Multi-Attribute guided Thermal Face Image Translation based on Latent Diffusion Model

Problem Statement

Modern surveillance systems increasingly rely on thermal cameras for nighttime operations, yet most facial recognition models are trained on visible-light datasets. This domain shift causes significant performance degradation when processing infrared inputs. Traditional feature-based methods for Heterogeneous Face Recognition (HFR) have proven ineffective due to the sparse information and low resolution of thermal images. While Generative Adversarial Networks (GANs) offered a paradigm shift by converting thermal images to visible ones (Thermal-to-Visible or T2V), they suffer from unstable training, limited generalization, and often produce distorted or blurred outputs.

Recent diffusion models have improved image quality but introduce new challenges: slow inference speeds and difficulty in preserving critical identity attributes such as skin color, gender, and age. Existing diffusion-based approaches often fail to accurately capture these attributes due to the substantial gap between thermal and visible domains, leading to feature loss and identity distortion in generated images.

Methodology

The authors propose a novel framework based on a Latent Diffusion Model (LDM) designed to generate high-quality visible face images from thermal inputs while preserving identity and facial attributes. The pipeline consists of three primary components:

1. Multi-Attribute Classifier

To mitigate feature loss during restoration, the authors designed a classifier to extract key facial attributes (gender, age, and skin color) directly from thermal images.

• Training Strategy: A pre-trained classifier (based on EfficientNet) is first trained on visible images using the FairFace and UTKFace datasets. A copy of this network is then trained on paired infrared images.
• Consistency Loss: The training objective enforces consistency between the classification outputs of the visible and infrared networks using Mean Squared Error (MSE). This ensures the infrared classifier can perceive attributes similarly to the visible classifier.
• Prompt Generation: The classification results are converted into text prompts (e.g., "A face of {age} years old {gender} with {skin tone} skin") and encoded using a customized CLIP encoder. These embeddings serve as conditional inputs for the diffusion model.

2. Self-Attn Mamba Module

To address the computational inefficiency of standard diffusion models, the authors replace the original multi-head self-attention (MHSA) layers with a Self-Attn Mamba module.

• Mechanism: This module utilizes Selective State Space Models (SSMs) combined with convolutional layers. It processes input sequences via a state-space representation ($h_t = Ah_{t-1} + Bx_t$) and employs a bidirectional structure (BiMamba) to capture both forward and backward dependencies.
• Benefit: This design achieves linear complexity with respect to input length, enabling efficient global feature modeling with significantly reduced computational overhead compared to traditional attention mechanisms.

3. Conditioned Latent Diffusion Pipeline

The core generation process operates in a latent space rather than pixel space, utilizing a VQ-VAE architecture.

• Architecture: The system employs two VQ-VAE models: one for visible image reconstruction and one for thermal image encoding. A pre-trained CLIP encoder (fine-tuned on UTKFace and FairFace) processes the attribute prompts.
• Denoising Process: A UNet-based noise predictor ($\epsilon_\theta$) iteratively refines the latent features. The process is conditioned on the attribute embeddings via cross-attention mechanisms.
• Identity Preservation: An identity verification module is integrated into the training loop. It calculates an ID Loss based on the cosine similarity between the embeddings of the synthesized image and a reference visible image from the same subject, ensuring structural and feature fidelity.

Key Contributions

The paper outlines three primary contributions:

1. Multi-Attribute Classifier Integration: The design of a classifier within the LDM framework to extract critical facial attributes (gender, age, skin color) from thermal images, which are then used to guide the generation process.
2. Self-Attn Mamba Module: The introduction of a Mamba-based module to replace standard self-attention, achieving finer global feature modeling at a lower computational cost and significantly improving inference speed.
3. State-of-the-Art Performance: The proposal of a complete T2V framework that surpasses recent GAN and diffusion-based methods in both image quality and identity preservation across multiple datasets.

Experimental Results

The proposed method was evaluated on two benchmark datasets: ARL-VTF (500,000+ images) and Speaking Faces (142 subjects).

Image Quality Metrics

The model demonstrated superior performance in structural similarity and perceptual quality:

• ARL-VTF: Achieved an SSIM of 0.7642 and PSNR of 28.54, outperforming baselines like T2V-DDPM (SSIM 0.6712) and DiffuseIT. It also recorded the lowest FID (29.15) and LPIPS (0.1813).
• Speaking Faces: Recorded an SSIM of 0.7455 and PSNR of 30.11, with an FID of 21.16.

Identity Recognition Metrics

The method showed significant improvements in preserving identity:

• ARL-VTF: Achieved a Rank-1 accuracy of 81.83% and a Verification Rate at FAR=1% (VR@FAR) of 62.58%. This represents an 8.3% gain in VR@FAR over the second-best method (T2V-DDPM).
• Speaking Faces: Achieved a Rank-1 accuracy of 86.35% and VR@FAR=1% of 65.13%.

Efficiency and Ablation Studies

• Inference Speed: By integrating Self-Attn Mamba, the model reduced inference time to 36ms per image on an NVIDIA Tesla V100, compared to 126ms for T2V-DDPM.
• Parameter Efficiency: The proposed model uses 75.96M parameters (37% memory usage), a significant reduction compared to T2V-DDPM (273.68M parameters).
• Ablation: Experiments confirmed that each component (Mamba, Classifier, CLIP) contributed incrementally to performance. The full model (Variant D) yielded the best results, particularly in reproducing diverse skin tones and maintaining facial structure.

Significance and Claims

The authors claim that their approach effectively addresses the fundamental limitations of current T2V methods, specifically the trade-off between image quality, attribute preservation, and inference speed. By combining a multi-attribute classifier with a Latent Diffusion Model and the efficient Self-Attn Mamba module, the framework successfully generates high-fidelity visible images from thermal inputs that retain critical identity features such as skin tone, age, and gender. The results validate the effectiveness of this hybrid architecture in overcoming cross-modal discrepancies, establishing a new state-of-the-art for thermal-to-visible face translation. Future work is noted to explore robustness under varying thermal conditions.