💻 computer science

Enhancing Single-Image Facial Demorphing using Multimodal Large Language Models

This paper introduces a novel reference-free facial demorphing framework that leverages semantic embeddings from intermediate layers of Multimodal Large Language Models to condition a coupled diffusion-based reconstruction process, enabling the joint synthesis of constituent faces directly in the RGB domain with superior identity consistency and fine-grained detail preservation compared to existing latent-space approaches.

Original authors: Nitish Shukla, Arun Ross

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitish Shukla, Arun Ross

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Digital Smoothie"

Imagine you have two distinct smoothies: one is strawberry, and the other is blueberry. A "morphing attack" is like a hacker taking these two smoothies, blending them together in a blender, and pouring out a single purple drink.

In the world of face recognition, this "purple drink" is a fake photo created by blending two different people's faces. The goal of the bad guys is to trick security cameras (like at airports or on phones) into thinking this fake face belongs to both people at the same time, allowing them to bypass security.

Current security systems are good at shouting, "Hey, this photo is a fake!" (Detection). But they are terrible at answering the question: "Okay, it's fake, but who were the two original people?" (Reconstruction). Without knowing the original faces, the security team can't catch the impostors.

The Solution: The "Super-Detective" and the "Art Restorer"

This paper introduces a new method to reverse the blending process. It's like having a master art restorer who can look at a muddy, blended painting and separate it back into the two original, distinct portraits.

The authors built a system with two main parts working together:

The Super-Detective (The MLLM): This is a "Multimodal Large Language Model." Think of it as a very smart detective who looks at the fake photo and doesn't just see pixels; it understands the story of the face. It can reason about things like "This person has curly hair," "The background is a park," or "One person looks older than the other." Instead of just reading a text description, the system grabs the detective's internal "thoughts" (hidden data) to guide the process.
The Art Restorer (The Diffusion Model): This is a powerful AI that specializes in "un-blurring" images. It takes the fake photo and tries to peel it apart.

How They Work Together: The "Coupled Dance"

Usually, if you ask an AI to guess two people from a blended photo, it might get lazy and just output the same fake photo twice, or two random faces that don't look right.

The authors' secret sauce is making the AI do a "Coupled Dance."

Instead of guessing Person A and Person B separately, the AI guesses them simultaneously.
It uses the "Super-Detective's" thoughts to say, "Okay, I know Person A has a scar on the left cheek, so Person B must have the features that make up the rest of this blended face."
By forcing the AI to think about both people at the same time, they ensure the two new faces fit together perfectly to recreate the original blend, but are distinct from each other.

Why This is a Big Deal

Previous attempts to do this had some major flaws:

The "Compressed Map" Problem: Some old methods tried to do the work in a "compressed" digital space (like a low-resolution sketch). The paper argues this is like trying to restore a masterpiece using only a blurry thumbnail; you lose the tiny details like skin texture and hair strands. This new method works directly on the high-quality "pixels" (the full-resolution image), keeping all the fine details.
The "Text Bottleneck": Some methods asked the AI to write a description of the face (e.g., "a man with a beard") and then used that text to help. The paper found that this is like trying to describe a complex painting with only three words; you lose too much information. Instead, this method feeds the AI's raw, internal "thoughts" directly into the restoration process, preserving all the subtle clues.

The Results: Cracking the Code

The team tested their method on various types of "blended" faces, from simple computer edits to high-tech AI-generated fakes.

The Score: On standard tests, their method successfully recovered the original identities more than 96% of the time, even under very strict security settings where other methods failed.
The Quality: The restored faces weren't just recognizable; they looked sharp and clear, with much better image quality scores (PSNR) than previous attempts.
The "Middle" Layer Magic: They discovered that the "detective" (the AI model) is most helpful when you listen to its "middle thoughts" rather than its very first glance or its final conclusion. The middle layer seems to hold the perfect balance of identity details.

In Summary

This paper presents a new way to undo face morphing attacks. It uses a smart "detective" AI to understand the high-level story of a fake face and feeds those insights directly to a "restorer" AI that works on the full-resolution image. By having the restorer guess both original faces at the same time, it successfully separates the "purple smoothie" back into the original strawberry and blueberry, providing a powerful tool for forensic analysis and biometric security.

Technical Summary: Enhancing Single-Image Facial Demorphing using Multimodal Large Language Models

Problem Statement
Face recognition systems face a critical vulnerability to morphing attacks, where a composite image is crafted to simultaneously match multiple identities, facilitating identity fraud and unauthorized access. While existing Morph Attack Detection (MAD) methods can identify whether an image is morphed, they generally fail to recover the constituent original faces or identities, limiting their forensic utility. This paper addresses the challenge of single-image reference-free facial demorphing: the task of decomposing a single morphed image into its two original constituent faces without access to a trusted reference image. This is an ill-posed inverse problem, as infinitely many pairs of faces could theoretically generate the same morph, particularly when the morph is created using advanced generative models (e.g., StyleGAN, diffusion models) that entangle identity features in latent spaces.

Methodology
The authors propose a novel framework that reframes facial demorphing as a semantically-guided coupled conditional generation problem. The core of the approach involves a coupled Denoising Diffusion Probabilistic Model (DDPM) operating directly in the RGB domain, augmented by Multimodal Large Language Models (MLLMs) for semantic conditioning.

Coupled Diffusion Process: Unlike prior methods that reconstruct faces independently or operate in compressed latent spaces, this method treats the two constituent faces as a paired entity $(i_1, i_2)$ . The forward diffusion process adds noise to the joint 6-channel space (two 3-channel images) simultaneously. This enforces inter-output dependency, ensuring that the reverse denoising process generates both faces jointly, thereby preserving the inherent relationship established by the morphing process and preventing degenerate solutions (e.g., outputting the morph twice).
MLLM-Guided Conditioning: Instead of relying on text generation or raw visual features alone, the method extracts hidden state representations from intermediate transformer layers of a pretrained MLLM (specifically Qwen3-VL and LLaVA-1.6). These embeddings capture high-level semantic reasoning about facial attributes, identity cues, and structural characteristics.
- The MLLM hidden states are projected and injected into the diffusion UNet via cross-attention mechanisms.
- Crucially, the method bypasses the lossy cycle of generating text and re-encoding it; it directly utilizes the MLLM's internal multimodal reasoning capabilities to guide the denoising network.
RGB-Domain Operation: The diffusion model operates directly on pixel data (256x256 resolution) rather than latent spaces. The input to the denoising network at each step consists of the noisy target pair concatenated with the original morph image (9 channels total), allowing the network to attend to fine-grained perceptual details (hair, background, textures) often lost in latent representations.

Key Contributions

Prompt-Driven Reference-Free Framework: The authors introduce a reference-free demorphing system that leverages MLLMs to inject high-level semantic reasoning into the reconstruction process.
Coupled Conditional Generation: They formulate the problem as a joint generation task, enabling the simultaneous reconstruction of two distinct identities from a single morph while maintaining consistency.
Novel Conditioning Strategy: A new strategy is introduced that utilizes intermediate MLLM hidden states as conditioning signals, combining semantic guidance with direct RGB-domain morph conditioning.
Comprehensive Ablation Analysis: The study provides extensive analysis on the impact of MLLM embedding depths (initial, middle, last layers), the superiority of RGB-domain over latent-space demorphing, and the benefits of full MLLM embeddings versus raw ViT features.

Results
The proposed method was evaluated on multiple benchmarks covering landmark-based (AMSL, OpenCV, FaceMorph, WebMorph) and learning-based (StyleGAN, MorDiff) morphing techniques.

Identity Restoration: On landmark-based morphs, the method achieves Restoration Accuracy (RA) exceeding 96% at a strict 0.1% False Match Rate (FMR) using both ArcFace and AdaFace matchers.
Generative Morphs: On challenging StyleGAN morphs, the method significantly outperforms existing baselines. For instance, it achieves 9.13% RA at 0.1% FMR (ArcFace), a substantial improvement over prior GAN-based methods (e.g., lc-GAN at 2.12%).
Image Fidelity: The method demonstrates superior perceptual quality, with PSNR gains of 6–9 dB over existing methods on various datasets.
Retrieval Performance: In retrieval tasks with a large gallery of distractors, the method achieves a 94.67% mAP@1, outperforming all competing approaches, particularly on StyleGAN morphs where it improves mAP@1 by over 8% compared to the next best method.

Significance and Claims
The paper claims that its primary significance lies in shifting the paradigm of demorphing from purely pixel-based or latent-space optimization to semantically-guided generation. By leveraging the reasoning capabilities of MLLMs, the method can disentangle identities that are heavily entangled in modern generative morphs.

The authors emphasize that:

Intermediate MLLM layers encode the most identity-discriminative representations, balancing structural detail and semantic abstraction.
RGB-domain demorphing is fundamentally superior to latent-space approaches for this task, as it preserves fine-grained spatial cues necessary for identity recovery.
Direct hidden-state conditioning is more effective than text-based conditioning, as it avoids information bottlenecks and retains subtle visual cues.

The work concludes that integrating foundation model semantics with generative reconstruction offers a robust solution for biometric security, particularly in forensic scenarios where reference images are unavailable. The authors acknowledge that challenges remain in handling highly entangled generative morphs and reducing the computational cost of MLLM conditioning, but the results establish a new state-of-the-art for reference-free facial demorphing.