All in One: Unifying Deepfake Detection, Tampering Localization, and Source Tracing with a Robust Landmark-Identity Watermark

Imagine the internet is a giant, bustling marketplace where people trade photos. Lately, a new kind of "magic trick" has appeared called Deepfake. It's like a digital wizard that can take a photo of your face and swap it onto someone else's body, or make you say things you never said. It's so good that you can't tell the difference just by looking. This is dangerous because scammers and bad actors can use it to lie, steal money, or ruin reputations.

For a long time, the people trying to stop this (the "detectives") have been playing a game of catch-up. They build tools to spot the fake after it's made. But the wizards keep getting better, so the detectives are always one step behind.

This paper introduces a new, smarter way to fight back. Instead of waiting for the magic trick to happen, they put a super-secret, invisible security tag on the original photo before anyone can touch it. They call this system LIDMark.

Here is how it works, using a simple analogy:

The "Invisible ID Card" (The Watermark)

Imagine you have a very important passport photo. Before you even leave the house, you secretly write two things on the back of the photo using invisible ink that only a special scanner can see:

A Map of Your Face: A precise drawing of exactly where your eyes, nose, and mouth are supposed to be (136 points).
Your Secret ID Number: A unique code that says, "This photo belongs to Alice."

This is the LIDMark. It's 152 pieces of information packed into one tiny, invisible package.

The "Magic Scanner" (The Decoder)

Now, imagine a bad guy steals your photo and tries to use their magic wand to change your face (a Deepfake). They might stretch your nose, change your eyes, or swap your whole face with someone else's.

When the photo comes back to the detective, they use a special tool called the Factorized-Head Decoder (FHD). Think of this tool as a two-headed robot that looks at the damaged photo and tries to read the invisible ink:

Head #1 (The Geometer): This head tries to redraw the "Map of Your Face" based on the invisible ink. It asks, "Where should the nose be?"
Head #2 (The ID Reader): This head tries to read the "Secret ID Number." It asks, "Who does this photo belong to?"

The Three Superpowers

This system solves three big problems at once, which is why the authors call it an "All-in-One" solution:

1. Detecting the Lie (Is it real?)

The robot compares the "Map it drew from the invisible ink" (Head #1) against the "Map it sees with its eyes" on the damaged photo.

If the photo is real: The two maps match perfectly.
If the photo is a Deepfake: The bad guy's magic changed the face, so the "eye map" looks different from the "ink map." The robot sees a huge mismatch and shouts, "FAKE!"

2. Finding the Scars (Where is it fake?)

This is the clever part. Because the "Map" has 136 specific points, the robot can check them one by one.

If the nose is changed but the eyes are the same, the robot can point to the nose and say, "The nose is fake, but the eyes are real."
Most old systems just say "Fake" or "Real." This one gives you a forensic map showing exactly which parts were tampered with.

3. Catching the Thief (Who made it?)

Even if the face is totally destroyed, the "Secret ID Number" (Head #2) is designed to be super tough. It survives the magic wand.

The robot reads the code and says, "This photo was originally created by Alice."
This helps trace the source of the fake, even if the face has been swapped.

Why is this a big deal?

Previous methods were like having three different detectives, each with a different tool. One could only say "Fake," another could only find "Where," and a third could only find "Who." They didn't talk to each other, and they often failed if the photo was too damaged.

This new LIDMark system is like a Swiss Army Knife detective. It carries one invisible tag that does everything at once. It's:

Invisible: You can't see it with the naked eye (it doesn't ruin the photo).
Strong: It survives even if the photo is compressed, blurry, or heavily edited.
Smart: It doesn't just guess; it compares the original geometry to the current image to find the truth.

In short, the authors have built a digital "security seal" that turns a passive photo into an active witness, capable of defending itself against the most sophisticated face-swapping tricks in existence.

1. Problem Statement

The rapid advancement of deepfake technology poses severe threats to privacy and social security. Current forensic approaches suffer from two main limitations:

Reactive vs. Proactive: Traditional "passive" forensics rely on detecting artifacts, which often fail against unseen manipulation techniques. "Proactive" methods embed signals but are often limited to single tasks (e.g., only detection or only tracing).
Task Fragmentation: Existing proactive methods typically treat Deepfake Detection, Tampering Localization, and Source Tracing as independent tasks or pair them using complex dual-decoder architectures.
The "Localization Gap": Even advanced bifunctional methods (detection + tracing) fail to answer the critical question: "Where exactly is the face manipulated?" They lack the ability to pinpoint specific altered facial regions.
Trade-offs: Existing methods struggle to balance robustness, imperceptibility, and payload capacity. High-capacity payloads (needed for diverse forensic data) often degrade image quality or robustness.

2. Methodology: The LIDMark Framework

The authors propose a unified, "all-in-one" proactive forensic framework centered on a novel watermark and a specialized decoder.

A. The LIDMark Watermark (152-D Composite)

Instead of separate watermarks for different tasks, LIDMark structurally interweaves two heterogeneous payloads into a single 152-dimensional vector ( $W$ ):

Tamper-Sensitive Landmark Vector ( $W_L$ , 136-D): Encodes 68 normalized 2D facial landmarks. This component is designed to be sensitive to geometric manipulation. If the face is forged, the reconstructed geometry will deviate from the original.
Robust Source Identifier ( $W_{ID}$ , 16-D): A bipolar vector derived from a SHA-256 hash of the image filename. This component is designed to be robust against deepfake generation pipelines to enable source tracing.

B. Network Architecture

The framework consists of an Encoder, a Stochastic Manipulation Operator, and a Factorized-Head Decoder (FHD).

Encoder ( $E$ ): A two-stream fusion network that embeds $W$ into the cover image ( $I_{co}$ ) while preserving visual fidelity via global skip connections.
Stochastic Manipulation Operator ( $M$ ): Simulates diverse attacks during training, including common distortions (resize, blur, JPEG) and five distinct deepfake models (SimSwap, UniFace, CSCS, StarGAN-v2, InfoSwap).
Factorized-Head Decoder (FHD): The core innovation. Instead of using separate decoders for different tasks, FHD uses a shared backbone to extract features from the manipulated image, which are then split into two specialized heads:
- Regression Head: Reconstructs the 136-D landmark vector ( $\hat{W}_L$ ).
- Classification Head: Reconstructs the 16-D source identifier ( $\hat{W}_{ID}$ ).
- Significance: This factorization allows the network to jointly optimize for geometric reconstruction (sensitive) and identifier classification (robust) without interference.

C. The "Intrinsic-Extrinsic" Consistency Check

This mechanism enables Detection and Localization without needing the original image (blind reference):

Intrinsic Landmarks ( $\hat{W}_L$ ): Recovered by the FHD's regression head from the potentially manipulated image. These represent the original geometry.
Extrinsic Landmarks ( $W_{new}$ ): Re-detected from the manipulated image using a standard face-alignment library. These represent the current (potentially forged) geometry.
Logic:
- Detection: Calculate the Average Euclidean Distance (AED) between $\hat{W}_L$ and $W_{new}$ . A high AED indicates a deepfake; a low AED indicates a real image or benign distortion.
- Localization: By grouping landmarks by facial components (e.g., eyes, mouth), the system can compute AED locally. Regions with high AED are flagged as tampered.

D. Training Strategy

The model is trained using a multi-task loss function:

Imperceptibility Loss: Minimizes difference between watermarked and cover images.
Recovery Loss: Combines Mean Squared Error for landmarks and Binary Cross-Entropy for the identifier.
Adversarial Loss: Trains the encoder to fool a discriminator.
Two-Stage Training: Pre-training on common distortions followed by fine-tuning on specific deepfake models to ensure generalization.

3. Key Contributions

First Trifunctional Framework: LIDMark is the first method to unify Deepfake Detection, Tampering Localization, and Source Tracing into a single model and payload.
Novel Watermark Design (LIDMark): Introduces a 152-D composite watermark that structurally combines a sensitive geometric signal (landmarks) with a robust identifier, solving the capacity vs. robustness trade-off.
Factorized-Head Decoder (FHD): A novel architecture that decouples regression and classification tasks within a shared backbone, enabling robust reconstruction of heterogeneous data.
Intrinsic-Extrinsic Consistency Check: A blind-reference mechanism that enables pixel-level localization of tampering by comparing recovered original geometry against re-detected geometry.

4. Experimental Results

The framework was evaluated on CelebA-HQ and LFW datasets against state-of-the-art baselines (e.g., SepMark, DiffMark, KAD-Net).

Imperceptibility: Achieved superior visual quality (PSNR: 44.31 dB, SSIM: 0.99 at 256x256) despite embedding a massive 152-D payload, outperforming methods that only embed 30-D or 128-D payloads.
Deepfake Detection & Localization:
- Achieved an AUC of 0.9388 in distinguishing real images from deepfakes.
- Successfully localized tampered regions by identifying semantic components with high AED.
- Maintained low AED under common distortions (e.g., resize, blur) but showed high AED under deepfake attacks.
Source Tracing:
- Achieved the lowest Bit Error Rate (BER) across all baselines.
- Under severe deepfake attacks (e.g., SimSwap, UniFace), LIDMark maintained near-perfect recovery (BER < 1%), whereas competitors like CIN and EditGuard failed catastrophically (BER > 30-50%).
Generalization: The model trained on CelebA-HQ generalized effectively to the unseen LFW dataset, maintaining robustness and low BER.

5. Significance

Paradigm Shift: Moves the field from reactive, single-task detection to a proactive, holistic forensic solution.
Granular Analysis: Solves the "Where is it fake?" problem, providing actionable evidence for forensic analysts rather than just a binary "Real/Fake" verdict.
Efficiency: Eliminates the need for complex dual-decoder architectures, reducing computational overhead while increasing payload capacity.
Robustness: Demonstrates that high-capacity, multi-functional watermarks can be embedded imperceptibly and survive aggressive deepfake generation pipelines.

In conclusion, LIDMark represents a significant leap in proactive forensics, offering a unified, robust, and imperceptible solution to combat the escalating threat of deepfakes.