Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval

Imagine you have a priceless, original painting. A digital thief comes along, uses a magic eraser to remove a bird from the sky, and paints a fake bird in its place.

The Problem:
Most current security systems are like museum guards. Their only job is to spot the forgery. They can point at the fake bird and say, "Hey, that's not the original!" But they can't tell you what the real bird looked like, nor can they fix the painting. They just raise an alarm.

Other older systems tried to be "self-healing." They would hide a tiny, secret copy of the whole painting inside the paint itself. But this was like trying to hide a library inside a single grain of sand. If the painting got even a little bit dirty (compressed or edited), the secret library got crushed, and the healing failed.

The Solution: The "Multi-Scale Secret Blueprint"
This paper introduces a new system called Multi-Scale Hidden-Code Recovery. Think of it not as hiding the whole painting, but as hiding a smart, compressed blueprint that allows you to rebuild the missing parts perfectly.

Here is how it works, using simple analogies:

1. The "Zoom-Out" Blueprint (Multi-Scale Quantization)

Instead of trying to hide every single pixel of the original image (which is too much data), the system breaks the image down into layers of detail, like zooming out on a map.

Level 1 (The Big Picture): "There is a bird in the sky."
Level 2 (The Shape): "It's a blue bird with a curved beak."
Level 3 (The Details): "It has specific feather patterns."

The system hides these "levels" of information inside the image. Crucially, it uses a special training trick (called Dropout) to make sure that even the "Big Picture" level contains useful clues. This means if the thief destroys the fine details, the system can still remember the general shape and color of the bird from the lower levels.

2. The "Plug-and-Play" Adapter

One of the coolest features is that this system doesn't care how the image was protected.

Scenario A: You protect the image after it's made (Post-Hoc).
Scenario B: You protect the image while it's being generated by AI (In-Generation).

Think of this system as a universal power adapter. Whether you are using a US plug or a UK plug (different watermarking methods), this system fits right in and works immediately without needing to rebuild the whole wall.

3. The "Detective & Architect" Team (Recovery)

When a tampered image arrives, the system runs a two-step process:

The Detective (Localization): It scans the image to find exactly where the "fake bird" is. It draws a mask over the damaged area.
The Architect (Conditional Transformer): It looks at the secret blueprint hidden inside the image. It sees the "Big Picture" clues (Level 1 & 2) and the "Shape" clues (Level 3). It then uses an AI architect to reconstruct the missing bird, filling in the blank space with the original bird, not a guess.

4. The "Fact-Check" (Factual Retrieval)

The paper also introduces a new way to test if this works. They created a massive library called ImageNet-S.
Imagine you have a recovered photo of a bird, but you aren't 100% sure it's the exact original bird. The system doesn't just try to match the pixels; it asks: "Does this recovered bird look like the concept of the original bird?"
It uses a "semantic search" (like Google Images for concepts) to find the original photo in a database of millions. If the system can find the original photo just by looking at the repaired version, it proves the recovery is factually accurate.

Why This Matters

Robustness: Even if the image is compressed, blurred, or edited, the "blueprint" survives because it's hidden in a smart, layered way, not just in the fragile corners of the pixels.
Efficiency: It hides a tiny amount of data (the blueprint) instead of a massive amount (the whole image), making it much harder for attackers to destroy.
Truth: It moves beyond just saying "This is fake" to actually saying "Here is what the truth looked like."

In a Nutshell:
This paper teaches computers how to be digital restorers. Instead of just spotting a forgery, they can now look at a damaged, fake image, read the secret "multi-scale blueprint" hidden inside, and rebuild the original truth with high accuracy. It's like having a time machine that can repair a broken vase just by looking at the shards and a hidden instruction manual.

1. Problem Statement

The rapid advancement of generative AI (e.g., Stable Diffusion) has made digital image manipulation ubiquitous, challenging traditional methods of content authenticity. Current research primarily focuses on deepfake detection (identifying if an image is fake) and localization (highlighting tampered regions). However, a critical gap remains: recovering the original, untampered content for factual retrieval.

Existing recovery methods suffer from significant limitations:

Self-Recovery Watermarking: Traditional methods embed large amounts of pixel-level data (e.g., MSBs, block averages) directly into the image. This requires high bit-per-pixel (BPP) rates, making the watermark fragile against common degradations like compression or noise. Furthermore, the "coincidence problem" (where tampering destroys both the original data and its hidden recovery code) renders them unreliable.
Deepfake Recovery (Non-Watermarking): Methods like DFREC require paired training data (original vs. tampered) and prior knowledge of tampering regions, which is impractical in real-world scenarios where tampering types are unknown.
Scalability: Most existing recovery techniques are limited to specific domains (e.g., facial swapping) rather than general natural images.

The authors propose a unified framework that moves beyond mere detection to enable robust recovery and factual retrieval of the original image content, even when the tampering process is unknown.

2. Methodology

The proposed framework introduces a Multi-Scale Hidden-Code strategy that integrates seamlessly with both post-hoc and in-generation watermarking pipelines.

A. Multi-Scale Latent Quantization

Instead of embedding raw pixel data, the method encodes the image into a compact, discrete latent representation using a Vector Quantized Variational Autoencoder (VQ-VAE).

Multi-Scale Token Maps: The image is decomposed into a hierarchy of token maps ( $z_{s1}, \dots, z_{sK}$ ) at different resolutions (coarse to fine).
Dropout Training Strategy: To ensure semantic information is distributed across all scales (rather than concentrated only in the finest scales), the authors employ a dropout-based training strategy during VQ-VAE training. This forces the model to encode meaningful global structure even at lower resolutions, which is crucial for recovery when high-resolution tokens are lost.
Binary Bitstream: These quantized tokens are flattened and converted into a binary bitstream ( $h$ ). This drastically reduces the information volume compared to raw pixel embedding while preserving semantic fidelity.

B. Plug-and-Play Compatibility

The framework is designed to be agnostic to the underlying watermarking mechanism:

Post-Hoc Watermarking: It can be integrated with frameworks like EditGuard, where the hidden code is embedded after image generation.
In-Generation Watermarking: It can be adapted for diffusion-based generation (e.g., Gaussian Shading) by optimizing the initial noise vector ( $x_T$ ) to encode the hidden code before synthesis, using DDIM inversion.

C. Conditional Transformer Recovery

When a tampered image ( $I_d$ ) is encountered, the recovery process involves three steps:

Extraction: The hidden bitstream ( $h'$ ) and a tampering localization mask ( $M'_{loc}$ ) are extracted from the tampered image.
Patchification: The pixel-level localization mask is downsampled to align with the token-level resolution.
Conditional Autoregressive Reconstruction: A Conditional Transformer (based on the VAR architecture) reconstructs the image.
- It uses the extracted low-scale tokens ( $h'$ ) as a "clean prefix."
- It predicts missing high-scale tokens sequentially.
- Fusion Mechanism: The model fuses the predicted "clean" tokens with the "tampered" tokens from the input image, guided by the localization mask. Regions identified as tampered are replaced with model-predicted features, while intact regions are preserved.

D. Factual Retrieval

To evaluate the quality of recovery, the paper introduces Factual Retrieval. Instead of just measuring pixel similarity (PSNR/SSIM), it measures whether the recovered image can retrieve the original image (or its semantic label) from a large dataset using CLIP cosine similarity. This tests if the recovered image retains the correct semantic identity.

3. Key Contributions

Multi-Scale Hidden-Code Framework: A novel approach that encodes semantic information into compact, multi-scale quantized tokens, balancing capacity, imperceptibility, and robustness.
Plug-and-Play Integration: The first framework to demonstrate seamless compatibility with both post-hoc and in-generation watermarking paradigms, allowing flexible deployment across different protection pipelines.
ImageNet-S Benchmark: The authors constructed ImageNet-S, a new benchmark dataset providing triplets of (clean image, semantic label, tampering mask) to standardize the evaluation of recovery and retrieval tasks.
Conditional Recovery Strategy: A recovery mechanism that leverages a conditional Transformer to fuse extracted hidden codes with localization masks, enabling high-fidelity reconstruction without prior knowledge of tampering types.

4. Experimental Results

The method was evaluated on the ImageNet-S dataset using Stable Diffusion for tampering generation.

Retrieval Accuracy: The proposed method significantly outperformed baselines (HiNet, RePaint, VQGAN, VAR).
- Top-1 Label Accuracy: Achieved 92.31% (vs. 67.54% for VAR and 23.93% for HiNet).
- Top-1 Image Accuracy: Achieved 87.44%, demonstrating the ability to retrieve the exact original instance.
Robustness: The method maintained near-perfect bit accuracy (>99%) under various degradations (JPEG compression, Gaussian blur, noise), whereas traditional self-recovery watermarking dropped to ~50% accuracy.
Image Quality: The recovered images showed high structural similarity (SSIM ~0.82) and visual coherence, successfully restoring fine details (e.g., bird feathers, ostrich textures) that other methods failed to reconstruct.
Forgery Resistance: The "Content-Dependent Watermark" (CDW) design effectively prevented watermark transplantation. When attackers attempted to forge watermarks by applying extracted codes to different images, the Bit Accuracy dropped to ~0.5 (random guessing), and CLIP similarity scores confirmed semantic mismatches.

5. Significance

This work represents a paradigm shift in image authenticity research:

From Detection to Restoration: It moves the field beyond simply flagging fake images to actively restoring the truth, which is critical for legal, journalistic, and historical evidence.
Scalability: By using latent quantization rather than pixel embedding, it solves the capacity and robustness trade-off that has plagued self-recovery watermarking for decades.
Generalization: Unlike previous methods limited to faces, this approach works for general natural images with arbitrary tampering regions.
Standardization: The introduction of ImageNet-S provides a necessary benchmark for future research in image recovery and factual retrieval, addressing a lack of standardized evaluation metrics in this domain.

In summary, the paper presents a robust, scalable, and plug-and-play solution for recovering the "truth" from deepfakes, establishing a foundation for general-purpose image recovery systems.