💻 computer science

IP-Bench: Benchmark for Image Protection Methods in Image-to-Video Generation Scenarios

This paper introduces IP-Bench, the first systematic benchmark designed to evaluate the effectiveness and robustness of image protection methods against misuse in image-to-video generation scenarios by testing them across multiple models, attack strategies, and transferability conditions.

Original authors: Xiaofeng Li, Leyi Sheng, Zhen Sun, Zongmin Zhang, Jiaheng Wei, Xinlei He

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaofeng Li, Leyi Sheng, Zhen Sun, Zongmin Zhang, Jiaheng Wei, Xinlei He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, unique painting. You love it, but you're worried that someone might use a magical "copy machine" to turn your painting into a fake, moving video of a celebrity saying something they never said, or a news anchor reporting fake news. This is the problem of Image-to-Video (I2V) misuse.

To stop this, scientists have invented "digital invisible ink." They add tiny, invisible scratches to your painting. To your eyes, the painting looks perfect. But when the magical copy machine tries to scan it, the scratches confuse the machine, and the video it produces comes out as a glitchy, unrecognizable mess.

However, until now, nobody had a standardized test to see which "invisible ink" actually works best. Some inks might ruin the painting's beauty, while others might be wiped off by a simple rainstorm (or in this case, a simple photo edit).

Enter IP-Bench. Think of IP-Bench as the "Consumer Reports" or "Car Crash Test" for these digital protection methods.

Here is what the paper discovered, explained simply:

1. The Big Trade-Off: The "Scary Mask" vs. The "Pretty Face"

The researchers tested 6 different types of "invisible ink" against 5 different "magical copy machines." They found a frustrating rule: You can't have it both ways.

The "Scary Mask" (High Protection, Low Quality): One method, called Mist, is like wearing a heavy, terrifying mask. It completely stops the copy machine from working (it's very effective), but the mask is so heavy and ugly that your painting looks terrible and distorted.
The "Pretty Face" (Low Protection, High Quality): Another method, called EditShield, is like wearing a very light, invisible veil. Your painting looks perfect, but the copy machine sees right through it and makes a perfect fake video. It offers almost no protection.
The Middle Ground: Most other methods try to balance this, but they usually end up being mediocre at both protecting the image and keeping it looking good.

2. The "Rainstorm" Test (Robustness)

The researchers asked: What happens if someone tries to wash off the invisible ink? They simulated two common "attacks":

The "Compression" Attack (JPEG): Imagine taking your protected painting, squishing it into a smaller file size (like sending a text message), and then opening it again. This process acts like a powerful eraser.
- Result: Almost all the "invisible inks" were wiped away! The protection vanished, and the copy machine started working again. It turns out, most of these inks are painted on the "surface" (high-frequency details) that get smoothed out when you compress an image.
The "Noise" Attack (Gaussian Noise): Imagine sprinkling static dust over the painting.
- Result: This usually made the protection even worse, confusing the image further. However, one method (I2VGuard) was weirdly resilient; the dust actually seemed to make its protection stronger on some machines, like a chameleon blending in better when the background gets messy.

3. The "Swiss Army Knife" Problem (Transferability)

The researchers wanted to know: If I protect my painting for Machine A, will it also work on Machine B?

The Answer: Mostly No.
- The "ink" designed for one specific type of copy machine (like a U-Net) didn't work well on a different type (like a DiT). It's like having a key that opens your front door but doesn't open your back door.
- The Exception: Methods originally designed for still images (Image-to-Image) actually worked surprisingly well when moved to video (Image-to-Video). It seems the "foundation" of these machines is similar enough that a key made for one often fits the other.

The Main Takeaway

The paper concludes that we are currently in a "Goldilocks" zone where we haven't found the perfect solution yet.

If you want strong protection, you have to accept a ugly, distorted image.
If you want a beautiful image, you have to accept that hackers can still make fake videos.
And if someone just compresses the image (like saving it as a JPEG), almost all current protections disappear.

IP-Bench is the first tool that helps us measure these flaws clearly. It tells us that future inventions need to be smarter: they need to hide the "ink" deeper inside the image so it survives compression, and they need to find a way to protect the image without ruining its beauty.

In short: We have the tools to fight fake videos, but right now, they are either too ugly to use or too weak to stop the bad guys. IP-Bench is the scoreboard that tells us exactly how much work we have left to do.

1. Problem Statement

The rapid advancement of Image-to-Video (I2V) generation models (e.g., SVD, HunyuanVideo, Wan2.2) has lowered the barrier for creating high-quality videos, but it has also introduced significant security risks, such as the creation of deepfakes and the dissemination of misinformation.

The Gap: While various image protection methods exist to prevent unauthorized editing in Image-to-Image (I2I) scenarios (e.g., PhotoGuard, Mist), they have not been systematically evaluated in I2V contexts.
The Challenge: Existing evaluations lack a unified benchmark, often focusing on single models or metrics. Furthermore, current methods have not been rigorously tested against preprocessing attacks (e.g., compression, noise) or their ability to transfer across different I2V architectures (U-Net vs. Diffusion Transformers) and modalities (I2I to I2V).

2. Methodology: IP-Bench Framework

The authors propose IP-Bench, the first systematic benchmark designed to evaluate image protection methods specifically for I2V generation.

A. Experimental Setup

Protection Methods Evaluated (6):
- I2V-Specific: I2VGuard (targets temporal attention/VAE), VGMShield (feature-space PGD).
- I2I-Based (adapted): EditShield (EOT optimization), MIST (fused adversarial loss), PhotoGuard (encoder perturbation).
- Baseline: Random Noise.
Target Models (5): A diverse set of state-of-the-art I2V models covering different architectures:
- U-Net: SVD (Stable Video Diffusion).
- Diffusion Transformers (DiT): LTX, Skyreel, CogVideoX, Wan2.2.
Dataset: Flickr30k-MSCOCO-300 (300 high-quality images) with prompts generated by Qwen2.5-VL-7B-Instruct to ensure semantic alignment.
Threat Model: An adversary attempts to generate high-quality videos from protected images, potentially using preprocessing attacks (JPEG Compression, Gaussian Noise) to neutralize perturbations.

B. Evaluation Metrics

The benchmark assesses four key dimensions:

Protection Effectiveness:
- VBench Suite: Measures Subject Consistency, Motion Smoothness, Aesthetic Quality, and Imaging Quality. A "Degradation Rate" is calculated to quantify performance drops.
- Fréchet Video Distance (FVD): Quantifies the distributional shift between videos generated from original vs. protected images.
Visual Fidelity:
- Measures imperceptibility using PSNR, SSIM, LPIPS, CLIP Similarity, and FID.
Robustness:
- Tested against Elimination-based attacks (JPEG Compression, QF=75) and Addition-based attacks (Gaussian Noise, $\sigma=0.01$ ).
Transferability:
- Cross-Model: Performance when a method trained for one architecture is applied to another.
- Cross-Modality: Performance when I2I protection methods are applied to I2V generation.

3. Key Contributions

First Systematic Benchmark: IP-Bench establishes a reproducible framework for evaluating protection methods in I2V scenarios, covering effectiveness, fidelity, robustness, and transferability.
Discovery of Fundamental Trade-offs: The benchmark reveals a critical trade-off: methods that strongly degrade model performance (high protection) often severely compromise visual fidelity (low imperceptibility).
Robustness Analysis: It demonstrates that most existing methods are highly vulnerable to simple preprocessing attacks, particularly JPEG compression, which effectively "purifies" adversarial perturbations.
Open Framework: The authors release an extensible code framework to facilitate future research and the integration of new protection/attack strategies.

4. Key Results & Findings

A. Protection Effectiveness vs. Visual Fidelity

Mist: Provides the strongest protection (highest degradation rate ~6.6% on VBench, highest FVD) but suffers from poor visual quality (lowest PSNR/SSIM, high FID). It relies on high-magnitude perturbations that distort the image.
EditShield: Offers the best visual fidelity (PSNR: 38.31, SSIM: 0.955) but has negligible protection (degradation rate ~0.63%), failing to outperform random noise in many cases.
I2VGuard: Shows notable robustness against preprocessing attacks, maintaining a degradation rate above 0.02 across most models, though with moderate visual quality loss.
Conclusion: There is an inherent tension; increasing the perturbation budget improves protection but degrades image quality.

B. Robustness Against Preprocessing Attacks

JPEG Compression: Acts as a powerful "purification" step. It eliminates high-frequency adversarial signals, causing the protection effectiveness of almost all methods (except Mist, which remains partially effective) to drop significantly.
- Insight: Most perturbations reside in high-frequency bands (invisible to humans but targeted by JPEG quantization), making them fragile.
Gaussian Noise: Generally degrades protection effectiveness further by masking perturbations. However, I2VGuard showed an anomalous increase in effectiveness on some models (e.g., CogVideoX), suggesting its specific perturbation patterns align with temporal attention vulnerabilities in a way that random noise exacerbates.

C. Transferability

Cross-Model: Most methods show limited cross-model transferability. Performance variance is driven more by the target model's inherent sensitivity to perturbations than by the specific protection algorithm.
Cross-Modality: Image-level methods (I2I) transfer well to Video (I2V). Methods like Mist and PhotoGuard, which target static components (VAE, U-Net) common to both I2I and I2V, remain highly disruptive in video generation without needing complex temporal optimization.

5. Significance and Future Directions

Significance: IP-Bench highlights that current protection strategies are fragile in real-world deployment where images undergo compression or noise. It underscores the need for frequency-aware optimization (embedding perturbations in resilient mid-frequencies) rather than just high-frequency noise.
Limitations & Future Work:
- The benchmark currently lacks closed-source models (e.g., Sora, Kling), limiting black-box evaluation.
- Prompt distribution relies on a single LLM; future work should include human-crafted prompts.
- Standardized noise budgets and computational complexity analysis are needed for fairer comparisons.

In summary, IP-Bench provides a critical reality check for the field of AI safety, demonstrating that while image protection is possible, achieving a balance between robustness, imperceptibility, and cross-architecture generalization remains a significant unsolved challenge.