Video Deepfake Abuse: How Company Choices Predictably Shape Misuse Patterns
This paper argues that the release of open-weight video generation models trained on uncurated web data without adequate safeguards, combined with insufficient platform moderation, predictably drives the proliferation of non-consensual intimate imagery and child sexual abuse material, necessitating proactive risk management by developers and distributors to mitigate these harms.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Picture: The "Open Source" Double-Edged Sword
Imagine a world where anyone can build a high-powered 3D printer. In 2022, this happened with AI Image Generators (like Stable Diffusion). Suddenly, people could create incredibly realistic photos of anything they imagined.
This was great for artists and creators. But, just like a 3D printer can make a toy or a weapon, these AI tools were quickly used to make non-consensual intimate imagery (NCII)—fake photos of real people in compromising situations, including children.
The paper argues that the same thing is happening right now (in 2025) with AI Video Generators. We are crossing a threshold where making fake, realistic videos is becoming easy, cheap, and accessible to almost anyone.
The Main Problem: The "Recipe" vs. The "Chef"
The paper uses a supply chain analogy (Figure 1) to explain how these bad videos get made. Think of it like a cooking ecosystem:
- The Developers (The Chefs): Big companies (like Alibaba, Stability AI, etc.) create the base "recipe" (the AI model).
- The Distribution Platforms (The Pantries): Websites like Civitai and Hugging Face are where people download these recipes.
- The Modifiers (The Home Cooks): Thousands of people download the base recipe, tweak it, and add their own "spices" (fine-tuning) to make it better at specific things, like making NSFW (Not Safe For Work) content.
- The Users (The Diners): Millions of people download these tweaked recipes to make the final videos.
The Paper's Core Finding:
The paper found that a small number of "recipes" (models) are responsible for the vast majority of these bad videos. Specifically, models like Wan 2.x, Stable Video Diffusion, HunyuanVideo, and LTX-Video are being used disproportionately to create fake porn and abuse material.
Why Did This Happen? (The 2022 Lesson)
The paper looks back at 2022 to explain why this is happening now.
- The "Bad" Recipe: In 2022, a company released a model called Stable Diffusion 1.0. They gave away the "weights" (the code) for free, but they didn't filter the training data well. It was like teaching a chef using a cookbook that included recipes for poison. Because the data was messy, the AI learned to make bad stuff easily.
- The "Good" Recipe: Later, the same company released Stable Diffusion 2.0. They cleaned the cookbook, removing the "poison" recipes. The result? The AI became terrible at making NSFW content. The community actually complained that it was "too censored."
- The Lesson: When developers clean their data and add safety filters before releasing the model, it creates a huge barrier to making bad content. When they don't, the bad stuff spreads like wildfire.
The Current Situation: The "Cat is Out of the Bag"?
Some people argue: "Well, the bad models are already out there. It's too late. Even if we make safe models now, people will just use the old bad ones."
The paper says this is wrong. They use a Piracy Analogy:
- Think of illegal movie downloads. Even though pirated movies are technically available, search engines and companies work to make them harder to find. This doesn't stop everyone, but it stops the casual users and the teenagers who don't want to dig deep.
- Similarly, if developers and platforms make it harder to find and use bad AI models, the total amount of harm goes down significantly. It's about raising the "friction" (the effort required) to do bad things.
What Are the Solutions?
The paper suggests two main groups need to step up:
1. The Developers (The Chefs)
They need to stop releasing "naked" models. Instead, they should:
- Clean the Data: Make sure the AI isn't trained on porn or abuse material in the first place.
- Add Safety Nets: Use "unlearning" techniques to make the AI forget how to make bad stuff, even if someone tries to tweak it later.
- Test Themselves: Try to break their own models before releasing them to see if they can be misused.
- Be Transparent: Tell the public exactly what safety steps they took. Currently, most companies are silent about this (see Table 1 in the paper).
2. The Distribution Platforms (The Pantries)
Websites like Civitai and Hugging Face are the gatekeepers.
- They are currently hosting thousands of models specifically designed to make fake porn and child abuse material.
- The paper argues these platforms must actively police their shelves. If they don't remove these specific "bad recipes," they are amplifying the harm.
The Conclusion
The paper concludes that company choices matter.
- If developers release models without safety filters, they are predictably causing harm.
- If platforms don't remove models designed for abuse, they are making the problem worse.
It's not about banning AI. It's about risk management. Just like we don't ban cars because some people drive them dangerously, we don't need to ban AI video. But we do need to install brakes, seatbelts, and speed limits (safeguards) to prevent the worst outcomes. If companies take these steps now, they can drastically reduce the amount of fake, abusive content in the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.