🤖 AI

A Marketplace for AI-Generated Adult Content and Deepfakes

This paper analyzes a 14-month dataset from the Civitai platform to reveal that its monetized bounty system disproportionately incentivizes the creation of non-consensual, NSFW deepfakes targeting female celebrities, thereby highlighting significant gendered harms and governance challenges in community-driven generative AI marketplaces.

Original authors: Shalmoli Ghosh, Matthew R. DeVerna, Filippo Menczer

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shalmoli Ghosh, Matthew R. DeVerna, Filippo Menczer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling digital marketplace called Civitai. Think of it like a giant, high-tech flea market where people don't just buy and sell physical goods, but instead trade in AI-generated images and tools.

In this market, there is a special section called "Bounties." You can think of a Bounty like a "Wanted" poster, but instead of a criminal, you are looking for a specific picture or AI tool. A person (the "Supporter") posts a request, says, "I want an image of a cat wearing a tuxedo," and puts a cash reward (in the form of virtual coins called "buzz") on the table. Other users (the "creators") compete to make that image. The first one to do it right wins the money.

The researchers in this paper decided to investigate this marketplace to see what people are actually asking for, who is asking, and how the rules are being followed. They looked at every single "Wanted" poster posted over 14 months. Here is what they found, broken down simply:

1. The "Adult" Section is Taking Over

The researchers found that while people ask for all kinds of things (like art or logos), a huge chunk of the requests are for NSFW (Not Safe For Work) content—basically, adult or explicit material.

The Trend: At the beginning of their study, these requests were a minority. But over time, they grew steadily. By the end of the 14 months, nearly half of all requests were for adult content.
The Tool: Most of these requests aren't just for a single picture; they are asking for LoRAs. Think of a LoRA as a "special instruction manual" or a "filter" that you can download and use forever. Once someone creates a LoRA that can generate a specific type of adult image, anyone can use that tool to make infinite variations of it. The marketplace is dominated by people trying to buy these "instruction manuals" to steer AI models to do things they weren't originally designed to do.

2. The "Deepfake" Problem: Who is Being Targeted?

A "Deepfake" in this context is a request to generate an image of a real, living person (like a celebrity) who didn't actually pose for that picture.

The Gender Gap: The researchers found a shocking imbalance. If you look at all the requests to generate images of real people, 9 out of 10 are of women. The vast majority of these targets are female celebrities, actresses, and influencers.
The "Self" Loophole: Some people tried to get around the rules by claiming, "I am asking for a picture of myself" or "my wife." The researchers found these claims were often impossible to verify and didn't actually solve the problem of consent.

3. The "Super-Requesters"

The study looked at who is doing all the posting. It's not a fair, equal distribution where everyone posts a little bit.

The 20% Rule: About 20% of the users are responsible for roughly half of all the requests.
Deepfake Concentration: This concentration is even stronger for deepfakes. A very small group of people is driving most of the requests to generate images of real people.

4. The "Warning Sign" That Isn't Always There

Civitai has a rule: If you ask for a picture of a real person, the system is supposed to put a warning sign on the page. This sign says, "Hey, this is a real person. We only allow safe images, and the person can ask us to take this down."

The Gap: The researchers checked to see if this warning sign actually appeared.
- For "Safe" (SFW) requests of real people, the sign showed up 86% of the time.
- For "Adult" (NSFW) requests of real people, the sign was missing 42% of the time.
The Takeaway: Even though the platform has a rule, the "warning sign" is often missing exactly when it's needed most—when the requests are for explicit content.

Summary

The paper concludes that this marketplace has evolved into a place where a small group of users is heavily driving demand for adult content and images of real women, often using special AI tools (LoRAs) to bypass safety filters. While the platform has rules and warning signs, the enforcement is inconsistent, especially for the most sensitive and potentially harmful requests.

Important Note: The researchers only looked at the requests (what people asked for), not the final pictures that were created, nor the personal reasons why people made these requests. They simply mapped out the "menu" of what was being ordered in this digital restaurant.

Technical Summary: A Marketplace for AI-Generated Adult Content and Deepfakes

Problem Statement
Generative AI has democratized the creation of realistic synthetic media, leading to the emergence of community-driven platforms that blend social interaction with content generation. While platforms like Civitai facilitate the sharing of models and outputs, they also host a monetized "Bounties" feature where users commission specific AI-generated content in exchange for payment. This creates a structured marketplace coordinating demand, labor, and reward. However, the dynamics of this marketplace remain largely unexplored, particularly regarding the types of content incentivized, the prevalence of "Not Safe For Work" (NSFW) material, and the use of the system to generate non-consensual deepfakes (media depicting identifiable real individuals). Existing research has focused on generated images and models but has not systematically analyzed the demand-side signals and incentive structures of such bounty systems.

Methodology
The authors conducted a longitudinal analysis of all publicly available bounty requests on Civitai over a 14-month period (October 15, 2023 – January 30, 2025), resulting in a dataset of 4,847 bounties.

Data Collection: A custom pipeline using Python Selenium automated browser sessions to capture bounty metadata (titles, descriptions, timestamps, engagement metrics) and example images. Data collection occurred in two passes: a "blurred" pass to capture platform-provided explicitness ratings (PG, PG-13, R, X, XXX) and an "unblurred" pass to capture image URLs.
Classification Strategy:
- SFW/NSFW Labeling: Bounties were classified as NSFW if any example image carried an "R," "X," or "XXX" rating. To validate this, the authors compared Civitai's labels against OpenAI's content moderation API, finding that Civitai adopts a more conservative classification approach.
- Thematic Analysis: For Safe For Work (SFW) bounties, the authors employed an automated thematic classification pipeline. After manual annotation of a 150-bounty sample to establish a taxonomy (including categories like "Human deepfake," "Fictional characters," and "NSFW"), GPT-4.1 was selected for large-scale classification based on higher inter-rater reliability (Krippendorff's $\alpha$ = 0.8) compared to GPT-4o.
- Deepfake Identification: A two-step process identified deepfake requests: (1) GPT-4.1 analyzed SFW bounties for "Human deepfake" themes, and (2) titles and descriptions of NSFW bounties were analyzed for potential deepfake content. All AI-identified cases underwent manual review by a single human annotator to verify the targeting of real individuals, resulting in a confirmed set of 323 deepfake bounties.
Concentration Analysis: The authors quantified user participation concentration using Lorenz curves and Gini coefficients across SFW, NSFW, and deepfake categories.
Intervention Assessment: The study measured the effectiveness of Civitai's informational notice (a warning regarding real-person likeness) by calculating the proportion of deepfake bounties displaying this marker.

Key Contributions and Results
The study provides the first systematic characterization of a monetized, community-driven generative AI marketplace, revealing the following:

Marketplace Composition: The bounty system is dominated by requests for LoRAs (Low-Rank Adaptations) and images, which together constitute 92% of all bounties. LoRAs are significant as they allow users to steer models toward behaviors they were not originally trained to perform, effectively bypassing guardrails.
Prevalence and Growth of NSFW Content: NSFW requests are widespread and increasing. By the end of the study period, NSFW bounties comprised a majority of weekly requests (surpassing 50% by September 2024). Specifically, 48% of all bounties were NSFW, with video bounties showing the highest proportion (69%).
Deepfake Targeting: Deepfake requests constitute a non-negligible share of the market. There is a pronounced gender asymmetry: 95.8% of deepfake targets are female, with actors and actresses being the most frequently targeted profession.
User Concentration: Participation is uneven. Approximately 20% of requesters account for roughly half of all requests. This concentration is most pronounced in the deepfake category (Gini coefficient = 0.45), suggesting a small group of highly active users drives deepfake-related activity.
Enforcement Gaps: Civitai's intervention mechanism (an informational alert on deepfake bounties) is applied inconsistently. While 85.6% of SFW deepfake bounties displayed the alert, only 58.3% of NSFW deepfake bounties did so, leaving a substantial fraction of high-risk requests unmarked. Furthermore, despite a formal policy update in August 2024 prohibiting depictions of real individuals, such requests persisted on the platform.

Significance and Claims
The paper claims to offer a novel empirical lens into how monetized, community-driven generative AI platforms structure participation and enforce safeguards. The authors argue that their findings demonstrate how such marketplaces can produce gendered harms, specifically through the disproportionate targeting of women in deepfake requests.

The study highlights that the "Bounties" feature creates a direct economic incentive for generating content that may violate platform policies or ethical norms, including non-consensual intimate imagery. The authors conclude that the current governance model, which relies heavily on user self-reporting and inconsistent automated flagging, is insufficient. They suggest that effective governance requires combining content-based rules with behavioral signals (such as user activity concentration) and proactive monitoring. The paper does not propose new technical solutions but rather calls for closer scrutiny of platform incentives and the need for regulators and moderators to address the gaps between policy and enforcement.