How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

1. Problem Statement

The field of Trust & Safety (T&S)—which encompasses content moderation, policy enforcement, and user protection against online harms—is undergoing a paradigm shift due to Generative AI (GenAI). While the risks of GenAI misuse by attackers are widely acknowledged (e.g., deepfakes, automated scams), the potential for GenAI to empower defenders remains underexplored.

The core problem is the asymmetry in the sociotechnical arms race:

• Attackers are leveraging GenAI to scale the production of harmful content, lower the barrier to entry for malicious actors, and create sophisticated, personalized attacks that outpace existing detection systems.
• Defenders (platforms, law enforcement, civil society) face resource constraints, outdated detection mechanisms (e.g., hash-based matching), and a lack of operational playbooks to handle the volume and nuance of GenAI-generated threats.
• There is a critical gap in understanding how these dynamics will evolve across different domains (e.g., child safety vs. election integrity) and how defenders can strategically restructure their operations to leverage GenAI for defense.

2. Methodology

The authors conducted a qualitative participatory research study involving 43 Trust & Safety experts from Asia, Europe, and North America.

• Study Design: Five distinct workshops were held (one for each domain), lasting six hours each.
• Domains Studied:
  1. Child Safety (focusing on CSAM and grooming)
  2. Election Integrity (mis/disinformation)
  3. Hate and Harassment
  4. Scams (financial fraud, romance scams)
  5. Violent Extremism (VE)
• Participants: A mix of industry professionals, civil society representatives, academics, and law enforcement.
• Data Collection Techniques:
  • Memorable Experience Cards: Participants shared past interactions with GenAI.
  • Change Cards: Participants identified potential shifts in their domains over the next 1–2 years.
  • Attackers Discussion: Focused on how bad actors utilize GenAI.
  • Futuring Stories: Participants wrote optimistic narratives set in 2026 describing successful GenAI-enabled defense scenarios.
• Analysis: The team performed a reflexive thematic analysis on transcripts, artifacts (cards/stories), and video "clip reels" to identify cross-domain and domain-specific themes.

3. Key Contributions

The paper makes five primary contributions to the HCI and Trust & Safety literature:

1. Novel Qualitative Study: It provides the first comprehensive, multi-domain exploration of GenAI's impact from the perspective of frontline T&S experts.
2. Dual-Edged Narrative: It establishes a dominant narrative that GenAI is a "force multiplier" for both attackers and defenders, a view shared across all domains.
3. Co-evolution Framework: It describes the trajectory of the attacker-defender dynamic: GenAI initially enables attacks that overwhelm defenses, followed by a period where defenders leverage GenAI to develop advanced countermeasures.
4. Domain-Specific Nuance: It highlights structural differences in how GenAI impacts different domains (e.g., the immediate obsolescence of hash-based systems in Child Safety vs. the amplification of existing resource imbalances in Election Integrity).
5. Strategic Framework: It offers a roadmap for T&S organizations to transform operations, emphasizing the need for domain-specific interventions, cross-sector collaboration, and the integration of Responsible AI (RAI) principles.

4. Key Results and Findings

A. How GenAI Empowers Attackers

Experts identified five primary vectors of attack:

1. Scale and Speed: GenAI allows attackers to generate harmful content at a volume and speed that human review teams cannot match. It lowers the cost of operations (e.g., replacing "troll farms" with automated chatbots).
2. Lowered Barriers to Entry: Non-technical actors can now create sophisticated propaganda, deepfakes, and scams using accessible tools. "Jailbreaking" open-source models further democratizes access to harmful capabilities.
3. Enhanced Messaging: Attackers can generate high-quality, culturally tailored content in multiple languages simultaneously. They use "entertaining" content (memes, anime) to evade moderation filters while signaling extremist ideologies to in-groups.
4. Degradation of the Information Environment: The proliferation of synthetic media creates a "Liar's Dividend," where bad actors can dismiss real evidence as fake, eroding shared facts and trust.
5. Harmful Personal Relationships: Persona bots are used to build trust with vulnerable users (e.g., the lonely or elderly) to facilitate radicalization, sextortion, or scams.

B. How GenAI Empowers Defenders

Experts envision GenAI transforming defense through:

1. Detection and Mitigation at Scale: Using GenAI to automate the flagging of harmful content, generate synthetic training data for classifiers, and handle high-volume, low-complexity moderation tasks, freeing humans for nuanced "gray area" decisions.
2. Criminal Investigation: Leveraging GenAI to infiltrate extremist networks, disrupt attack chains (e.g., using deepfakes to deceive terrorist cells), and analyze complex financial/communication networks in scam investigations.
3. Moderator Wellbeing: Reducing human exposure to traumatic content by using AI for initial triage and prioritization.
4. Counternarratives: Using GenAI to craft persuasive, personalized counter-messaging and "nudges" to steer users away from radicalization or scams.
5. User Support: Deploying "comfort bots" to guide victims through reporting processes and provide emotional support.
6. Cross-Sector Collaboration: The threat of GenAI is seen as a catalyst for breaking down silos between platforms, law enforcement, and civil society to share signals and data in real-time.

C. Domain-Specific Variations

• Child Safety: Faces the most immediate disruption. GenAI-generated CSAM (including deepfakes of real victims) threatens to overwhelm hash-based detection systems (like PhotoDNA), requiring a fundamental restructuring of defense.
• Election Integrity: GenAI exacerbates existing problems of low-quality information and voter suppression. The challenge is less about new detection tech and more about the lack of resources to counter state-sponsored disinformation.
• Hate & Harassment: GenAI amplifies the volume of toxic content and widens the range of targets. Defense requires moving beyond simple keyword filtering to understanding cultural context and coded language.
• Scams: The for-profit nature of scams means a rapid "arms race." GenAI lowers the barrier for organized crime to scale operations, but also offers defenders tools to trace financial flows and network structures.
• Violent Extremism: GenAI enables the creation of "borderline" content that evades policy filters but recruits followers. It also allows for the automation of complex attack chains (recruitment, fundraising, weapon manufacturing).

5. Significance and Implications

The paper concludes that the Trust & Safety landscape is at a critical inflection point. The significance of the work lies in its call to action for the HCI and AI communities:

• Operational Transformation: T&S organizations must move away from "one-size-fits-all" playbooks. They need to invest in domain-specific GenAI experiments and cultivate in-house expertise to translate technical capabilities into operational reality.
• Responsible AI (RAI) Integration: The RAI community must collaborate with T&S experts to build "golden datasets" and benchmarks that reflect real-world adversarial tactics. Current safety guardrails often block defensive use cases (e.g., preventing researchers from analyzing CSAM patterns), creating an imbalance where attackers have more flexibility than defenders.
• Strategic Investment: There is an urgent need to fund the development of GenAI tools for defense, particularly for cross-sector signal sharing and the automation of investigative workflows.
• Human-Centric Design: While automation is necessary for scale, the paper emphasizes that human judgment remains essential for nuanced adjudication, moderator wellbeing, and ethical oversight.

In summary, the paper argues that while GenAI poses an existential threat to current Trust & Safety models, it also holds the key to a more robust, scalable, and effective defense ecosystem—if organizations can rapidly adapt their strategies, policies, and technologies.