From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI

Technical Summary: From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI

Problem Statement
The paper addresses the evolving security and safety landscape of Generative AI (GenAI) as it transitions from producing synthetic content (text, images, audio, video) to executing autonomous actions through tool chains and external APIs. While existing literature extensively covers content-level threats (e.g., deepfakes, disinformation) and model-level attacks (e.g., poisoning, jailbreaking), there is a lack of an end-to-end analytical framework that connects these domains with the emerging risks of "agentic" systems. The authors argue that the delegation of consequential real-world actions to autonomous agents introduces an operational risk frontier that expands attacker access, increases system autonomy, and widens the scope of potential harm beyond traditional content-generation paradigms.

Methodology
This work is an analytical review rather than a systematic survey or empirical study. The authors employ a synthetic analysis approach, integrating evidence from peer-reviewed studies, institutional risk reports (e.g., International AI Safety Reports), vendor security disclosures, standards documents (e.g., OWASP, C2PA), and documented incidents.

• Analytical Lens: The paper utilizes a "content-to-action continuum" to organize threats, moving from static artifacts to dynamic operations.
• Taxonomy Construction: The authors develop a threat taxonomy organized by three escalation dimensions:
  1. Attacker Access: Ranging from direct model input to third-party content and cross-system tool chains.
  2. Victim Agency: Ranging from passive content consumption to autonomous multi-step execution with tool permissions.
  3. Impact Scope: Ranging from single output manipulation to cross-system cascading effects.
• Comparative Analysis: The study compares the timing of capability deployment against defensive responses and governance frameworks, drawing parallels with historical transitions in SQL injection, mobile security, and IoT.

Key Contributions
The paper makes four primary contributions to the field:

1. Security–Safety Framing: It distinguishes between security failures (often addressed by technical countermeasures) and safety harms (which depend more on platform policy, cross-institutional coordination, and legislation). The authors argue that governance delays disproportionately impact the safety side of the equation.
2. Cross-Layer Analytical Synthesis: It unifies content-level weaponization, model-level attacks, and agentic exploitation into a single framework. This highlights how architectural shifts (e.g., the introduction of the Model Context Protocol, MCP) expand the attack surface and change the nature of downstream harm.
3. Integrated Technical and Governance Analysis: The paper demonstrates that several promising technical defenses (e.g., provenance mechanisms, watermark verification, MCP safeguards) presuppose forms of institutional coordination that current governance arrangements do not yet provide. It identifies a "structural" governance gap, not merely a temporal one.
4. Externalized Model Use as a Paradigm Shift: It reframes prompt injection and agentic attacks not as isolated vulnerabilities but as consequences of a broader shift where models access external data and execute actions. Consequently, defenses designed for input–output systems do not cleanly transfer to connected, action-bearing environments.

Key Results and Findings

• Capability Escalation: Modern GenAI systems exhibit high-fidelity generation (human detection accuracy near chance levels, ~62%), low-cost scalable production (marginal costs <$1/M tokens), fine-grained controllability (identity adaptation), and autonomous agency (tool use). These properties collectively lower the barrier for malicious actors to conduct large-scale, targeted attacks.
• Threat Taxonomy (Levels 1–5): The authors categorize adversarial attacks into five escalating levels:
  • Level 1 (Direct Prompt Injection): Hijacking goals or leaking prompts via direct input.
  • Level 2 (Jailbreaking): Bypassing safety guardrails via crafted adversarial inputs (e.g., GCG, PAIR, Crescendo).
  • Level 3 (Indirect Prompt Injection): Embedding malicious instructions in third-party content (e.g., PoisonedRAG) that the model retrieves.
  • Level 4 (Agentic Exploitation): Exploiting tool chains and MCP servers (e.g., tool poisoning, EchoLeak) to execute malicious actions.
  • Level 5 (Multi-Agent Exploitation): Emerging threats involving inter-agent trust exploitation and cross-organizational cascading compromise.
• Defense Limitations:
  • Detection: Current detectors for text, image, and audio suffer from distributional shifts, fragility to post-processing, and an inability to distinguish human editing from generation.
  • Watermarking: Faces a fundamental trade-off between robustness, quality, and detectability. Watermarks are often stripped by platforms or removed via regeneration attacks.
  • Alignment: Safety training is superficial; "shadow alignment" can be removed with minimal cost (e.g., <$200 or few adversarial examples), and alignment often degrades reasoning capabilities.
  • Agentic Security: Defenses like instruction hierarchies and runtime verification (e.g., AgentSpec) are nascent. Tool poisoning achieves high success rates (e.g., 72.8% ASR in MCPTox benchmarks), and more capable models are paradoxically more susceptible to indirect prompt injection.
• Governance Gap: The paper documents a significant lag between capability deployment and governance. For instance, the MCP protocol launched without authentication in November 2024, while the first agentic governance frameworks (e.g., IMDA, NIST initiatives) appeared in early 2026. This 14-month gap is shorter in absolute terms than historical precedents (e.g., SQL injection took ~6 years) but occurs against a backdrop of much faster capability growth, leading to more severe practical consequences.

Significance and Claims
The paper claims that the shift from content generation to agentic action fundamentally reshapes the threat surface. It argues that the "deployment-to-governance" ordering observed in AI security is consistent with historical technological transitions but is accelerating. The significance of this work lies in its assertion that technical defenses alone are insufficient; they often rely on institutional coordination (e.g., standardized watermarking, trusted registries for MCP servers) that does not yet exist.

The authors conclude that as systems move from generating content to executing real-world actions, the potential for irreversible harm increases because governance gaps allow agents to perform file writes, API calls, or financial transfers before human or automated review can intervene. They posit that future research must address formal agent trust models, instruction-data separation, and the tension between capability and security, particularly as communication moves from discrete tokens to continuous vector flows in latent reasoning.