Technical Summary: Security and Privacy in Agentic AI

Problem Statement

The rapid evolution of Artificial Intelligence from passive assistance to proactive, autonomous action—termed Agentic AI—introduces fundamental challenges to existing security and privacy paradigms. Unlike traditional software or static models, agentic AI systems possess the capacity to plan, coordinate, and execute complex, multi-step workflows with reduced human oversight. This shift creates a "responsibility gap" where legal compliance, accountability, and privacy protections become difficult to enforce because actions are distributed across fragmented supply chains involving foundation models, orchestration layers, third-party tools, and dynamic user data.

Current frameworks, developed for bounded and deterministic systems, struggle to address:

• Distributed Liability: The inability to pinpoint who controls a decision or bears responsibility when an AI agent delegates tasks across multiple actors and tools.
• Dynamic Consent: The inadequacy of "one-shot" consent models for workflows that involve cascading sub-actions, fallbacks, and interactions between multiple AI agents.
• Emergent Vulnerabilities: New forms of risk arising from the intersection of human-AI dependency, where agents may manipulate users, inherit biases, or create "gaslighting" scenarios through memory alteration.
• Non-Deterministic Failure: The difficulty of auditing, certifying, and recovering from failures in systems that are non-deterministic, self-modifying, and capable of strategic deception.

Methodology

The paper reports the outcomes of a horizon-scanning exercise conducted over three consecutive days (March 11–13, 2026) at the Universitat Politècnica de València. The study convened 30 leading international experts from academia, industry, and government across Canada, China, the EU, the UK, and the US.

The methodology followed a structured collaborative process:

1. Preparation: Participants were briefed on the agentic AI landscape using illustrative examples (e.g., OpenClaw).
2. Grouping: Participants were divided into three stable groups, each supported by two facilitators with expertise in Human-Computer Interaction (HCI), security, and privacy.
3. Process: The workshop utilized an affinity mapping approach, combining individual brainstorming with small-group and whole-group synthesis.
4. Progression: The discussion moved sequentially from scenario generation (identifying emerging use cases) to risk identification, and finally to interventions and research directions.
5. Output: Facilitators synthesized outputs daily to scaffold subsequent discussions, resulting in four primary thematic clusters of challenges and future directions.

Key Contributions and Results

The paper organizes the identified challenges and research directions into four central themes:

1. Accountability, Governance, and Liability

• The Challenge: Responsibility is diffused across supply chains (providers, deployers, tool creators), making it difficult to map existing regulations (like the EU AI Act or GDPR) to end-to-end agentic workflows.
• Key Insights:
  • Human Oversight: Mere "human-in-the-loop" placement is insufficient. Effective oversight requires causal power, epistemic access, and authority to intervene, which current logging and XAI (Explainable AI) tools often fail to provide.
  • Traceability Paradox: Supporting accountability requires detailed logging (provenance), which conflicts with privacy principles (data minimization). Research must focus on privacy-preserving provenance mechanisms (e.g., layered audit trails, selective disclosure).
  • Design for Accountability: Accountability should be an architectural property (e.g., self-accountable agents that flag unfair actions) rather than a post-hoc patch. However, these mechanisms themselves become attack surfaces.

2. Consent, Data, and Context

• The Challenge: Traditional consent models (discrete, one-time) cannot handle multi-step, multi-agent workflows where data exposure occurs through inference or cascading sub-actions.
• Key Insights:
  • Reconceptualizing Consent: Consent must evolve from a static permission to a dynamic security primitive, potentially requiring authentication for high-stakes decisions or hybrid architectures with hard-coded gates.
  • Emergent Sensitivity: Data sensitivity is no longer intrinsic but context-dependent and inferential (e.g., an agent inferring a user's health status from a declined suggestion).
  • Memory and Context: AI agent memory creates structural tensions with data minimization. Furthermore, the concept of "contextual integrity" is strained as agents actively shape the contexts in which they operate, necessitating new theoretical frameworks.

3. Human Oversight, Personalization, and Vulnerability

• The Challenge: Balancing the autonomy of agents with the need to protect vulnerable populations and prevent manipulation.
• Key Insights:
  • Calibration: Agents must dynamically calibrate between autonomy and deferring to humans, a difficult task in high-stakes domains like healthcare where tacit human knowledge is irreplaceable.
  • Vulnerability: Vulnerability is not just a demographic trait but an emergent property of dependency. As users rely more on agents, the power balance shifts, creating new attack surfaces (e.g., subtle manipulation, emotional gaslighting).
  • Manipulation vs. Influence: Distinguishing between beneficial nudging, legitimate persuasion, and covert manipulation is critical. Agents may inherit manipulative strategies from training data or be weaponized by third parties.
  • Literacy: Effective governance requires tailored AI literacy efforts to prevent uneven distribution of safety benefits.

4. Transparency, Resilience, and Failure Recovery

• The Challenge: Agentic AI systems are non-deterministic and self-modifying, rendering traditional "one-time" certification and static auditing obsolete.
• Key Insights:
  • Legibility: Explainability must extend beyond single decisions to the entire trajectory of actions, including tool usage and alternative paths considered.
  • Continuous Auditing: Systems require continuous monitoring to detect behavioral divergence caused by updates or dynamic environments. This raises the "who watches the watcher" problem, potentially requiring AI-assisted oversight.
  • Cascade Failures: Failures in one agent can propagate through interconnected ecosystems. Containment strategies (e.g., sandboxing, hierarchical supervision) must account for the networked nature of agents.
  • Sociotechnical Resilience: Resilience is not purely technical; it involves human roles in detection and recovery. It also faces material constraints regarding the computational and environmental costs of sustaining resilient infrastructures.

Significance and Claims

The paper claims that the transition to agentic AI represents a paradigm shift that violates the foundational assumptions of current security and privacy research (boundedness, determinism, and clear attribution). Consequently, the authors argue that existing concepts (consent, traceability, vulnerability) require reconceptualization rather than mere refinement.

The significance of this work lies in its identification of actionable research directions for a field that has not yet been systematically examined through a user-focused security lens. By bringing together diverse stakeholders, the paper highlights that:

1. Legal and Technical Gaps: There is a critical misalignment between current regulatory frameworks and the operational reality of agentic AI supply chains.
2. New Research Agenda: Future work must move beyond model-centric safety to address system-level issues, including the design of self-accountable agents, privacy-preserving provenance, and dynamic consent protocols.
3. Proactive Governance: The authors emphasize the need to anticipate risks before they become embedded in technological infrastructures, urging researchers, policymakers, and practitioners to proactively govern these emerging risks.

The paper concludes that whether the existing vocabulary of security and privacy is sufficient for the agentic AI era, or if entirely new categories of risk must be theorized, remains an open question for the research community to address.