Technical Summary: Towards Trustworthy Agentic AI

Problem Statement

The paradigm shift from static Large Language Models (LLMs) to agentic AI—systems capable of autonomous planning, tool invocation, and multi-step reasoning—has introduced qualitatively new failure modes that challenge trustworthiness. Unlike single-turn chatbots, agentic systems produce multi-step trajectories where intermediate states (plans, tool calls, memory updates) can directly affect real-world outcomes. A minor error early in a trajectory can cascade into high-impact actions, and the interaction with external tools expands the attack surface beyond conventional LLM safety evaluations.

Current trustworthiness assessments often focus on final outputs or single-turn interactions, failing to address the systemic risks of long-horizon autonomy, tool-mediated exfiltration, and indirect prompt injection. Furthermore, existing surveys often treat trustworthiness dimensions (safety, security, privacy, fairness) in isolation or focus on general AI principles rather than the specific workflow of agentic systems. There is a critical need for a unified framework that maps risks and mitigations across the agent lifecycle and provides comparable evaluation metrics for high-stakes deployments.

Methodology and Framework

This survey adopts a workflow-centric lens to structure the analysis of trustworthy agentic AI. The authors define agentic AI through a five-stage lifecycle: Perceive → Plan → Act → Reflect → Learn. This framework serves as a consistent interface for mapping threats, defenses, and evaluation signals across diverse agent designs.

The paper focuses on two core trustworthiness dimensions critical for high-risk deployments:

1. Safety and Robustness: Addressing unintended harm, specification errors, and reliability under distribution shifts or adversarial perturbations.
2. Privacy and System Security: Addressing data leakage, credential theft, supply chain vulnerabilities, and the integrity of the execution environment.

Other aspects (value alignment, transparency, fairness, accountability) are treated as supporting context rather than parallel chapters.

The methodology involves:

• Risk-to-Method Mapping: For each core dimension, the authors identify where risks emerge along the agent workflow and summarize stage-targeted mitigation strategies (e.g., constrained optimization, runtime shielding, sandboxing).
• Assurance Stack Analysis: The paper proposes a layered defense-in-depth strategy spanning pre-deployment (red teaming, formal requirements), training-time (safe RL, preference optimization), runtime (verification, least-privilege access), and post-hoc (telemetry, audit logs) phases.
• Unified Evaluation Hub: The authors consolidate fragmented metrics and benchmarks into a single reference, distinguishing between outcome metrics (e.g., success rate, catastrophic event rate) and process metrics (e.g., constraint violation rates, trace completeness).

Key Contributions

1. A Focused Examination of Core Dimensions

The paper concentrates specifically on Safety and Robustness and Privacy and System Security, providing a detailed taxonomy of risks and mitigations aligned with the agent lifecycle.

• Safety & Robustness: Identifies risks such as data poisoning, OOD generalization, brittle heuristics, and cascading execution failures. Mitigations include adversarial training, constrained Markov Decision Processes (CMDPs), runtime shielding, and hierarchical decomposition.
• Privacy & Security: Highlights risks like indirect prompt injection, memory poisoning, credential leakage via tool chains, and supply chain compromises. Mitigations include zero-trust intake, differential privacy, credential vaulting, and software bills of materials (SBOMs).

2. Risk-to-Method Mapping Along the Agent Workflow

The survey provides a consistent Definition → Risks → Methods structure for each dimension. It illustrates how failures can occur at specific stages:

• Perceive: Input sanitization, OOD detection, and adversarial training.
• Plan: Safe RL, constitutional constraints, and conservative planning.
• Act: Runtime shields, sandboxing, and least-privilege tool access.
• Reflect: Simulation-based testing, trace auditing, and formal verification.
• Learn: Regression gating, canary rollouts, and incident-driven dataset updates.

3. A Consolidated Evaluation Hub

The authors unify cross-dimension metrics and benchmark families into a single reference (Section 4). Key contributions include:

• Metric Dictionary: Standardized definitions for metrics such as CER (Catastrophic Event Rate), CVR (Constraint Violation Rate), DCR (Decision Coverage Rate), and ASR (Attack Success Rate).
• Scenario-to-Metric Mapping: Guidance for release gating in high-risk scenarios, distinguishing between mandatory thresholds (e.g., CER caps) and diagnostic metrics.
• Evaluation Pipeline: A staged pipeline from offline regression and simulation stress testing to shadow mode and canary rollouts, emphasizing the need for both outcome and process signals.

4. Case Study on Real-World Failures

The paper presents a case study of security failures in open-source agentic systems (OpenClaw and Moltbook). It analyzes the "lethal trifecta" (access to private data, processing untrusted content, and external communication) and maps specific vulnerabilities (insecure defaults, prompt injection, credential leakage) to the survey's lifecycle framework, demonstrating the necessity of defense-in-depth.

Results and Findings

• System-Level Nature of Trust: Trustworthiness cannot be achieved by model alignment alone; it requires system-level controls across the entire workflow, including tool interfaces, memory management, and runtime policies.
• Trade-offs: The survey highlights inherent tensions, such as the trade-off between utility and safety (stricter runtime policies may block benign edge cases) and privacy and personalization (memory retention improves performance but increases leakage risk).
• Evaluation Gaps: Current benchmarks often fail to capture long-horizon compounding errors and interactive adversarial dynamics. The paper argues for "process-aware" evaluation that tracks trace completeness and constraint compliance, not just final task success.
• Supply Chain Vulnerability: The OpenClaw/Moltbook case study reveals that agent ecosystems are highly vulnerable to supply chain attacks (malicious plugins, unverified updates), necessitating strict provenance checks and SBOMs.

Significance and Claims

The authors position this survey as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments (e.g., autonomous driving, healthcare, financial trading).

• Operationalizing Trust: The paper claims to move beyond theoretical principles by offering a concrete "evaluation hub" with standardized metrics and release-gating strategies, enabling consistent comparison across different agent architectures.
• Holistic View: By integrating a multi-dimensional trust taxonomy with a workflow lens, the survey addresses the gap between static LLM safety and the dynamic, tool-mediated risks of agentic AI.
• Open Challenges: The paper concludes by outlining critical open problems, including self-evolving agents under concept drift, runtime monitoring for rare events, privacy-preserving personalization, and the trust–utility trade-off. It emphasizes that trustworthy deployment requires continuous monitoring and iterative improvement rather than one-time certification.

The survey does not propose new algorithms or experimental results but synthesizes the current state of knowledge to provide a structured framework for assessing and mitigating risks in the emerging field of agentic AI.