Technical Summary: Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

Problem Statement

Large Language Model (LLM) agents have evolved from stateless chatbots into autonomous "data agents" capable of querying databases, retrieving from private corpora, calling external APIs, maintaining persistent memory, and coordinating with other agents. While traditional LLM privacy concerns focused on training data memorization and final output filtering, data agents introduce a complex privacy landscape. These agents operate across heterogeneous data surfaces, execute multi-step workflows where sensitive information flows through intermediate artifacts (queries, tool arguments, reasoning traces), maintain state across sessions, and act with delegated permissions.

The core problem identified is that privacy in this setting cannot be enforced solely at the final output. Sensitive information can leak through:

1. Intermediate channels: Generated queries, retrieved rows, and tool arguments.
2. Persistent memory: Data written in one session surfacing in another.
3. Compositional leakage: Aggregating benign outputs across steps or agents to infer sensitive information.
4. Contextual violations: Sharing data that is technically permitted but inappropriate for the specific context or recipient.

Existing literature on these risks is scattered across distinct research communities (e.g., retrieval-augmented generation, text-to-SQL, agent memory, access control) and is typically organized by attack type rather than the data surfaces involved. This fragmentation prevents a unified understanding of end-to-end privacy risks in data agents.

Methodology

The authors adopt a data-centric view, organizing the survey around the data an agent touches rather than by attack vectors. The methodology involves:

1. Taxonomization: Systematically categorizing the "data surfaces" (storage and transport mechanisms) an agent interacts with.
2. Risk Mapping: Identifying specific privacy risks (disclosure outcomes and attack vectors) associated with each surface.
3. Governance Analysis: Surveying existing mechanisms (access control, information-flow control, etc.) and mapping their applicability to specific surfaces and risks.
4. Benchmark Evaluation: Reviewing current evaluation benchmarks to identify gaps in measuring end-to-end privacy across multiple surfaces under a unified policy.

The survey covers literature primarily from 2023 to 2026 across security, data management, and machine learning venues, including preprints.

Key Contributions

1. Taxonomy of Data Surfaces

The paper defines six distinct data surfaces where agents operate, each with unique access models and leakage paths:

• Databases & Warehouses: Rows, columns, and schema. Risks include query leakage and schema inference.
• Tabular & File Data: Records and document contents. Risks involve leakage through generated code and intermediate variables.
• RAG Corpora & Vector Stores: Passages, embeddings, and retrieval scores. Risks include membership inference and embedding inversion.
• Tools & External APIs: Call arguments, responses, and side effects. Risks involve prompt injection and untrusted input.
• Agent Memory: Stored facts and user history. Risks include cross-session leakage and memory poisoning.
• Multi-Agent Communication: Inter-agent messages and shared context. Risks involve data pooling across agents.

2. Taxonomy of Privacy Risks

The authors distinguish between Disclosure Outcomes (what is leaked) and Attack Vectors (how it is triggered):

• Outcomes: Direct output leakage, intermediate-step leakage, schema/metadata leakage, inference/aggregation leakage, compositional leakage (aggregation across steps/agents), and policy/consent violations.
• Vectors: Prompt injection, memory poisoning, over-permissioned access, and compromised peers.
• Key Finding: Compositional leakage and cross-session inference are the least-protected risks, as they emerge only when combining multiple steps or agents.

3. Governance Mechanisms and Coverage

The survey catalogs mechanisms to address these risks, including:

• Access Control: Limits initial data reach (rows/columns).
• Information-Flow Control (IFC): Tracks data labels through execution, constraining flow to unauthorized sinks.
• Runtime Governance: Enforces policies on actions and tool calls.
• Privacy-Preserving Transformations: Redaction, abstraction, and synthetic substitution.
• Differential Privacy: Formal guarantees for aggregate releases.
• Contextual Privacy Reasoning: Judging appropriateness based on norms.

Critical Gap Identified: No single mechanism covers all risks. Information-Flow Control is the only mechanism identified that addresses both compositional leakage and cross-session inference, making it the most critical single control for builders to adopt.

4. Benchmark Analysis

The paper maps existing benchmarks (e.g., SecureSQL, AgentDojo, MEXTRA) to specific surfaces and risks.

• Finding: Current benchmarks are siloed; they evaluate single surfaces (e.g., just SQL or just memory) or single-step interactions.
• Gap: There is no benchmark that drives an agent across its full data surfaces (database, RAG, tools, memory, multi-agent) under a single, explicit privacy policy. This prevents the evaluation of compositional and cross-surface risks.

5. Application Domains

The survey analyzes how risks and controls vary across five domains:

• Enterprise Analytics: Dominated by over-return and inference via joins; requires query-level access control.
• Personal Assistants: High risk of injection and inappropriate sharing; requires contextual reasoning and runtime checks.
• Scientific/Institutional: Risks of intermediate-step leakage and reconstruction; requires confidential execution.
• Smart Spaces: Inference from sensor streams and consent issues; requires purpose-based access control.
• Compliance Assistants: The agent's own access is the risk; requires strong auditing and provenance.

Results and Open Problems

The paper does not present new experimental results but synthesizes the state of the field to highlight critical deficiencies:

1. Compositional Leakage is Under-Protected: Risks that emerge from combining multiple steps or agents are the least addressed by current controls.
2. Lack of End-to-End Evaluation: The field lacks a benchmark that evaluates an agent's entire workflow against a unified privacy policy.
3. Forgetting and Revocation: No surveyed mechanism effectively supports the deletion or revocation of data already written to persistent memory or indexes.
4. Utility-Privacy Trade-off: Current mechanisms often fix a privacy budget in advance. The paper suggests a "utility-first" approach (fixing utility requirements and minimizing privacy cost) is better suited for data agents but remains largely unimplemented.
5. Third-Party Governance: Vetting and scoping data access for third-party plugins and skills is an open supply-chain problem.

Significance

The paper claims significance by unifying two previously separate research communities: Agent Security and Data Management. By organizing the literature around the data an agent touches rather than the attack type, the authors provide a reference framework that:

• Clarifies that privacy in data agents is a property of the entire execution, not just the output.
• Identifies Information-Flow Control as the most robust single mechanism for the most complex risks (compositional and cross-session).
• Calls for the development of end-to-end benchmarks that test agents across all data surfaces under explicit policies, arguing this is the "instrument the field most lacks."
• Highlights that data agents currently operate with the obligations of data systems (handling sensitive data) without the corresponding governance infrastructure, framing the supply of this governance as the central open problem for the field.