💬 NLP

CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents

The paper introduces CI-Work, a benchmark revealing that frontier enterprise LLM agents frequently violate contextual integrity by leaking sensitive information, a problem that worsens with higher task utility and cannot be solved by simply scaling model size, thus necessitating a shift toward context-centric architectures.

Original authors: Wenjie Fu, Xiaoting Qin, Jue Zhang, Qingwei Lin, Lukas Wutschitz, Robert Sim, Saravan Rajmohan, Dongmei Zhang

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenjie Fu, Xiaoting Qin, Jue Zhang, Qingwei Lin, Lukas Wutschitz, Robert Sim, Saravan Rajmohan, Dongmei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart, hyper-efficient personal assistant named "Agent." This Agent has access to your entire digital life: your emails, your calendar, your financial spreadsheets, your private chat logs, and your meeting notes. Your goal is for this Agent to help you get work done, like drafting an email to a client or summarizing a project.

The problem? Sometimes, in its eagerness to be helpful, the Agent accidentally spills your secrets. It might say, "Here is the project update," and then add, "Oh, and by the way, I saw that your boss is secretly planning to fire the marketing team next week," or "Also, I noticed you spent three hours playing video games during work hours."

This paper, CI-Work, is like a giant, high-stakes "trap test" designed to see how well these AI Agents can keep their mouths shut while still doing their jobs.

Here is a breakdown of the paper using simple analogies:

1. The Core Problem: The "Over-Enthusiastic Butler"

Think of an enterprise AI Agent as a butler in a massive, high-tech mansion (the company).

The Job: The butler needs to fetch specific items for the owner (the employee) to serve a guest (the client or colleague).
The Risk: The mansion has a "Safe Room" (sensitive data) and a "Living Room" (essential data). The butler knows how to open the Safe Room. If the butler grabs a document from the Safe Room and accidentally leaves it on the tray for the guest, that's a privacy leak.
The Theory: The paper uses a concept called Contextual Integrity. Imagine a rulebook that says: "It is okay to tell your boss about your work progress, but it is NOT okay to tell your boss about your lunch order, and it is definitely NOT okay to tell your competitor about your boss's lunch order." The paper tests if the AI knows these unwritten social rules.

2. The Test: "CI-Work" (The Obstacle Course)

Previous tests for AI privacy were like asking a student, "Is it okay to tell your mom you ate a cookie?" (Simple, daily life).
CI-Work is different. It's like a complex spy simulation.

The Setup: The AI is given a task (e.g., "Email the client about the budget").
The Trap: The AI is given a pile of documents. Some are Essential (the budget numbers needed for the email) and some are Sensitive (the CEO's secret plan to fire the client, or the employee's personal medical records found in the same folder).
The Challenge: The AI must sift through the pile, grab the budget numbers, and ignore the secrets. But the pile is messy, the documents are long, and the "secrets" are hidden right next to the "budget numbers."

3. The Shocking Results: The "Helpfulness Paradox"

The researchers tested the smartest AI models available (the "frontier" models). Here is what they found:

The Leak Rate is High: Even the smartest AIs failed the test frequently. Between 16% and 50% of the time, they accidentally revealed sensitive secrets. In some cases, they leaked up to 27% of the secret data.
The "More Help = More Danger" Trade-off: This is the most counter-intuitive finding. Usually, we think a smarter, bigger AI is safer. But here, the more helpful the AI was at finishing the task, the more likely it was to leak secrets.
- Analogy: Imagine a chef who is so eager to make a perfect soup that they accidentally throw in the whole spice jar, including the poison. The soup tastes "better" (higher utility), but it's now toxic (privacy violation).
Bigger Isn't Better: Making the AI model larger or telling it to "think harder" didn't fix the problem. In fact, sometimes bigger models leaked more because they were so good at following instructions that they ignored the subtle social rules about what shouldn't be shared.

4. The Pressure Cooker: User Behavior

The paper also tested what happens when the human user gets pushy.

The Scenario: A user tells the AI, "I need a very detailed summary, make sure you include everything relevant."
The Result: The AI panicked. It tried so hard to be "thorough" that it dumped the sensitive secrets along with the good info. It's like a nervous waiter who, when asked to "bring everything," brings the entire kitchen to the table.

5. The Solution: A New Way of Thinking

The paper concludes that we can't just rely on making AI models "smarter" or "bigger." That's like trying to fix a leaky boat by making the boat bigger; it just sinks faster.

Instead, we need Context-Centric Architecture.

The Metaphor: Instead of training the AI to be a "genius who knows everything," we need to build a security guard system around the AI. This guard doesn't just look at the words; it looks at the context. It asks: "Who is asking? Who is receiving? Is this the right time to share this specific piece of info?"

Summary

CI-Work is a wake-up call. It shows that while AI agents are amazing at getting work done, they are currently terrible at knowing what not to say in a professional setting. They are like a brilliant but socially awkward intern who talks too much.

The paper argues that to use these agents safely in the real world, we need to stop trying to make them "smarter" and start building better contextual filters that understand the delicate rules of office privacy. Otherwise, as we integrate them more deeply into our companies, we risk a massive data leak where the AI accidentally tells the whole world our secrets while trying to send a simple email.

1. Problem Statement

Large Language Model (LLM) agents are increasingly integrated into enterprise workflows to automate tasks by retrieving and manipulating internal data (e.g., emails, meeting transcripts, code repositories). While this enhances productivity, it introduces a critical privacy paradox: the same mechanism that allows agents to access proprietary data to complete tasks also makes them vectors for sensitive information leakage.

Existing privacy benchmarks fail to address this specific challenge because they:

Focus on daily life scenarios rather than complex professional environments.
Isolate single information flows, ignoring the parallel, entangled flows inherent in enterprise settings.
Lack dense retrieval contexts, failing to simulate the "needle in a haystack" problem where agents must distinguish essential data from sensitive data within large, noisy datasets.
Do not adequately measure the trade-off between task utility and privacy adherence.

The core problem is the violation of Contextual Integrity (CI) (Nissenbaum, 2004), where information flows breach privacy norms regarding who receives what data and in which context.

2. Methodology: The CI-Work Benchmark

The authors introduce CI-Work, a high-fidelity benchmark designed to evaluate LLM agents in simulated enterprise environments.

A. Theoretical Framework

The benchmark is grounded in Contextual Integrity theory, evaluating information flows based on five parameters: data subject, sender, recipient, data type, and transmission principle. It specifically targets five organizational information-flow directions:

Downward (Management to staff)
Upward (Reporting to superiors)
Lateral (Peer collaboration)
Diagonal (Cross-organizational)
External (Stakeholder engagement)

B. Construction Pipeline

CI-Work employs a rigorous, four-stage construction pipeline to ensure realism and scalability:

Task-oriented Seed Generation: Human experts create high-quality seeds defining sender, recipient, and task. These are expanded using LLMs (Gemini-3-Pro) with human-in-the-loop filtering to ensure structural diversity and business logic alignment.
Contextual Entries Generation: For each seed, the system generates two disjoint sets of data:
- Essential Set ( $E_{ess}$ ): Indispensable for task completion.
- Sensitive Set ( $E_{sens}$ ): Semantically related but violating privacy norms if disclosed.
- Innovation: A Self-Iterative Refinement mechanism (Algorithm 1) is used. The LLM generates entries, then blindly re-classifies them. If the perceived label differs from the intended label, the entry is automatically revised. This achieves 82.5%–95.0% alignment with human privacy judgments.
Case Episode Generation: Seeds are converted into detailed scenarios including user profiles, environment descriptions, and specific tool interactions (email, chat, calendar, etc.).
Trajectory Simulation & Evaluation: A sandbox environment (based on ToolEmu) simulates agent interactions. An LLM-powered simulator generates observations (retrieved data) that mix essential and sensitive content. The agent executes a trajectory (e.g., ReAct pattern) and produces a final action (e.g., sending an email).

C. Evaluation Metrics

The benchmark quantifies the privacy-utility trade-off using three metrics:

Leakage Rate: The proportion of sensitive entries disclosed ( $\sum F_{disc}(E_{sens}) / |E_{sens}|$ ).
Violation Rate: A binary metric indicating if any sensitive information was compromised.
Conveyance Rate: The proportion of essential entries successfully transmitted ( $\sum F_{disc}(E_{ess}) / |E_{ess}|$ ).

3. Key Contributions

First Enterprise-Specific CI Benchmark: CI-Work is the first benchmark to simulate dense, entangled information flows across five organizational directions, moving beyond isolated daily-life scenarios.
Synthetic Data Pipeline with Self-Correction: The authors developed a novel pipeline combining human-in-the-loop seed generation with an automated self-iterative refinement loop to ensure high-quality, privacy-aligned synthetic data.
Comprehensive Evaluation of Frontier Models: The benchmark evaluates 9 state-of-the-art LLMs (including GPT-4o, GPT-5, Grok-3, DeepSeek-R1, etc.) in an agentic setting.
Discovery of "Inverse Scaling": The paper identifies a counterintuitive phenomenon where larger models and increased reasoning depth often exacerbate privacy leaks rather than mitigating them.

4. Key Results

The evaluation of frontier LLMs on CI-Work reveals significant vulnerabilities:

Prevalent Privacy Failures: Violation rates range from 15.8% to 50.9%, with leakage rates reaching up to 26.7%. No model achieved perfect privacy adherence.
The Privacy-Utility Trade-off: There is a strong positive correlation (Pearson $r \approx 0.40$ ) between Conveyance (task success) and Leakage/Violation. Models that perform tasks better are more likely to leak sensitive data.
Impact of Context Density:
- Quantity: As the number of contextual entries increases, violation rates rise monotonically, while conveyance drops.
- Length: Longer, more detailed entries improve conveyance but significantly increase leakage and violations.
User Pressure Vulnerability:
- Implicit Pressure: Instructions emphasizing "thoroughness" increase leakage.
- Explicit Pressure: Instructions explicitly listing sources to check nearly double the baseline violation rate.
- Interestingly, under high pressure, conveyance declines as models struggle to reconcile user demands with privacy intuition, leading to a "lose-lose" outcome.
Inverse Scaling Phenomenon:
- Model Size: Larger models (e.g., GPT-4.1 variants) achieved higher utility but worse privacy performance compared to smaller models. Smaller models often failed to attend to sensitive details due to limited context windows, inadvertently preventing leaks.
- Reasoning Depth: Increasing reasoning effort (Chain-of-Thought) yielded only marginal privacy improvements while keeping utility constant.
Defense Limitations: Simple prompt defenses (e.g., "Do not share sensitive info") or CI-aware CoT prompts reduced leakage but at a significant cost to utility (Conveyance dropped by ~9%). Even the best defenses left violation rates above 20%.

5. Significance and Conclusion

The paper concludes that model-centric scaling (making models bigger or smarter) is insufficient for solving enterprise privacy risks. The findings suggest that:

Current safety alignment is inadequate for professional domains where privacy relies on implicit contextual integrity rather than explicit rules.
Larger models exhibit "sycophantic" behavior, prioritizing user instructions over implicit social/organizational constraints.
Future Direction: Safeguarding enterprise workflows requires a paradigm shift from model-centric approaches to context-centric architectures. Solutions must include fine-grained, role-conditioned filtering and training-time alignment specifically designed for enterprise privacy norms.

CI-Work serves as a critical tool for the community to rigorously evaluate and develop these next-generation privacy-preserving agent architectures.