💻 computer science

WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks

This paper introduces WeClawArena, an auditable sandbox and benchmark designed to evaluate the utility and security of cross-user agent collaboration in human-centered networks by simulating multi-party interactions over personal workspaces with 620 task variants that include both benign and adversarial scenarios.

Original authors: Prince Zizhuang Wang, Aojie Yuan, Haiyue Zhang, Xiyang Hu, Yue Zhao, Shuli Jiang

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prince Zizhuang Wang, Aojie Yuan, Haiyue Zhang, Xiyang Hu, Yue Zhao, Shuli Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where everyone has a personal digital assistant—an AI that doesn't just answer questions, but actually does things for you. It can manage your calendar, book your flights, write your code, and even negotiate deals on your behalf. This is the promise of "human-centered agent networks," where these digital helpers act as your representatives in a complex digital world. But here's the catch: in this new world, your assistant isn't just talking to your assistant; they are working together across different digital "offices." Your assistant holds your private files, your bank account limits, and your personal rules. Your friend's assistant holds theirs. They have to collaborate to get a job done, but they can't just walk into each other's offices and grab whatever they want. It's like a group of spies trying to solve a puzzle together, where every spy has their own locked briefcase, and they can only share information if they have the right keys and permissions.

The big question is: Can these AI teams work together effectively without accidentally leaking secrets, breaking the rules, or getting tricked by a bad actor? If a hacker whispers a lie to one agent, does the whole team believe it? If an agent is asked to share a secret, will it say "no" even if the task seems urgent? Scientists have been testing how well AI uses tools and talks to each other, but they haven't had a safe, controlled playground to see what happens when these agents try to collaborate across private boundaries while under attack. Without this, we don't know if our future digital helpers are actually safe to use in the real world.

This is where WeClawArena comes in. Think of it as a high-tech, digital "escape room" designed specifically to test these AI teams. The researchers built a sandbox—a safe, simulated environment where they created 124 different scenarios, like negotiating a business deal, booking a group trip, or fixing a software bug. They then expanded these into 620 different versions of each scenario. In the "good" versions, the agents just try to solve the problem. In the "bad" versions, the researchers inject tricky situations: a hacker might send a fake message, poison a piece of data, or try to trick an agent into breaking a privacy rule.

The team ran these simulations with several different AI models to see how they handled the pressure. They found that while some AIs are great at finishing the job, they often fail to notice when they are being tricked or when they are accidentally sharing private information. For example, in their tests, one of the top-performing models (Claude Opus 4.7) was the best at resisting attacks, but even it wasn't perfect. The study showed that an AI can successfully complete a task—like finalizing a trade or booking a flight—while simultaneously leaking a secret budget limit or accepting a fake approval. It's like a waiter who successfully brings your food to the table but also accidentally tells the kitchen your credit card number.

The researchers discovered that the type of mistake an AI makes depends heavily on the situation. In trading and bidding scenarios, the agents were most likely to fall for security tricks (like fake data). In software development and travel planning, they were more likely to mess up privacy or governance rules (like sharing private notes or skipping approval steps). Crucially, the paper proves that you can't just look at whether the AI finished the task to know if it's safe. An AI can "win" the game by finishing the task but still lose the security battle by letting a hacker win.

By recording every single message, tool click, and decision the agents made, WeClawArena allows researchers to audit exactly how and why an attack succeeded. It turns out that the path to a disaster is often paved with small, seemingly harmless steps that the AI takes while trying to be helpful. The study suggests that as we move toward a future where AI agents work together for us, we need to test them not just on how smart they are, but on how well they can say "no" to the wrong requests and protect our private digital spaces. The results show that while we are getting better at building these agents, we still have a long way to go before they are truly safe to let loose in a world where they hold the keys to our digital lives.

Technical Summary: WeClawArena

1. Problem Statement

The paper addresses the emerging paradigm of human-centered agent networks, where persistent personal agents act on behalf of users, maintaining state and communicating with other agents through social and task relations. While existing benchmarks evaluate tool use (e.g., $\tau$ -bench, $\tau^2$ -Bench) or multi-agent coordination (e.g., MultiAgentBench), they fail to capture the specific challenges of multi-party owned-agent collaboration over personal workspaces.

In this setting, agents do not operate in a shared sandbox but must coordinate across distinct, owner-scoped environments containing private files, databases, tools, and policies. This creates a complex problem space where:

Task Utility: Agents must complete collaborative tasks (e.g., bargaining, trading) using only the information and tools visible within their own owner's workspace, requiring them to combine partial information without bypassing boundaries.
Final Harm: Adversarial pressure can travel through the same collaboration channels used for legitimate work, leading to harms in four specific surfaces: Collaboration (goal hijacking, false handoffs), Security (poisoned evidence, unsafe tool use), Privacy (unauthorized disclosure violating contextual integrity), and Governance (invalid authority paths, missing consent).
Auditability: It is difficult to distinguish between a task failure due to agent incompetence and a task failure caused by an attack, or conversely, a task that appears successful but results in privacy leakage or governance violations.

Current benchmarks lack an end-to-end runtime sandbox that simultaneously tests task completion and verifiable attack success across these personal boundaries.

2. Methodology: WeClawArena

The authors introduce WeClawArena, an auditable benchmark and runtime sandbox designed to simulate and evaluate cross-user agent collaboration.

Benchmark Construction

Scope: The benchmark contains 124 base tasks expanded into 620 scenario variants across six domains: Bargaining, Bidding, Travel, SWE-Workspace, Clinical, and Trading.
Scenario Design: Each base task is paired with one benign control (no attacker) and four attack-vector variants targeting the four harm surfaces (Collaboration, Security, Privacy, Governance).
Task Definition: A task is defined as a multi-party owned-agent tool-use collaboration problem. Agents must read, update, and validate records within their own owner-scoped workspaces (files, SQLite databases, policies) to satisfy a task contract.
Attack Construction: Attacks are delivered via injected messages, files, database rows, or artifacts. Crucially, attacks do not directly edit final score fields; they must succeed only if agents encounter, route, accept, or act upon the malicious material through normal operations.

Runtime Sandbox

Architecture: The system runs in a Docker-backed environment where each user has a private workspace. A Message Gateway routes all peer communication, tool calls, and resource operations.
Evidence Capture: The sandbox passively records all runtime events (messages, tool calls, resource operations, policy decisions) into bounded evidence packets.
Evaluation Protocol:
- Utility (TSR): Measured by whether the collaborative tool-use task was completed based on final workspace states and evidence traces.
- Attack Success Rate (ASR): Judged by an LLM (GPT-5.2) post-simulation. The judge analyzes the bounded evidence packet to determine if final harm occurred on the intended surface and if a clear evidence link exists between the harm and the attack vector.
- Separation: Utility and ASR are evaluated separately to avoid conflating task failure with security failure.

3. Key Contributions

Formalization: The paper formalizes the setting of multi-party cross-user tool-use collaboration where personal workspaces serve as both operational substrates and constraint surfaces.
Benchmark: The creation of WeClawArena, comprising 124 base tasks and 620 variants across six diverse domains, explicitly modeling owner-scoped resources and attack vectors.
Runtime Sandbox & Protocol: A controlled environment that records audit evidence, enabling the separate evaluation of task utility and attack success. This allows for the diagnosis of task breakdowns, privacy leaks, poisoned evidence, and invalid authority paths.
Empirical Analysis: A comprehensive evaluation of seven state-of-the-art LLMs (including Claude Opus 4.7, Claude Sonnet 4.5, Claude Opus 4.1, Kimi K2, Kimi K2 Thinking, Qwen3 235B, Qwen3 32B, and DeepSeek V3.2) demonstrating that high task utility does not guarantee resistance to attacks, and that different models exhibit varying vulnerabilities across different harm surfaces.

4. Key Results

Task Utility (TSR): Performance varies significantly by domain. Claude Opus 4.7 demonstrated the strongest overall utility profile, particularly in Travel, SWE-Workspace, and Bidding. Claude Sonnet 4.5 led in Bargaining. SWE-Workspace remained the most difficult domain, with the best model achieving only a 34.0% success rate across all variants.
Attack Resistance (ASR):
- Model Variation: Claude Opus 4.7 showed the highest resistance to attacks across all harm surfaces. Open-weight models generally clustered lower in resistance.
- Surface Variation: Governance and Security attacks had the highest success rates across models, while Collaboration attacks were the least successful.
- Domain Variation: Security pressure was dominant in Bargaining and Bidding, whereas Privacy and Governance pressure were most prevalent in SWE-Workspace and Travel.
Utility-Risk Tradeoff: The study found that TSR drop and final harm are correlated but distinct. Some models maintained high utility while suffering high attack success (e.g., completing a task while leaking private data), while others failed the task entirely due to attack-induced derailment.
Failure Modes: Analysis revealed that 117 rows resulted in both task success and attack success, indicating that agents can complete visible joint tasks while producing final cross-user harm (e.g., accepting poisoned evidence or bypassing approval gates).

5. Significance and Claims

The paper claims that WeClawArena establishes a concrete benchmark setting for human-centered agent networks, shifting the focus from single-agent tool use to delegated agents operating over owner-scoped files, records, and decision rights.

Diagnostic Capability: The sandbox provides a reproducible testbed to diagnose where collaborative personal agents succeed, where they break down, and how harmful influence travels through legitimate work channels.
Necessity of Separation: The results emphasize the necessity of evaluating utility and attack success separately. Relying solely on task success rates (TSR) can mask critical security and privacy failures, while focusing only on attack success may overlook the utility costs of defensive behaviors.
Auditability: By recording bounded runtime evidence, the framework supports the audit of "who acted, which messages mattered, and why the final outcome occurred," addressing the need for accountability in multi-agent systems.

The authors position WeClawArena not as a definitive solution to all agent security issues, but as a necessary step toward understanding the specific risks of cross-user collaboration in persistent personal-agent networks.