Technical Summary: MCP-Persona

Problem Statement

The Model Context Protocol (MCP) has become a standard for connecting Large Language Models (LLMs) with external tools and data. However, existing benchmarks predominantly evaluate agents on generic, stateless information-seeking tools. They fail to address the complexities of real-world personalized applications (e.g., social media, enterprise collaboration suites) where tools are tightly coupled with individual user accounts, historical behaviors, and local databases.

Evaluating agents in these domains faces three critical barriers:

1. Deployment Costs: Realistic testing requires access to private user data and substantial human effort for environment setup.
2. Privacy Constraints: Operational security limits the collection and sharing of personal contexts, hindering open-source benchmark construction.
3. Simulation Stability: Maintaining a stable, executable environment that simulates heterogeneous users while ensuring fair evaluation is technically non-trivial.

Consequently, the performance of state-of-the-art (SOTA) agents on personalized, stateful tool use remains largely unexplored and under-evaluated.

Methodology: MCP-Persona

To bridge this gap, the authors introduce MCP-Persona, the first benchmark designed to evaluate agent performance on real-world personalized MCP tools via environment simulation. The system is built upon the interaction of three core components: Tools, Contexts (user profiles), and Tasks.

1. Tool-Traverse (Tool Simulation)

Instead of relying on static documentation, the authors propose a "traverse-then-simulate" paradigm to create high-fidelity local replicas of MCP servers.

• Data Collection: They manually deployed 12 diverse MCP servers (including Lark, Slack, Rednote, Instagram) with sandbox environments and test accounts.
• Authentic Function Call (FC) Traversal:
  • Bootstrapping: Human annotators create valid "seed" function calls to record nominal behaviors.
  • Adversarial Induction: An LLM-driven pipeline generates invalid inputs (e.g., type mismatches, schema violations) to capture error handling logic and decision boundaries.
• Code-Based Simulation: Using the collected successful and failed traces, an LLM autonomously synthesizes executable Python code for each tool. This code acts as a stateful simulator, including a Context Handler module to manage entity persistence (load, save, query, modify) and logic to validate inputs against the dynamic context.

2. Context-Tree (Context Construction)

To support stateful operations, the authors construct a structured, mutable Context-Tree for each server.

• Hierarchy: A tree structure is defined based on tool parameters (e.g., User → Calendar → Event).
• Content Sourcing: Nodes are populated using four methods:
  1. Enumerate: Constrained sampling from fixed sets.
  2. Free-Form: LLM-generated text.
  3. Random: Synthetic generation with format constraints.
  4. Authentic: Sanitized real-world content (e.g., actual Rednote posts) with sensitive data masked.
• Linking: Cross-entity links are established by sampling identifiers from generated content to ensure relational consistency.

3. Persona-Gen (Task Generation)

A two-stage pipeline generates 173 human-verified personalized tasks.

• Prototype Synthesis: Tool invocation chains are sampled based on dependency, diversity, and coherence constraints. An LLM generates abstract instruction skeletons ($S_{proto}$) with placeholders.
• Context Injection & Fuzzification: Realistic values from the Context-Trees replace placeholders. Crucially, instruction fuzzification is applied: execution-specific parameters (implicit context) are removed from the instruction to mimic real-world ambiguity (e.g., a user saying "message my supervisor" without specifying the supervisor's ID, requiring the agent to infer it from the environment).
• Human Verification: Annotators review tasks to ensure alignment between the fuzzy instruction, the decoupled context, and the ground-truth tool chain.

Evaluation Metrics

The benchmark employs both Checkpoint-Based (LLM-judged intermediate steps) and Execution-Based (CRUD operation success rates) evaluations. Metrics include Checkpoint Accuracy (Acc), Success Rate at 0.8 (SR-0.8), and Execution Accuracy (Exec-Acc).

Key Results

Extensive experiments were conducted on over ten SOTA models (including GPT-5, Claude-Sonnet-4.5, and various open-source models) across single-server and cross-server scenarios.

1. Significant Performance Gaps: Even the most advanced models struggle, with no model achieving an accuracy exceeding 50% on the benchmark. Claude-Sonnet-4.5 achieved the highest overall accuracy (41.50%), followed by GPT-5 (41.45%).
2. Task Difficulty:
   • Email tasks were the easiest due to simple operations.
   • Social Media and Collaboration platforms were more challenging due to cross-user interactions and implicit entity resolution.
   • Cross-Server (Hodgepodge) tasks were the most difficult, highlighting failures in coordinating tools across different ecosystems.
3. Failure Modes: Trajectory analysis identified three recurring failure archetypes:
   • Under-exploration: Agents fail to query the environment for missing implicit information (e.g., failing to resolve a user ID from a name).
   • Skipping Dependencies: Agents skip necessary intermediate steps (e.g., converting a phone number to a user ID before scheduling a meeting).
   • Long Context Degradation: Performance drops significantly as context windows fill with voluminous payloads, causing agents to lose track of initial constraints.
4. Simulation Fidelity: The Tool-Traverse method significantly outperformed a documentation-only baseline (Vanilla), achieving 94.0% behavioral alignment compared to 58.0%, proving that traversing real servers yields more accurate error handling and response structures.
5. Skill Enhancement: Providing dedicated skill documents (especially manually refined ones) improved performance, though a significant gap remained.

Significance and Contributions

The paper claims three primary contributions:

1. Tool-Traverse: A rigorous simulation method that faithfully replicates real-world personal MCP tools, overcoming the inability to open-evaluate account-bound servers.
2. MCP-Persona Benchmark: A new evaluation platform comprising 173 human-verified tasks across four representative personalized scenarios (Social Media, Collaboration, Content Management, Email), addressing the critical need for testing agents in communication and collaboration workflows.
3. Empirical Insights: The study reveals that SOTA agents have significant limitations in personalized tool usage, specifically in implicit grounding, multi-step state maintenance, and cross-tool coordination.

The authors position MCP-Persona as a reproducible, privacy-preserving testbed intended to accelerate the development of agents capable of handling the nuanced, context-rich workflows of real users, bridging the gap between general-purpose assistants and deeply personalized, skill-driven counterparts.