MAPS: A Multilingual Benchmark for Agent Performance and Security

Technical Summary: MAPS: A Multilingual Benchmark for Agent Performance and Security

1. Problem Statement

As AI transitions from passive Large Language Models (LLMs) to autonomous Agentic AI systems—which use reasoning, memory, and external tools to solve complex tasks—a critical vulnerability has emerged. While LLMs are known to exhibit performance degradation in non-English languages, the impact of this "multilingual gap" on agentic reasoning, planning, and tool use remains largely unexplored.

Existing agentic benchmarks (e.g., GAIA, SWE-Bench) are almost exclusively English-centric. This creates a risk where non-English users may encounter unreliable agents, incorrect tool executions, or heightened security vulnerabilities (e.g., bypassing safety filters through translated adversarial prompts), leading to real-world harm in sensitive domains like banking or software engineering.

2. Methodology

The authors propose MAPS, a comprehensive multilingual benchmark suite. The methodology is structured into three core phases:

• Benchmark Construction: MAPS extends four established, high-authority agentic benchmarks into 11 diverse languages (plus English), totaling 805 tasks and 9,660 instances:
  1. GAIA: Real-world assistant tasks (web navigation, retrieval).
  2. SWE-Bench: Software engineering (bug fixing/patch validation).
  3. MATH: Mathematical reasoning (symbolic/stepwise planning).
  4. Agent Security Benchmark (ASB): Adversarial robustness (policy adherence).
• Hybrid Translation Pipeline: To ensure high semantic and structural fidelity, the authors used a multi-stage pipeline:
  • NMT + LLM: Neural Machine Translation (Google Translate) provides structural alignment, which is then refined by an LLM (Cohere Command A) for contextual fluency.
  • Verification: An automated adequacy/integrity check is followed by human expert verification by native speakers. The authors specifically focused on "answerability"—ensuring the translation preserves enough intent for the task to remain solvable.
• Experimental Design: To isolate the "Multilingual Effect," the authors kept the agent architectures, tools, and environments constant, varying only the input language. They evaluated leading open-source agents (e.g., OpenDeepResearch, SWE-agent) across different LLM backbones (GPT-o1, GPT-4, etc.).

3. Key Contributions

• The MAPS Suite: The first standardized, multi-domain, and security-aware multilingual evaluation framework for agentic AI.
• Quantifiable Multilingual Analysis: The first evidence-based study showing how linguistic load affects agentic decision-making and security.
• Open-Source Resources: Release of the full dataset and a verified subset on Hugging Face to facilitate equitable AI development.

4. Key Results

The study reveals a significant "Multilingual Effect" that varies by task type:

• Performance Degradation in Language-Heavy Tasks: Agents showed notable declines in GAIA (up to 16% drop) and ASB. This is because these tasks rely heavily on natural language for planning and instruction following.
• Resilience in Language-Light Tasks: SWE-Bench and MATH showed minimal degradation. This is attributed to their reliance on structured syntax (code and mathematical notation) rather than flexible natural language.
• Security Vulnerabilities: Multilingual inputs amplify security risks. The authors found that direct translations of English "jailbreak" prompts could bypass safety mechanisms in non-English contexts, increasing the Attack Success Rate (ASR) in ASB by up to 27%.
• Root Cause (Planning Drift): Using multilingual embeddings, the authors demonstrated that degradation often stems from planning inconsistency. In non-English settings, an agent's initial decomposition of a task often drifts away from the original intent compared to its English counterpart.
• Translation Limits: While deterministic translation (pre-translating inputs to English) recovers some performance, it does not close the gap, suggesting the issue lies in the model's underlying cross-lingual reasoning capabilities rather than just translation noise.

5. Significance

MAPS shifts the focus of multilingual AI research from simple text generation to complex, tool-augmented decision-making. The findings serve as a warning to developers: an agent that is safe and capable in English is not guaranteed to be safe or capable in other languages. The paper provides actionable guidelines, suggesting that multilingual red-teaming and linguistic composition analysis should be standard components of the agent development lifecycle to ensure global reliability and equitable access.