Technical Summary: An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures

Problem Statement

Multilingual multi-agent systems, while increasingly deployed as general-purpose tools, exhibit substantial performance degradation when processing user requests in languages other than English. Prior research has documented this cross-lingual decline in tasks ranging from function calling to complex web interactions. However, existing studies fail to systematically characterize how task-critical information is lost or distorted within the multi-agent system, specifically at the "request-to-plan" boundary.

The core problem addressed is the planning-grounding failure: a mismatch between a user's multilingual request and the generated executable plan such that, even if the plan is executed perfectly, the result fails to satisfy the original intent. The authors hypothesize that as language-resource availability declines, these specific structural failures become the dominant cause of execution errors, yet they remain uncharacterized and unmitigated in current state-of-the-art (SOTA) systems.

Methodology

1. Taxonomy Derivation

The authors developed a systematic procedure to identify recurring failure modes:

• Data Selection: They analyzed cases where an agent succeeded on an English query but failed on a corresponding non-English query.
• Qualitative Analysis: Researchers manually analyzed failed plan pairs, comparing the successful English plan, the failed non-English plan, and the original query to diagnose discrepancies.
• Saturation: The process continued until new samples yielded no new failure categories, resulting in a consolidated taxonomy of five planning-grounding failures:
  1. Entity Grounding: Mis-translating, hallucinating, or substituting target entities (e.g., people, places, documents).
  2. Source Grounding: Altering specified source identifiers (e.g., file names, URLs) or omitting required post-retrieval filtering criteria.
  3. Temporal Grounding: Changing or omitting time periods specified in the query.
  4. Operation Grounding: Following unsound reasoning chains or logical sequences that alter the task objective, even if surface constraints are preserved.
  5. Answer-Format Grounding: Producing a substantively correct answer but in the wrong format (e.g., including extra text or metadata when a raw value is required).

2. Mitigation Strategy: TART

To test the actionability of this taxonomy, the authors introduced TART (Taxonomy-guided Actionable Representation).

• Mechanism: TART is a semantic parsing protocol that converts a raw user query (in any language) into a structured JSON representation before the planning phase begins.
• Structure: The JSON fields correspond one-to-one with the five failure categories (entities, time constraints, source constraints, attachment types, operations, and answer types).
• Integration: This structured representation is injected into the system prompts of the planner, coordinator, and downstream worker agents. It acts as a "semantic contract," ensuring that the constraints and commitments of the original request remain explicit throughout the entire execution pipeline.
• Design Philosophy: TART requires no changes to underlying model parameters or the multi-agent framework architecture; it functions as an interface layer.

3. Experimental Setup

The evaluation was conducted across multiple dimensions to ensure robustness:

• Datasets:
  • GAIA-MAPS: A multilingual benchmark of complex, real-world agentic tasks (165 tasks per language). The study extended this to 11 languages, including low-resource languages (Igbo, Yoruba, Bengali, Swahili, Kyrgyz, Nyanja) via machine translation.
  • MULTITAT: A controlled dataset for multilingual table-and-text reasoning.
• Models: Three distinct LLM backbones were tested: GPT-5-mini, Mistral-Large-3, and Qwen3-VL-235B-A22B.
• Agent Configurations:
  • OWL: A SOTA multi-agent system for GAIA tasks, featuring a dedicated planner, coordinator, and specialized workers (web, document, coding).
  • 3-Agent System: A simplified setup for MULTITAT (Planner, Reasoning/Coding Agent, Answer Agent).
• Evaluation Metric: Exact Match (EM) accuracy.

Key Results

Performance Improvements

TART consistently improved end-to-end performance across all languages, models, and datasets:

• GAIA-MAPS (OWL System):
  • Using GPT-5-mini, TART raised the average accuracy across 11 languages from 24.9% to 30.5% (+5.6 percentage points).
  • Improvements were observed in both high-resource (e.g., German +9.1, Hindi +9.7) and low-resource languages (e.g., Yoruba +10.9, Igbo +7.9).
  • Similar gains were observed with Mistral-Large-3 (mean +5.9 points) and Qwen3-VL (mean +3.6 points).
• MULTITAT:
  • For Mistral-Large-3, TART increased mean accuracy from 21.0% to 31.0% (+10.0 points, a 47.6% relative improvement).
  • For Qwen3-VL, mean accuracy increased from 30.7% to 33.8% (+3.1 points).

Failure Analysis and Resource Gradient

Using an LLM-as-a-Judge to categorize failures across all languages, the authors identified a clear language-resource gradient:

• As language-resource availability decreases (moving from English/German to low-resource languages like Igbo and Yoruba), the share of "Other" (uncategorized) failures decreases.
• Conversely, the share of taxonomy-covered failures (specifically Operation and Entity grounding) increases sharply.
• In the lowest-resource languages, operation grounding becomes the dominant failure mode. This confirms that multilingual degradation is not random but follows a systematic pattern that TART is specifically designed to counteract.

Ablation Studies

• Component Contribution: The "Operations" field in TART was found to be most critical for low-resource languages, recovering a significant portion of the performance gap (e.g., +7.3 points for Igbo).
• Task Complexity: TART showed consistent improvements on Level 1 and Level 2 tasks. Performance on Level 3 (most complex) tasks remained largely unchanged, suggesting that while TART fixes planning-grounding errors, the intrinsic complexity of long-horizon execution chains (more steps, more tool calls) introduces new failure points beyond the scope of the initial planning constraints.

Significance and Claims

The paper claims to be the first work to:

1. Systematically characterize planning failures across languages in multi-agent systems.
2. Derive and evaluate a mitigation strategy (TART) directly from such a taxonomy.

The authors argue that their findings demonstrate that multilingual failures at the request-to-plan boundary are systematic enough to characterize and structured enough to mitigate. By exposing the five semantic commitments (entities, sources, temporal, operations, format) as explicit constraints, TART allows multi-agent architectures to preserve user intent through "inspectable semantic contracts" rather than relying solely on unconstrained natural language planning.

The study concludes that while TART significantly improves performance, particularly in low-resource settings, it does not eliminate all failure modes, especially those arising from the intrinsic complexity of long-horizon tasks (Level 3). The work advocates for a shift in multi-agent design toward explicit, structured representations of user intent to bridge the gap between multilingual input and executable action.