💬 NLP

SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

This paper introduces SEATauBench, the first evaluation framework for Southeast Asian sovereign AI that adapts TauBench to five regional languages, revealing that while English agent capabilities transfer well to localized conversations, their performance and robustness degrade significantly as task contexts and tool specifications are fully localized.

Original authors: My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, highly trained robot assistant. You've tested it extensively in English, and it's great at booking flights, buying clothes, and managing phone plans. It seems like a superhero.

But then, you try to send this same robot to work in Southeast Asia, where people speak Thai, Vietnamese, Indonesian, Filipino, and Mandarin. You ask yourself: "If it's so good in English, will it just work in these other languages too?"

This paper, SEATauBench, is like a giant, realistic stress test designed to find the answer. The researchers built a special "training ground" to see how well these AI agents actually handle real-world tasks when everything around them is translated into local languages.

Here is the story of what they found, broken down simply:

1. The Setup: The "Three-Layer Cake" of Difficulty

The researchers didn't just translate the whole thing at once. They built a test with four levels, like climbing a ladder, to see exactly where the robot starts to stumble.

Level 1 (The English Baseline): The robot speaks English, the tools are in English, and the rules are in English. This is the "easy mode" to see how good the robot is at its core.
Level 2 (The Conversation Switch): The robot and the human user start talking in a local language (like Thai), but the robot's internal tools and rulebooks are still in English. It's like talking to a friend in Spanish, but the instructions on the map are still in English.
Level 3 (The Tool Switch): Now, the conversation is in English, but the robot's tools (like the "search flight" button or the "check balance" menu) are described in the local language. It's like having a remote control where all the buttons are labeled in Thai.
Level 4 (The Full Immersion): Everything is in the local language. The chat, the tools, the rules, and the database are all translated. This is the "hard mode" where the robot has to think, read, and act entirely in a new language.

2. The Big Surprise: The "Glass Ceiling"

The paper found something very interesting.

At Level 2 (Just talking): The robot did pretty well! If you just asked it to chat in Vietnamese or Thai, it could handle the conversation almost as well as it did in English. It was like a tourist who learned a few phrases and could order food.
At Level 3 & 4 (Tools and Rules): This is where the robot hit a glass ceiling. As soon as the tools and rules were translated, the robot's performance dropped sharply.
- Imagine a chef who can speak perfect Italian but can't read an Italian recipe book or use Italian measuring cups. They might chat nicely, but they can't actually cook the meal.
- The paper found that when the "recipe book" (the tool specifications) and the "kitchen rules" (domain policies) were in a local language, the robots made many more mistakes. They got confused, failed to book the flight, or gave the wrong phone plan.

3. The "Language Drift" Mystery

The researchers also checked if the robots were "slipping" and accidentally speaking English when they were supposed to be speaking Thai or Filipino.

The Finding: The robots did slip up and speak English sometimes, especially in the hardest scenarios.
The Twist: Surprisingly, this "slipping" wasn't the main reason they failed. Even when the robot spoke the correct local language perfectly, it still failed the task if the tools were confusing.
The Metaphor: It's like a driver who is driving perfectly on the right side of the road (speaking the right language) but still crashes because they can't read the local traffic signs (the tools). The problem isn't the language they speak; it's their ability to understand the local environment.

4. The "One Size Does Not Fit All" Problem

The paper also looked at different types of robots (different AI models).

Some robots were better at certain languages than others. For example, a robot that was great at English didn't necessarily become great at Thai just because it was smart.
They found that Filipino was a surprisingly good "test case." If a robot performed well in Filipino, it was likely to perform well in the other Southeast Asian languages too. It's like finding a "canary in the coal mine" for the whole region.

5. The Main Takeaway

The paper concludes that we cannot assume an AI that is smart in English is automatically ready for Southeast Asia.

The Old Way: We used to think, "If it works in English, it works everywhere."
The New Reality: The paper shows that "English-only" tests are like testing a car only on a smooth highway. When you put that car on a bumpy, unpaved road with local signs (the localized tools and rules), it breaks down.

In short: SEATauBench is a diagnostic tool that proves we need to build and test AI specifically for local languages and tools, not just translate English tests and hope for the best. The gap between "talking the language" and "working in the language" is huge, and this benchmark helps us measure exactly how big that gap is.

Technical Summary: SEATauBench

Problem Statement

While Southeast Asian (SEA) AI development has expanded rapidly, the capabilities of AI agents in regional languages remain poorly understood. Existing benchmarks primarily assess static model behaviors (e.g., reasoning, translation, or knowledge retrieval) rather than the dynamic, multi-turn, tool-mediated interactions required for real-world sovereign AI deployment in sectors like commerce, travel, and services. Furthermore, current agent evaluation frameworks are heavily centered on English, creating a gap where English-centric assessments may not accurately reflect agent performance in linguistically diverse, low-resource SEA contexts. There is a critical need for a diagnostic framework that evaluates how agents perform when not only the conversation language changes, but also the tool specifications and task domains are localized.

Methodology

The authors introduce SEATauBench, the first agent-focused evaluation framework for SEA sovereign AI. The framework adapts the English-based $\tau^2$ -Bench (Yao et al., 2024; Barres et al., 2025) into five target languages: Mandarin Chinese, Vietnamese, Thai, Indonesian, and Filipino.

Adaptation Pipeline

To ensure task semantics remain consistent while localizing the interface, the authors developed a two-phase, non-breaking translation pipeline:

Offline Translation: Static domain assets (task definitions, policies, database content, tool docstrings, and schemas) are translated. Crucially, the pipeline masks executable tokens (IDs, status codes, tool names, and enum literals) to prevent translation from breaking the execution layer. A schema-first approach establishes a glossary to ensure terminological consistency.
Runtime Localization: During inference, the environment patches the agent's view. Tool schemas (descriptions, enum choices) are rendered in the target language ( $L_2$ ), but arguments supplied by the agent are normalized back to canonical English values before tool execution. Tool outputs are then localized back into $L_2$ for the user interaction. This ensures the agent operates in a monolingual $L_2$ context while maintaining metric comparability with the English baseline.

Evaluation Scenarios

SEATauBench evaluates agents across four progressively localized scenarios to isolate specific capability gaps:

S1 (English Only): Baseline performance in English.
S2 (L2 Interaction): The user-agent conversation is in $L_2$ , but tools, policies, and contexts remain in English.
S3 (L2 Tool): Tool schemas are translated into $L_2$ , while conversation and domain context remain in English. This includes both monolingual and mixed-language tool settings.
S4 (L2 Domain): A fully localized setting where dialogue, tool schemas, policies, task descriptions, and database content are all in $L_2$ .

Experimental Setup

The study evaluates three recent Large Language Model (LLM) agents:

GPT-5-Mini: A proprietary model.
Qwen3-235B-A22B-Instruct: A representative open-source model.
Kimi-K2.5: A larger-scale open-source model.

Performance is measured using pass@1 (quality/success rate) and a derived robustness metric ( $\rho_3$ ), which measures the probability of consistent success across three independent trials. The evaluation covers three domains: Retail, Airline, and Telecom.

Key Results

Performance Degradation in Localized Settings

The results reveal a consistent and sharp decline in agent performance as the evaluation setting moves away from English-only:

Interaction vs. Full Localization: While agents transfer reasonably well when only the conversation language changes (S2), quality and robustness degrade significantly when tool interfaces and task contexts are also localized (S3 and S4).
Domain Adaptation: The most severe performance drops occur in the L2 Domain (S4) scenario. For example, in Thai and Filipino, pass@1 scores drop to the low-0.4 range, compared to higher English baselines.
Language Variance: The severity of degradation varies by language. Thai and Filipino show the largest declines, potentially due to distinct writing systems or lower-resource coverage. However, even Latin-script languages like Filipino exhibit significant weaknesses when agents must reason over fully localized dialogues and schemas.

Quality vs. Robustness Divergence

The study finds that quality (pass@1) and robustness ( $\rho_3$ ) do not always correlate. In some languages (e.g., Indonesian, Filipino), pass@1 drops in the L2 Domain while robustness remains relatively stable, suggesting that while agents solve fewer tasks overall, their successful attempts are reproducible. Conversely, Thai shows declines in both metrics, indicating a breakdown in both average success and consistency.

Model and Domain Differences

Model Variance: No single model dominates across all SEA languages. Kimi-K2.5 generally performs best in Vietnamese, Filipino, and Chinese, while GPT-5-Mini leads in Indonesian and Thai. Qwen3-235B consistently shows lower pass@1 but often exhibits stronger robustness in lower-resource settings.
Domain Difficulty: The Retail domain is consistently the most challenging across all languages due to complex policies and state updates. Notably, domain-specific strengths in English (e.g., Qwen performing well in Retail) do not necessarily transfer to localized settings.

Error Analysis and Language Drift

Error Types: Critical errors (deviations from user intent) increase significantly in L2 settings, particularly in Thai and Filipino.
Language Drift: While agents do exhibit "language drift" (switching to English mid-conversation), the analysis shows a very weak correlation ( $R^2 = 0.014$ ) between language correctness and task success. This suggests that task failures are driven more by the inability to reason over localized schemas and policies rather than simple language switching.
Mixed-Language Tools: Introducing tools in multiple languages simultaneously causes only a modest drop in performance compared to monolingual localization, suggesting agents can handle some multilingual tool diversity.

Significance and Claims

The paper claims that SEATauBench exposes a critical gap between the growth of SEA evaluation resources and the readiness of current agents for sovereign AI deployment. Its primary contributions are:

Diagnostic Benchmark: It provides the first framework to systematically evaluate agent capabilities in SEA languages across varying levels of localization, moving beyond static benchmarks to interactive, tool-mediated tasks.
Unreliability of English Proxies: The study demonstrates that English-only evaluations are an unreliable proxy for SEA language capabilities. High English performance does not guarantee robustness in localized environments, particularly when tool schemas and domain policies are translated.
Adaptation Pipeline: The authors provide a reusable, non-breaking translation pipeline that allows for the rapid adaptation of tool-agent benchmarks to new languages while preserving execution integrity.
Recommendation: Based on correlation analysis, the authors suggest that the Filipino subset of SEATauBench can serve as a representative proxy for estimating performance across other SEA languages, offering a more efficient evaluation strategy for the region.

Ultimately, SEATauBench establishes a rigorous foundation for building reliable, multilingual agents for linguistically diverse regions, highlighting that sovereign AI requires more than just translation; it demands robust adaptation of tool interfaces and domain reasoning capabilities.