💬 NLP

Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps

This paper introduces RabakBench, a human-in-the-loop validated multilingual safety benchmark for Singapore's linguistic landscape that reveals significant performance gaps in state-of-the-art LLM guardrails when handling low-resource languages like Singlish, Chinese, Malay, and Tamil.

Original authors: Gabriel Chua, Leanne Tan, Ziyu Ge, Roy Ka-Wei Lee

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriel Chua, Leanne Tan, Ziyu Ge, Roy Ka-Wei Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-trained security guard for a building. This guard is excellent at spotting trouble when people speak in "Standard English." They know exactly what a rude comment, a threat, or a dangerous idea looks like in that language.

But now, imagine that same guard is asked to watch over a bustling, multicultural neighborhood where people speak a mix of languages, slang, and local dialects. Suddenly, the guard gets confused. They might miss a real threat because it's hidden in a local joke, or they might wrongly arrest a harmless neighbor because they don't understand a cultural phrase.

This paper, RABAKBENCH, is about building a new "test" to see how well these AI security guards handle that messy, real-world neighborhood. The neighborhood in question is Singapore, a place where people constantly mix English with Chinese, Malay, and Tamil words (a mix called "Singlish").

Here is how the researchers built their test and what they found, explained simply:

1. The Problem: The "Localization Blind Spot"

Current AI safety systems are like guards who only studied a textbook on Standard English. They fail when faced with:

Code-mixing: Sentences that jump between languages.
Slang: Local words that change meaning depending on context.
Dialects: Regional ways of speaking that aren't "proper" English.

The researchers found that these AI guards often make two big mistakes:

False Negatives: They miss actual hate speech or threats because the words look like harmless slang to them.
False Positives: They flag harmless cultural jokes as dangerous, effectively silencing normal conversation.

2. The Solution: Building "RABAKBENCH"

To fix this, the team built a massive testing ground called RABAKBENCH (named after a local word meaning "extreme" or "intense"). Think of it as a simulator for AI safety.

They didn't just write these test sentences themselves; they used a clever three-step factory process:

Step 1: The "Red Team" Attack (Generate)
Imagine a group of hackers trying to trick the AI guards. The researchers used AI to generate thousands of tricky, real-world examples of Singlish, including things that sound dangerous but might be safe, and things that sound safe but are actually toxic. They even used a "critic" AI to help the "attacker" AI get better at finding weak spots.
Step 2: The "Super-Labelers" (Label)
Labeling these tricky sentences by hand is expensive and hard. So, the researchers tested several different AI models to see which ones were best at understanding human judgment. They found three "star" AI models that agreed with human experts about 70–80% of the time. They used these AI models to label the data, but they kept a human in the loop to double-check the work, ensuring the cultural nuances were correct.
Step 3: The "Cultural Translator" (Translate)
They took their Singlish test cases and translated them into Chinese, Malay, and Tamil. But here's the catch: standard translators often "sanitize" bad words to make them polite. The researchers built a special translation process that ensured the toxicity stayed the same. If a sentence was mean in Singlish, the translation had to be equally mean in Malay or Tamil, preserving the original "flavor" and intent.

The result? A dataset of over 5,000 examples covering four languages, with detailed labels on exactly why something is unsafe (e.g., is it hate speech? is it just an insult? is it a threat?).

3. The Results: The Guards Failed the Test

The researchers took 13 of the world's top AI safety systems (both commercial ones like Google and OpenAI, and open-source ones) and ran them through this new test.

The verdict was harsh:

Performance Dropped: Systems that scored 80%+ on standard English tests often dropped to below 50% on this local test.
The "Tamil Gap": The performance was especially poor for Tamil, where many systems scored below 30%.
One-Size-Fits-All Doesn't Work: The study showed that you can't just train a guard on English and expect them to work everywhere. They need specific training for local cultures.

4. Why This Matters

The paper argues that if we want AI to be safe for everyone, we can't just rely on standard English rules. We need to build safety systems that understand the "local flavor."

The Takeaway:
RABAKBENCH is like a driver's license test for AI safety in multicultural environments. It proves that current AI drivers are great on the highway (Standard English) but terrible in the city streets (local dialects). The researchers have released their test questions and answers to the public so that other developers can build better, more culturally aware safety guards for the future.

What the paper does NOT claim:

It does not claim to have fixed the AI systems yet; it only built the test to show they are broken.
It does not suggest these tests should be used for clinical or medical safety.
It does not promise that future AI will automatically be perfect; it just provides a roadmap for how to measure and improve them.

Technical Summary: RABAKBENCH

Problem Statement

Large Language Models (LLMs) frequently exhibit significant safety alignment failures when processing low-resource language varieties, including code-mixed vernaculars, regional dialects, and slang. While current guardrails are often robust in standard English, they suffer from a "localization blind spot," failing to detect localized harms while simultaneously flagging benign cultural vernaculars as unsafe. This issue is particularly acute in highly multilingual societies like Singapore, where users fluidly switch between Singlish (a creole blending English, Malay, Hokkien, and Tamil), Chinese, Malay, and Tamil. Constructing safety benchmarks for these contexts has historically been hindered by the high cost of manual annotation and the necessity for deep, localized cultural fluency.

Methodology

The authors introduce RABAKBENCH, a multilingual safety benchmark and a scalable, three-stage pipeline designed to generate, annotate, and extend safety data for Singapore's linguistic landscape. The framework integrates human-in-the-loop validation at critical junctures to ensure cultural accuracy and toxicity preservation.

1. Generate: Adversarial Example Generation

The pipeline begins by constructing a high-quality corpus of Singlish test cases through a hybrid approach:

Local Content Transformation: Organic, unstructured Singlish comments from local web forums are curated and standardized into instruction-style queries using template-based transformations.
Automated Guardrail Red-Teaming: An iterative multi-agent loop (inspired by PAIR) is deployed against five baseline guardrails (including LionGuard, OpenAI Moderation, and LlamaGuard 3). An Attack LLM generates candidate Singlish prompts, while a Critic LLM analyzes the target classifier's output to identify misclassifications (false negatives and false positives).
Human Verification: A human reviewer filters all candidate outputs to ensure authenticity and consistency with local usage, discarding nonsensical entries.

2. Label: Weak-Supervision via LLM Agreement

To address the high cost of manual annotation for Singlish, the authors employ a weak-supervision strategy:

LLM Selection: Six candidate LLMs were evaluated against a gold-standard set of 50 examples labeled by six human experts fluent in Singlish. Using the Alt-Test methodology, the authors statistically validated that specific models (Gemini 2.0 Flash, o3-mini-low, and Claude 3.5 Haiku) align significantly with human judgments.
Majority Voting: The three selected models independently assign binary labels across six fine-grained harm categories. Final labels are determined by majority vote to ensure stability.
Validation: The selected models achieved Cohen's kappa scores between 0.68 and 0.72 against human consensus, indicating substantial agreement.

3. Translate: Toxicity-Preserving Multilingual Extension

The Singlish corpus is extended into Chinese, Malay, and Tamil using a translation setup designed to preserve both semantic meaning and the intended level of toxicity:

Expert-Curated Few-Shot Construction: A pool of 20 manually verified, high-fidelity examples per language was created through iterative workshops involving Singapore-based language professionals. These examples were selected through a three-round process involving LLM candidates and human refinement.
Model Selection and Optimization: Several state-of-the-art LLMs were benchmarked for translation fidelity using direct semantic similarity and back-translation consistency. GPT-4o mini was selected as the optimal model.
Dynamic Prompting: The number of few-shot examples ( $k$ ) was dynamically ranked based on semantic similarity to the target input ( $k=15$ for Chinese, $k=10$ for Malay, $k=20$ for Tamil) to ensure the model received contextually relevant demonstrations of slang and toxicity.
Final Validation: A secondary group of language experts rated a stratified sample of the translated corpus, confirming that machine translations were comparable to human-verified baselines.

Key Contributions

RABAKBENCH Dataset: The release of the first open safety benchmark covering Singaporean language variants (Singlish, Chinese, Malay, Tamil) with over 5,000 examples. It features a fine-grained harm taxonomy with severity levels (Level 1 and Level 2) across six categories: Hateful, Insults, Sexual, Physical Violence, Self-Harm, and All Other Misconduct.
Generate-Label-Translate Framework: A reproducible pipeline that combines scalable LLM generation, efficient weak supervision via majority voting, and high-fidelity translation, all underpinned by human verification at critical stages.
Toxicity-Preserving Translation Method: A systematic approach for extending safety datasets to new languages while maintaining the original harmful intent and emotional intensity, addressing the common failure of standard translation models to sanitize or mistranslate regional idioms.
Comprehensive Evaluation: A benchmarking of 13 contemporary moderation systems (both commercial and open-source) against the dataset.

Experimental Results

Evaluations of 13 guardrails on RABAKBENCH revealed significant performance degradation compared to standard English-centric benchmarks:

Performance Gap: Models with high F1 scores on standard English datasets (e.g., WildGuard at 86.1%, DuoGuard at 74.9%) saw their scores drop drastically on RABAKBENCH (to 47.27% and 45.03%, respectively).
Linguistic Bottlenecks: Performance varied widely across languages. While some models (e.g., Qwen3Guard, gpt-oss-safeguard) achieved F1 scores above 80% in Chinese and Malay, performance in Tamil was particularly poor, with many models scoring below 30%.
Commercial vs. Open Source: Widely used commercial tools like Perspective API achieved an average F1 of only 28.40%, whereas newer open-source models like Qwen3Guard and gpt-oss-safeguard demonstrated superior multilingual capabilities, though still struggling with specific localized nuances.

Significance and Claims

The paper claims that RABAKBENCH provides a reproducible framework for building safety benchmarks in underserved communities, moving beyond the "one-size-fits-all" safety policies that fail in global deployments. The authors argue that the observed performance gaps highlight a critical "localization blind spot" where current alignment techniques are over-sensitive to keywords but under-sensitive to the pragmatic intent of regional vernaculars. This over-censorship risks "digital marginalization," forcing users to revert to standard English and eroding trust in localized AI.

The authors position RABAKBENCH not merely as a dataset for Singapore, but as a task-agnostic blueprint for evaluating safety in code-mixed and low-resource environments globally. By utilizing the Alt-Test to build high-quality datasets without requiring massive human annotation teams, the framework offers a scalable model for advancing multilingual AI safety. The dataset and evaluation code are publicly available to facilitate further research.