Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarin

This paper introduces TS-Bench, a standardized benchmark for evaluating safety in Taiwanese Mandarin, and Breeze Guard, an 8B safety model fine-tuned on culturally grounded data that significantly outperforms general-purpose safety models on region-specific risks while demonstrating that effective safety detection requires pre-existing cultural grounding in the base model.

The Gist

Imagine you have a very smart, well-read security guard for a building. This guard has read millions of books and knows how to spot danger in English perfectly. They can tell you if someone is trying to steal a wallet or spread a virus in a story written in English.

But now, imagine this guard is hired to protect a specific neighborhood in Taiwan. The people there speak a local dialect of Mandarin, use unique slang, and face specific types of tricks that don't exist in the English-speaking world.

The guard tries their best, but they keep missing the dangers. Why? Because they don't understand the local culture. They might see a phrase like "Taiwanese women" and think it's just a description, not realizing that in this specific neighborhood, it's actually a nasty insult. They might see a message about "ATM installments" and think it's a normal bank notice, not realizing it's a classic scam script used only in Taiwan.

This paper is about building a new, specialized security guard (called Breeze Guard) and a new test (called TS-Bench) to make sure AI is safe for Taiwanese people.

Here is the breakdown in simple terms:

1. The Problem: The "Global Guard" Has Blind Spots

The authors explain that big, famous AI safety models are like global security guards. They are great at catching generic bad stuff (like violence or hate speech in English), but they are blind to local cultural nuances.

• The Analogy: Imagine a guard who knows that "apple" is a fruit. But in Taiwan, people might use a specific word for a type of scam that sounds like "apple" but means something totally different. The global guard misses it because they've never heard that specific local joke or trick.
• The Result: Scammers, fake doctors, and political manipulators in Taiwan can easily trick these global AI models because the models don't "get" the local culture.

2. The Solution: A New Test (TS-Bench)

Before they could fix the guard, they needed a way to test if the guard was actually doing a good job in Taiwan. So, they created TS-Bench.

• What it is: A test bank with 400 questions. Half are dangerous traps, and half are safe questions that look like traps (to trick the guard into being too paranoid).
• The Content: These aren't just random questions. They are filled with local flavor:
  • Scams: Fake messages pretending to be from Shopee (a popular shopping app) or the government.
  • Finance: "Investment teachers" promising free stocks in LINE groups.
  • Health: Myths like "eating shrimp with lemon creates poison."
  • Politics & Hate: Specific insults used in Taiwan's political debates or against certain ethnic groups (like the Hakka community).
• The Goal: To see if an AI can spot these local dangers that a standard English-trained AI would miss.

3. The New Guard: Breeze Guard

They didn't just train a new guard from scratch. They took a smart base model called Breeze 2 (which already knows Taiwanese culture because it was trained on lots of local text) and gave it special safety training.

• The Core Idea: You can't just teach a guard safety rules; they need to understand the culture first. If the guard doesn't know what "Hakka" means in a local context, no amount of safety training will help them spot the insult.
• The Training: They fed the model thousands of examples of local scams and hate speech, teaching it to say "Stop!" when it sees them.
• The Result: When tested on TS-Bench, Breeze Guard crushed the competition. It was much better at spotting local scams and cultural hate speech than the best global safety models.

4. The Trade-off: Specialized vs. General

The paper admits something important: Breeze Guard is a specialist, not a generalist.

• The Analogy: Think of Breeze Guard as a Taiwanese detective. They are amazing at solving crimes in Taipei. But if you ask them to solve a crime in New York using English slang, they might not be as good as a New York detective.
• The Data: On English safety tests, Breeze Guard did okay, but not as well as the global models. This is expected because it was trained specifically for Taiwan. The authors argue this is a fair trade-off: you want a guard who knows your neighborhood best, even if they aren't the best at guarding every neighborhood in the world.

5. How It Works: "Thinking" vs. "Instinct"

The paper also tested two ways the guard can think:

1. Instinct Mode (No-Think): The guard instantly says "Safe" or "Danger." This is fast.
2. Thinking Mode (Chain-of-Thought): The guard takes a moment to say, "Wait, this looks like a scam because it mentions an ATM and uses urgent language..." This is slower but often more accurate for tricky, complex scams.

The Big Takeaway

This paper proves that one size does not fit all when it comes to AI safety. To keep people safe in Taiwan, you can't just use a model trained on American or British data. You need a model that understands the local language, the local jokes, the local scams, and the local culture.

Breeze Guard and TS-Bench are the first steps toward making AI trustworthy for everyone, not just the English-speaking world. It's about giving the security guard the right local map so they don't miss the real dangers hiding in plain sight.

Technical Summary

Here is a detailed technical summary of the paper "Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarin."

1. Problem Statement

While global Large Language Model (LLM) safety models perform well on standard multilingual benchmarks, they exhibit significant cultural and linguistic blind spots when applied to Taiwanese Mandarin.

• The Gap: Global models are trained predominantly on English data and lack deep coverage of region-specific risks. They often fail to detect harmful content that relies on local cultural cues, such as specific scam narratives, slang, idioms, and political terminology unique to Taiwan.
• Consequences: This leads to high false negatives (missing harmful content like localized financial fraud or hate speech) and potential misinterpretation of culturally grounded expressions as benign.
• Hypothesis: The authors posit that effective safety detection requires cultural grounding acquired during the base model's pre-training. Safety fine-tuning alone cannot inject new socio-linguistic knowledge from scratch; it can only adjust decision boundaries based on existing knowledge.

2. Methodology

The paper introduces a two-pronged approach: a new evaluation benchmark (TS-Bench) and a specialized safety model (Breeze Guard).

A. TS-Bench (Taiwan Safety Benchmark)

A standardized evaluation suite designed to quantify region-specific safety risks.

• Composition: 400 human-curated, single-turn prompts.
• Structure: Balanced between 200 harmful and 200 harmless examples.
• Risk Categories: The benchmark targets six specific domains prevalent in Taiwan:
  1. Scams (SCAM): Localized phishing (e.g., "ATM installment cancellation," impersonating Shopee/PChome).
  2. Financial Malpractice (FIN_MALPRACTICE): "Investment teacher" pump-and-dump schemes, illegal fundraising.
  3. Medical Misinformation (HEALTH_MISINFO): Folk remedies, food-combination myths (e.g., shrimp + lemon = arsenic).
  4. Gender Bias (GENDER_BIAS): Localized slurs (e.g., "Tai-Nu").
  5. Group Hate (GROUP_HATE): Ethnic stereotypes (e.g., Hakka stereotypes, slurs against Indigenous peoples or migrant workers).
  6. Political Manipulation (POL_MANIPULATION): Weaponized political labels (e.g., "Green Taliban," "Blue-White Pigs").
• Design Feature: Includes "hard negatives"—benign queries that share vocabulary and topics with harmful ones (e.g., a legitimate bank notification vs. a scam notification)—to test for over-refusal and false alarms.

B. Breeze Guard (The Model)

An 8B-parameter safety model tailored for Taiwanese Mandarin.

• Base Model: Built upon Breeze 2, a general-purpose LLM derived from Llama 3 but continued pre-trained on a large, high-quality corpus of Traditional Chinese (news, forums, legal texts). This ensures the model possesses intrinsic cultural and linguistic grounding.
• Training Strategy:
  • Supervised Fine-Tuning (SFT): Trained on a large-scale, synthesized dataset of ~12,000 prompts (expanded to ~24,000 conversation records via "think" and "no-think" variants).
  • Data Synthesis: Generated using advanced LLMs with human-in-the-loop (HITL) verification. The data includes adversarial variants, code-switching, and specific local cultural references.
  • Architecture: A text-only safety model (vision components of Breeze 2 are frozen). It supports two inference modes:
    • Think Mode: Generates a Chain-of-Thought (CoT) reasoning trace before the final label.
    • No-Think Mode: Directly outputs the safety label for faster inference.
• Output: Deterministic classification (Safe/Unsafe) with optional explanations.

3. Key Contributions

1. TS-Bench: The first standardized benchmark dedicated to measuring safety performance on localized tasks in Taiwan, covering six critical risk categories with culturally specific cues.
2. Breeze Guard: A state-of-the-art 8B safety model for Taiwanese Mandarin that demonstrates the efficacy of building safety models on culturally grounded base models rather than generic ones.
3. Empirical Validation of Cultural Grounding: The paper provides evidence that base models with strong cultural priors (Breeze 2) significantly outperform globally trained models (Granite Guardian) in detecting localized harms, even before safety fine-tuning.

4. Experimental Results

The authors evaluated Breeze Guard against Granite Guardian 3.3 (8B), a leading general-purpose safety model.

Performance on TS-Bench (Taiwan-Specific)

• Overall F1 Score: Breeze Guard achieved 0.86 (No-Think) vs. Granite Guardian's 0.69 (+0.17 improvement).
• Category Gains:
  • Scams (SCAM): Massive improvement (+0.67 F1), rising from 0.18 to 0.93. This highlights the failure of global models to recognize local scam scripts.
  • Financial Malpractice: +0.42 F1 improvement.
  • Group Hate & Political Manipulation: Significant gains, showing better recognition of local slurs and political rhetoric.
• Hard Negatives: Breeze Guard demonstrated robustness in distinguishing benign lookalikes from harmful content, reducing false positives in nuanced contexts.

Performance on Global Benchmarks (English-Centric)

• ToxicChat & AegisSafetyTest: Breeze Guard showed slightly lower performance compared to Granite Guardian (e.g., 0.49 F1 vs. 0.65 on ToxicChat).
• Interpretation: This is an expected trade-off. Breeze Guard is specialized for Taiwanese risks, while global benchmarks focus on English-centric toxicity. However, Breeze Guard still achieved competitive results (0.83 F1 on AegisSafetyTest in Think mode), suggesting reasonable generalization.

Inference Modes

• Think Mode: Performed better on Scams and English benchmarks, indicating that explicit reasoning helps disambiguate complex fraud and bridge domain gaps.
• No-Think Mode: Slightly better aggregate performance on Financial Malpractice and Group Hate, offering faster inference for in-domain content.

5. Significance and Impact

• Paradigm Shift: The work challenges the notion that safety can be achieved solely through post-hoc fine-tuning on translated data. It proves that cultural grounding in the pre-training phase is a prerequisite for effective safety detection in non-English contexts.
• Trustworthy AI in Taiwan: Breeze Guard provides a deployable defense layer for AI systems in Taiwan, addressing critical risks like financial fraud and political manipulation that generic models miss.
• Methodological Blueprint: The approach (culturally grounded taxonomy + seed-driven synthetic data + HITL verification) offers a replicable framework for developing safety models for other regions with unique linguistic and threat landscapes.
• Open Science: The authors plan to release the benchmark specifications, safety templates, and evaluation scripts to foster reproducibility and further research in localized AI safety.

Conclusion

The paper successfully demonstrates that localization is critical for AI safety. By combining a culturally grounded base model (Breeze 2) with a targeted safety dataset and a rigorous local benchmark (TS-Bench), the authors created Breeze Guard, which significantly outperforms global leaders in detecting Taiwan-specific harms while maintaining competitive performance on global standards. This establishes a new foundation for trustworthy AI deployment in the region.