ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

This paper introduces ThaiSafetyBench, an open-source benchmark of 1,954 culturally nuanced Thai prompts that reveals significant safety gaps in current LLMs—particularly open-source models and those facing culturally specific attacks—while providing a high-performance classifier and leaderboard to advance safety evaluation in the Thai context.

The Gist

Imagine you have built a super-smart robot librarian who can answer any question in any language. You've trained it mostly on English books, so it's very good at knowing what's "safe" to say in English. But what happens when you ask it a question in Thai? Or what if the question relies on deep Thai cultural secrets, like local gossip, specific social rules, or sensitive topics about the Thai monarchy?

This paper is like a safety inspector who just arrived to test that robot librarian, but specifically for the Thai language and culture. Here is the story of their findings, broken down simply:

1. The Problem: The "English Glasses" Blind Spot

Most safety tests for AI are like checking a car's brakes using only English instructions. The researchers found that while the AI is great at following safety rules in English, it often gets confused or breaks the rules when speaking Thai.

Why? Because the AI doesn't understand the "vibe" or the hidden cultural rules of Thailand. It's like a tourist who knows the dictionary definition of a word but doesn't know that saying it in a specific Thai village is considered extremely rude or dangerous.

2. The Solution: ThaiSafetyBench (The "Thai Trap" Test)

To fix this, the team built a new test called ThaiSafetyBench.

• The Trap: They created 1,954 tricky questions (prompts) written in Thai.
• The Mix: Some questions were just general "bad" things (like "How do I make a bomb?"), but the real test was the culturally specific traps. These were questions designed to trick the AI using Thai slang, local social norms, or sensitive political topics that an English-trained AI wouldn't understand.
• The Goal: See if the AI would accidentally say something harmful when asked in a Thai context.

3. The Results: Who Passed the Test?

They tested 24 different AI models (both the big, expensive ones from companies like Google/OpenAI and the free, open-source ones anyone can download).

• The "Rich" Kids vs. The "Open" Kids: The big, closed-source models (like GPT-4.1) were like experienced security guards. They knew exactly when to say "No" and kept their cool. The open-source models were more like enthusiastic interns; some were great, but many let the "bad guys" (harmful requests) slip through the door.
• The Cultural Gap: The most shocking finding was that the AI was much worse at handling Thai-specific cultural traps than general bad questions. It's as if the guard knows how to stop a thief with a gun, but gets confused when someone tries to bribe them with a local delicacy. The AI didn't understand the cultural nuance, so it failed the safety check.
• Size Matters (But Not Everything): Generally, bigger AI models were safer (like a bigger net catching more fish). However, some smaller models that were specifically trained on Thai data performed surprisingly well, proving that quality training data matters more than just raw size.

4. The Toolkit: Making Safety Cheaper and Easier

Evaluating AI usually requires expensive supercomputers or paying for expensive AI judges. To help everyone else do this too, the team created two free tools:

• ThaiSafetyClassifier: A tiny, lightweight AI "spotter" that can look at a conversation and instantly say, "That's safe" or "That's dangerous." It's like a metal detector that is cheap to run but just as accurate as the expensive security scanners.
• The Leaderboard: They built a public scoreboard (like a video game high-score list) where anyone can see how safe their AI is in Thai. This encourages everyone to keep improving their models to get a better rank.

The Big Takeaway

This paper is a wake-up call. You can't just translate safety rules from English to Thai and expect them to work. Safety needs to be culturally baked in.

If we want AI to be safe for Thai people, we need to test it with Thai traps, understand Thai culture, and build tools that respect those specific nuances. Otherwise, our AI might be polite in English but accidentally offensive or dangerous in Thai.

In short: They built a Thai-specific "stress test" for AI, found that many models are culturally blind, and gave the community free tools to fix it.

Technical Summary

Here is a detailed technical summary of the paper "THAISAFETYBENCH: ASSESSING LANGUAGE MODEL SAFETY IN THAI CULTURAL CONTEXTS" presented at the ICLR 2026 Workshop.

1. Problem Statement

Large Language Model (LLM) safety evaluation has historically been dominated by English-centric datasets and benchmarks. This creates a significant gap in understanding and mitigating risks in non-English languages, particularly those with distinct cultural, social, and political nuances.

• The Gap: Existing safety datasets often fail to capture cultural specificities, leading to higher Attack Success Rates (ASR) for non-English languages compared to English.
• The Thai Context: There is a lack of publicly available, high-quality safety datasets tailored to the Thai language and culture. This hinders the ability to evaluate model robustness against culturally specific attacks (e.g., those involving the monarchy, local social etiquette, or regional misinformation) and limits the development of safe Thai-tuned models.

2. Methodology

A. Dataset Construction: ThaiSafetyBench

The authors introduce ThaiSafetyBench, a curated dataset of 1,954 malicious prompts in Thai.

• Taxonomy: The dataset adopts a hierarchical taxonomy covering 6 risk areas and 17 types of harm, including:
  • Discrimination, Exclusion, Toxicity
  • Human–Chatbot Interaction Harms (e.g., anthropomorphism, mental health overreliance)
  • Information Hazards (leakage of private/sensitive data)
  • Malicious Uses (illegal activities, unethical actions)
  • Misinformation Harms
  • Thai Socio-Cultural Harm (unique to Thailand, covering border issues, cultural destruction, COVID-19 misinformation, Monarchy, and Misbehavior).
• Construction Sources:
  1. Translation: 48.1% of samples are translated from the English Do-Not-Answer dataset, refined by native Thai annotators for naturalness.
  2. Curation: 19.2% are filtered from existing Thai safety datasets (e.g., Typhoon2), removing noisy or irrelevant entries.
  3. Generation: 32.7% are newly generated using uncensored LLMs (Grok 3) and manual crafting based on Thai social etiquette guidelines, specifically targeting cultural nuances.
• Compliance: To adhere to Thai regulations regarding the Monarchy, the publicly released subset excludes specific "Monarchy" category samples, resulting in a public release of 1,889 samples.

B. Evaluation Framework

• Models Evaluated: 24 LLMs were tested, including:
  • Closed-source commercial models (e.g., GPT-5, Claude 4.5).
  • Open-source multilingual models (e.g., Llama 3.3, Qwen2.5, Gemma 3).
  • Southeast Asia-tuned and Thai-tuned models (e.g., SeaLLMs, Typhoon2, OpenThaiGPT).
• Judging Mechanism: An automated "LLM-as-a-Judge" approach was used. GPT-4.1 and Gemini-2.5-Pro served as independent evaluators to classify responses as "safe" or "harmful."
• Metric: The primary metric is the Attack Success Rate (ASR), defined as the proportion of harmful responses. The final score is the average ASR across both judges.

C. Reproducibility Tool: ThaiSafetyClassifier

To reduce the cost and latency of using large LLMs for evaluation, the authors fine-tuned a DeBERTaV3-base model (using LoRA) as a binary classifier.

• Input: Prompt-Response pairs.
• Training: Trained on 46,893 prompt-response pairs with class-balanced loss to handle imbalance (79.5% safe vs. 20.5% harmful).
• Performance: Achieved a weighted F1 score of 84.4%, showing high correlation with GPT-4.1 judgments.

3. Key Results and Analysis

• Closed-Source vs. Open-Source: Closed-source models (e.g., GPT-5, Claude 4.5) generally demonstrated superior safety performance (lower ASR) compared to open-source counterparts. However, some open-source models (e.g., SeaLLMs-v3-7B) showed safety levels comparable to closed-source models.
• Cultural Vulnerability: A critical finding is that Thai-specific, culturally contextualized attacks have a significantly higher ASR than general Thai-language attacks. Models struggle more with prompts rooted in local cultural norms, social hierarchies, and political sensitivities.
• Risk Area Performance:
  • Models performed well against Information Hazards (low leakage).
  • Models performed poorly against Thai Socio-Cultural Harms, indicating a persistent "cultural attack gap."
• Model Size and Safety:
  • Scaling Law: Generally, larger models exhibit lower ASR (better safety).
  • Training Data Quality: Smaller models with high-quality, safety-aligned training data (like SeaLLMs) can outperform larger, less-aligned models.
  • Continual Pretraining (CPT): The impact of CPT (e.g., Thai-tuning) on safety is inconsistent; success depends heavily on the quality and safety-filtering of the pretraining data.

4. Key Contributions

1. ThaiSafetyBench Dataset: The first open-source, culturally grounded safety benchmark for Thai, featuring 1,954 prompts across 6 risk areas with a hierarchical taxonomy.
2. Comprehensive Evaluation: A systematic assessment of 24 diverse LLMs, providing the first comparative safety rankings for Thai-tuned models.
3. ThaiSafetyBench Leaderboard: A public platform for tracking model safety performance and encouraging community submissions.
4. ThaiSafetyClassifier: A lightweight, high-performance (F1 84.4%) classifier released to enable cost-effective, reproducible safety evaluations without relying on expensive API calls to judge models.

5. Significance and Future Work

• Significance: This work highlights that safety alignment is not language-agnostic. It proves that models trained primarily on English data or generic multilingual data fail to handle specific cultural risks in Thai. It provides the necessary infrastructure (dataset, classifier, leaderboard) to drive the development of safer, culturally aware AI in Thailand.
• Limitations: The current evaluation relies on simple prompt-based jailbreaks and does not assess the utility of "jailbroken" responses or overly restrictive models that reject benign queries.
• Future Directions: The authors plan to expand the dataset with non-risky prompts, explore sophisticated jailbreaking techniques, and use the benchmark to actively train safer Thai cultural LLMs.

In conclusion, ThaiSafetyBench serves as a critical step toward "Principled Design for Trustworthy AI" by demonstrating that true safety requires deep integration of local cultural context, not just linguistic translation.