Technical Summary: TukaBench

Problem Statement

Safety evaluation for Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African languages, critically underexplored. While existing jailbreak benchmarks (e.g., JailbreakBench) and safety evaluations cover high-resource languages, they fail to account for the linguistic and cultural nuances of LRLs. Previous work has shown that translating unsafe prompts into low-resource languages can bypass safety safeguards, yet there is a lack of human-curated benchmarks specifically for African languages to systematically study how language, cultural grounding, and prompt evasiveness affect model safety. Furthermore, current evaluation methods often rely on "LLM-as-a-judge" frameworks, which may suffer from reduced reliability in low-resource settings and fail to distinguish between genuine safety refusals and model comprehension failures.

Methodology

The authors introduce TukaBench, a jailbreak benchmark designed for seven African languages: Amharic, Hausa, Igbo, Chichewa, Kiswahili, Yorùbá, and isiXhosa. The benchmark extends JailbreakBench (JBB) through four complementary settings to isolate the effects of language, culture, and code-switching:

1. Afri-JBB-Harm/Benign: Direct human translations of JBB harmful and benign prompts into target languages, preserving original Western contexts and named entities.
2. Afri-JBB-Culture: JBB harmful prompts adapted to African contexts (introducing culturally relevant named entities and references) before human translation.
3. AfriJail-Mono: 343 culturally grounded adversarial prompts originally authored in English by data collectors in Nigeria through interaction with GPT-5.2, covering local governance, regional conflicts, and specific social harms. These were then translated and culturally localized.
4. AfriJail-CS: Code-switched versions of AfriJail-Mono, mixing African languages with English within single utterances to reflect common multilingual speech patterns.

Construction Pipeline:
All prompts underwent a three-stage pipeline: machine translation (using Google Translate or AfriqueQwen-8B for Yorùbá), automatic quality estimation (using SSA-COMET-QE), and human post-editing by native speakers. Post-editors corrected errors and performed cultural localization where required.

Evaluation Protocol:

• Models: Thirteen LLMs were evaluated, including closed-source models (GPT-3.5, 4o, 5.2, Grok-3/4/4.3, Gemini-2.5/3.1) and open-weight models (Gemma, Llama, DeepSeek, Qwen, GPT-OSS).
• Attack Strategies: Models were tested under Direct Prompting and Boundary Point Jailbreaking (BPJ), a black-box attack that adds character-level noise to elicit non-refusing responses.
• Labeling Scheme: A three-way evaluation scheme was introduced:
  • Jailbroken: The model fulfills the harmful intent.
  • Refused: The model explicitly refuses.
  • Deflected: The model responds off-target or fails to comprehend the prompt (neither a refusal nor a harmful completion).
• Validation: To address the reliability of LLM-as-a-judge in LRLs, a subset of 1,500 model responses was validated by human native-speaker annotators.

Key Results

1. Language and Cultural Grounding Impact:

• Deflection vs. Attack Success: Switching from English to African languages does not consistently increase the Attack Success Rate (ASR). Instead, it significantly increases the Deflection Rate. Models often fail to understand prompts in LRLs, responding off-target rather than refusing or complying.
• Cultural Grounding: Culturally adapted prompts (Afri-JBB-Culture and AfriJail-Mono) resulted in higher rates of both Jailbroken and Deflected responses compared to direct translations. This suggests that translation-only benchmarks systematically underestimate deployment risks, as culturally grounded prompts better match real-world contexts and exploit model vulnerabilities.
• Resource and Script Effects: Higher-resource languages (e.g., Swahili) generally exhibited lower failure rates (both Jailbroken and Deflected). However, Amharic (written in the non-Latin Ge'ez script) showed significantly higher deflection rates than other languages with comparable resource levels, suggesting that tokenization inefficiencies ("token tax") degrade comprehension.

2. Attack Strategies:

• BPJ: Boundary Point Jailbreaking increased ASR but did not eliminate deflection, indicating that while noise can bypass filters, comprehension remains a bottleneck.
• Code-Switching: Code-switched prompts (AfriJail-CS) reduced deflection rates compared to monolingual African prompts, suggesting that English fragments help models engage with the request. However, this did not uniformly increase ASR; improved comprehension sometimes led to refusal rather than compliance, depending on the model's underlying safety behavior.

3. Evaluation Reliability:

• Judge-Human Agreement: The reliability of LLM-as-a-judge decreased for lower-resource languages and non-Latin scripts. Agreement was highest for Swahili (~80%) and lowest for Amharic and lower-resource Latin-script languages like Yorùbá and Igbo (<60%).
• Comprehension Failures: The introduction of the Deflection label revealed that many responses previously misclassified as "safe" or "refused" were actually comprehension failures. When Deflection and Jailbroken rates are considered jointly, a clearer resource-based pattern emerges: lower-resource languages exhibit higher combined failure rates.

Significance and Claims

The paper claims that TukaBench provides the first human-curated jailbreak benchmark for African languages, moving beyond direct translation to include cultural adaptation and code-switching. Its primary contributions are:

1. Revealing Comprehension Bottlenecks: The study demonstrates that in LRLs, safety evaluation is constrained not just by model alignment but by fundamental comprehension failures. High deflection rates indicate that models often do not understand the prompt, rendering standard ASR metrics insufficient.
2. Cultural Grounding as a Vulnerability: The results show that culturally grounded prompts elicit higher failure rates than direct translations, suggesting that safety evaluations must account for local context and named entities to be accurate.
3. Methodological Improvement: By introducing the Deflection category and validating LLM-as-a-judge outputs with human annotations, the paper highlights the limitations of current automated evaluation in low-resource settings and proposes a more nuanced framework for assessing safety.

The authors conclude that while newer model generations show improvements, significant gaps remain in comprehension and safety for African languages, particularly for non-Latin scripts and lower-resource languages. They emphasize that safety assessment in these regions requires moving beyond simple translation to culturally grounded evaluation.