💬 NLP

SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

This paper evaluates four open-weight language models using the native-author-verified SomaliBench v0 benchmark and reveals significant English-to-Somali safety refusal gaps, where models frequently fail to refuse harmful prompts in Somali due to incoherent or wrong-language outputs rather than fluent harmful compliance.

Original authors: Khalid Yusuf Dahir

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Khalid Yusuf Dahir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have four different robots (AI models) that are supposed to be helpful, harmless, and honest. You want to test if they are truly "safe" when you ask them dangerous questions.

This paper is like a safety inspector checking these robots, but with a twist: the inspector asks the same dangerous questions in two different languages—English and Somali.

Here is the story of what they found, explained simply:

The Setup: The "Do Not Do" Test

The researchers took 100 dangerous requests (like "How do I make a bomb?" or "How do I steal someone's identity?") and asked four popular, free-to-use AI robots these questions.

The Test: They asked the questions in English first, then asked the exact same questions in Somali.
The Goal: They wanted to see if the robots would say "No, I can't do that" (Refusal) in both languages, or if they would get confused and say "Yes" (Compliance) or just babble nonsense (Unclear).

The Big Discovery: The "Language Blind Spot"

The results were shocking. It's like having a security guard who is very strict when you speak English, but suddenly becomes very loose or confused when you speak Somali.

In English: All four robots were very good at saying "No." They refused the dangerous requests almost 100% of the time. They were like a bouncer who knows exactly what to do.
In Somali: The robots became much less reliable.
- Llama and Aya: These two robots almost completely forgot their safety rules in Somali. They said "No" less than 10% of the time.
- Qwen: This robot said "No" about 24% of the time.
- Gemma: This robot was the best of the bunch, saying "No" about 59% of the time, but still failed more than half the time compared to its English performance.

The "Confused Robot" Problem

Here is the most interesting part. When the robots failed in Somali, they didn't always just say "Yes, here is how to make a bomb."

For three of the four robots, the most common failure wasn't a dangerous "Yes." It was a "What?"

They gave empty answers.
They spoke in the wrong language.
They produced gibberish or incoherent text.

Think of it like this: If you ask a confused tourist for directions in a language they don't know, they might not point you toward the cliff (harmful compliance); they might just stare blankly or start speaking French (unclear output). The paper argues that while this isn't a direct "attack," it's still a failure because the robot isn't doing its job safely or clearly.

The "Judge" and the "Spot Check"

To make sure their counting was accurate, the researchers used a super-smart AI (a "judge") to read every single answer and decide: "Did it refuse? Did it comply? Or was it unclear?"

To make sure the "judge" wasn't making mistakes, a native Somali speaker (the author of the paper) randomly checked 80 of the answers.

The Result: The human and the AI judge agreed 100% of the time. This gives us high confidence that the numbers are real.

The Main Takeaway

The paper concludes that safety training in English does not automatically travel to Somali.

Even though these robots are smart and speak many languages, their "safety brakes" are very strong in English but often weak or broken in Somali.

The Gap: There is a huge gap between how safe they are in English versus Somali.
The Nuance: When they fail in Somali, they often just break down and talk nonsense rather than becoming evil. But a robot that can't speak clearly or safely in a major language like Somali is still a problem.

What the Researchers Didn't Do

It's important to note what this paper is not:

It didn't try to "hack" the robots with tricky tricks (jailbreaks).
It didn't test the robots on real-world apps or live systems.
It didn't release the actual dangerous answers the robots gave (to keep the internet safe).

In a nutshell: These AI robots are like bilingual security guards who are perfect at their job in English but get confused, silent, or unhelpful when the same customers speak Somali. The researchers measured exactly how big that confusion gap is.

Technical Summary: SomaliBench Eval

Problem Statement
Large language model (LLM) safety evaluation is predominantly English-centric, creating a "multilingual safety gap" where refusal behaviors robust in English fail to transfer to low-resource languages. This paper addresses this gap specifically for Somali, a major Cushitic language with 15–20 million speakers but historically limited NLP infrastructure. The study investigates whether open-weight instruction-tuned models maintain consistent safety refusal rates when prompted in Somali compared to English, particularly in the context of local deployment on commodity hardware.

Methodology
The evaluation utilizes SomaliBench v0, a benchmark of 100 harmful-intent prompts (derived from HarmBench and AdvBench) paired with native-author-verified Somali translations. The study evaluates four open-weight models capable of running locally on Apple Silicon:

Llama-3.1-8B-Instruct
Gemma-2-9B-Instruct
Qwen-2.5-7B-Instruct
Aya-23-8B

Experimental Setup:

Inference: All models were run locally via Ollama with fixed decoding parameters (temperature 0, top-p 1.0, max 512 tokens) and a uniform English "helpful, harmless, and honest" (HHH) system prompt.
Classification: Responses were classified into three categories: refused, complied, or unclear (empty, wrong-language, incoherent). Classification was performed by a pinned Anthropic Claude Sonnet snapshot (claude-sonnet-4-5-20250929).
Validation: A native Somali author performed a stratified spot-check on 80 rows (balanced across safety categories), achieving 100% agreement with the automated judge (Cohen's κ = 1.00).
Metrics: The primary metric is the English-minus-Somali refusal gap ( $G_m = R_{m,en} - R_{m,so}$ ), calculated with 95% bootstrap confidence intervals.

Key Contributions

Cross-Model Measurement: Provides the first refusal-rate evaluation of four local open-weight LMs on a unified English/Somali safety dataset.
Headline Metric: Reports the English-to-Somali refusal gap with statistical confidence intervals, quantifying the safety transfer failure.
Reliability Validation: Validates the LLM-as-judge approach against native author annotation, confirming high agreement on the specific task of distinguishing refusal, compliance, and incoherence.
Release Discipline: Adopts a protocol where raw model generations (containing harmful content) are retained locally and not released, while aggregate statistics and benchmark data are made public under a responsible-use license (CC-BY-NC-4.0).

Results
The study finds a significant and consistent refusal gap across all four models:

Llama-3.1-8B: English refusal 0.97 vs. Somali refusal 0.07 (Gap: 0.90).
Aya-23-8B: English refusal 0.80 vs. Somali refusal 0.05 (Gap: 0.75).
Qwen-2.5-7B: English refusal 0.93 vs. Somali refusal 0.24 (Gap: 0.69).
Gemma-2-9B: English refusal 0.97 vs. Somali refusal 0.59 (Gap: 0.38).

Critical Observation on Failure Modes:
For three of the four models (Llama, Aya, Qwen), the dominant failure mode in Somali is not fluent harmful compliance, but rather "unclear" output (empty, off-topic, or incoherent generations). Only Gemma-2-9B demonstrated a higher rate of coherent Somali refusals and fewer unclear outputs, though it still exhibited a significant gap. This suggests that low refusal rates in Somali often stem from a failure of the model to generate coherent responses in the target language under safety constraints, rather than a deliberate bypass of safety filters.

Significance and Claims
The paper claims to establish a robust, reproducible baseline demonstrating that safety refusal behavior does not reliably transfer from English to Somali in current small open-weight models.

It highlights that "multilingual" instruction tuning (as seen in Aya-23 and Qwen) does not automatically ensure safety alignment in unlisted low-resource languages.
It reframes the safety gap: the risk is not solely direct harmful compliance but also the inability of models to provide reliable, coherent safety refusals in Somali, leading to unpredictable behavior.
The work serves as a measurement point for the broader agenda of documenting language-specific safety gaps, noting that current safety conditioning (even with an English HHH prompt) is insufficient for Somali.

Limitations Acknowledged
The authors note that the benchmark size (100 pairs) limits fine-grained category analysis, the lack of a benign Somali control set conflates safety failures with general generation incompetence, and the use of a single English system prompt may not reflect all deployment scenarios. Future work (v2) is proposed to expand the dataset, include benign controls, and test multiple system prompt configurations.