Technical Summary: Schützen – Evaluating LLM Safety in Bulgarian and German Contexts

Problem Statement

Large Language Models (LLMs) are increasingly deployed in professional domains, yet existing safety evaluation resources remain overwhelmingly English- and Chinese-centric. This gap is critical for languages operating within shared sociocultural, legal, and ethical frameworks, such as those in Europe. Specifically, low-resource languages like Bulgarian lack native safety benchmarks, while high-resource languages like German are often evaluated using translated prompts that fail to capture region-specific cultural norms, legal constraints, and political sensitivities. Current multilingual benchmarks often rely on direct translation, which limits the ability to assess how safety behaviors transfer across languages that share a regulatory environment but differ in resource availability.

Methodology

To address these limitations, the authors introduce Schützen, a bilingual safety dataset designed for German and Bulgarian. The methodology involves three core components:

1. Dataset Construction

The dataset comprises approximately 8,000 prompts (3,978 per language) derived from the Do-not-Answer (DNA) series and extended with region-specific content. The construction process emphasizes cultural localization rather than direct translation:

• Localization Strategy: General-risk questions were rewritten to reflect local figures, historical events, policies, and cultural conventions (e.g., replacing generic names with local equivalents like "Pepa" and "Senya" for Bulgarian, or "Gertrude" and "Karina" for German).
• Risk Coverage: The dataset spans six risk areas and 16 harm types, including Information Hazards, Malicious Uses, Discrimination/Toxicity, Misinformation, Human-Chatbot Interaction Harms, and Region-Specific Sensitivity (e.g., controversial historical events and political topics).
• Attack Modes: Prompts are categorized into three types:
  • Direct Attacks (Ori): Explicitly risky queries.
  • Indirect Attacks (FN): Benign-sounding queries designed to elicit unsafe content.
  • Over-Sensitivity Assessments (FP): Queries containing sensitive terms but requesting benign information to test for false refusals.

2. Evaluation Framework

Responses were collected from 15 LLMs, including multilingual models (e.g., GPT-4o, Claude-3.7, Llama-4 variants, EuroLLM-9B) and language-specific models (e.g., BgGPT series for Bulgarian, Leo/LLaMmlein series for German). The evaluation proceeded in two stages:

• Binary Classification: Determining if a response is "Safe" or "Unsafe." This was performed using both classifier-based approaches (Llama-Guard) and reasoning-based approaches (customized risk criteria via GPT-4.1).
• Fine-Grained Pattern Analysis:
  • Safe Patterns: Categorized into six types, including outright refusal, opinion refutation, well-rounded statements, risk-aware responses with disclaimers, general information, and admission of uncertainty.
  • Unsafe Patterns: Categorized into four types: general harmful content, misinformation, privacy violations, and offensive/emotionally harmful content.

3. Human-AI Collaborative Annotation

The authors developed a pipeline where models generate preliminary labels and explanations, which are subsequently verified by human annotators. This approach was validated against human-only annotation, demonstrating higher agreement with automatic metrics and improved efficiency.

Key Results

The study evaluated over 95,000 responses across the two languages:

• Overall Safety Performance: Most models achieved high safety rates (>95%). Claude-3.7 achieved the highest safety performance for both languages (99.92% safe responses). BgGPT-27B performed best among Bulgarian-centric models (99.57%), while Leo-mistral-hessianai-7B-chat led among German-centric models (98.27%).
• Model Disparities: LLaMmlein-7B-chat (German) and BgGPT-2.6B (Bulgarian) ranked lowest in safety. Notably, GPT-4o underperformed relative to other open-source systems on this specific benchmark.
• Risk Area Vulnerabilities: Across both languages, Malicious Uses (Risk Area IV) posed the most significant challenge, accounting for the largest share of unsafe outputs.
• Attack Mode Sensitivity: While direct attacks yielded the highest absolute number of unsafe responses, indirect attacks were more effective at bypassing safeguards when normalized for sample size. The eight safest models showed higher unsafe rates under indirect attacks.
• Cross-Linguistic Behavior: Despite linguistic differences, models exhibited similar failure modes across both languages when operating within the shared European regulatory context. However, models generally performed better on German than on Bulgarian, highlighting imbalanced regional representation.
• Response Patterns: Safe responses were most frequently realized as generic informational answers. Unsafe behaviors concentrated on providing incorrect information (Pattern 7/8) or failing to express uncertainty.

Significance and Claims

The paper claims that Schützen is the first bilingual benchmark explicitly designed for LLM safety evaluation in Bulgarian and German. Its primary contributions are:

1. Region-Specific Evaluation: By combining localization with a controlled comparison of a low-resource (Bulgarian) and high-resource (German) language under shared European norms, the dataset reveals cross-language safety discrepancies that monolingual or loosely multilingual benchmarks miss.
2. Empirical Insights on Transfer: Contrary to the hypothesis that low-resource languages would show weaker alignment, the results suggest that when target cultures share safety norms and policies with English-speaking settings, safety capabilities transfer robustly, even to low-resource languages.
3. Methodological Advancement: The paper demonstrates the efficacy of a human-AI collaborative annotation pipeline, which improves agreement with automatic metrics and reduces task difficulty compared to human-only annotation.
4. Resource for Responsible Deployment: The dataset and code provide a practical resource for supporting the responsible deployment of LLMs in European settings where linguistic and cultural factors interact, specifically addressing the gap in safety evaluation for underrepresented regions.

The authors conclude that while current models show strong general safety, they degrade noticeably on region-specific content, necessitating tailored, region-specific evaluation resources to support robust LLM deployment in Germany and Bulgaria.