💬 NLP

Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

This paper introduces TF-RefusalBench, a multilingual benchmark derived from Swiss Supreme Court rulings, to measure and mitigate "over-alignment" in criminal law contexts, demonstrating that abliteration effectively eliminates harmful model refusals while preserving task performance.

Original authors: Arthur Wuhrmann, Gaetan Stein, Daniel Brunner, Andrei Kucharavy

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arthur Wuhrmann, Gaetan Stein, Daniel Brunner, Andrei Kucharavy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The Over-Protective Librarian

Imagine a highly intelligent, multilingual librarian (the AI) working in a Swiss court. This librarian's job is to translate and summarize legal documents. However, many of these documents describe terrible crimes, like child abuse or sexual violence.

The librarian has been trained with a very strict "safety rulebook." The rulebook says: "If you see anything bad, stop immediately, refuse to read it, and shout a warning to the room."

In a normal library, this is a good thing. But in a criminal court, this is a disaster. The judges and clerks need to read these specific details to do their jobs. They aren't trying to hurt anyone; they are trying to solve a case.

Because of the AI's strict training, it keeps refusing to do its job. It says, "I can't translate this, it's too disturbing!" or it adds a giant, distracting warning label to the middle of the translation. The paper calls this "Over-Alignment." The AI is so aligned with "being safe" that it stops being helpful for its actual job.

The Experiment: The "TF-RefusalBench"

To measure exactly how bad this problem is, the authors built a special test called TF-RefusalBench.

Think of this as a "stress test" for the AI. They took 100 real, very serious court cases from Switzerland (in German, French, and Italian) and created thousands of variations of requests. They asked the AI to:

Translate the text into different languages.
Summarize the text.
Give the instructions in different languages.

They tested five different AI models to see how often they would:

Refuse: Say "No, I won't do this."
Disclaimer: Do the job but add a scary warning like "This content is disturbing."

The Findings:

It's not just about saying "No": Some models never said "No," but they added warning labels to 25% of their work. This is just as annoying for a judge as a refusal because it breaks their focus.
Language matters: The AI's reaction changed depending on which language it was speaking. For example, one model was much more likely to refuse when speaking French than when speaking German, even if the story was exactly the same.
It's random: Sometimes the AI would refuse the same request twice in a row, and other times it would do the job perfectly. It's a bit like a coin flip.

The Solutions: How to Fix the Librarian

The authors tested two ways to stop the AI from being so over-protective without making it dangerous.

1. The "System Prompt" (The Gentle Reminder)
They tried giving the AI a specific note at the start of every conversation, like: "You are a court assistant. Your job is to translate these documents faithfully, even if they are disturbing. Do not add warnings."

Result: This helped a little. It cut the refusal rate in half for one model, but it didn't fix everything. It's like telling a nervous librarian, "It's okay, just do your job," but they are still a bit jittery.

2. "Abliteration" (The Surgical Removal)
This is the big discovery. The authors found that the AI's "refusal" behavior is controlled by a specific, tiny switch in its brain (a mathematical direction in its code).

The Analogy: Imagine the AI has a "Refusal Muscle." The authors found a way to surgically remove just that muscle without hurting the rest of the body. They call this Abliteration.
Result: This worked perfectly. The AI stopped refusing entirely. It stopped adding warning labels. It did the translation perfectly.
The Catch: The paper warns that this is a "double-edged sword." By removing the refusal muscle, the AI becomes slightly more vulnerable to actual bad requests (like someone asking it to write a bomb recipe). However, for the specific, controlled environment of a Swiss court (where the AI is locked inside a secure building and only handles court documents), the authors argue this trade-off is worth it.

The Bottom Line

The paper argues that we can't just look at how often an AI says "No" to judge if it's safe. We also have to look at how often it adds annoying warnings.

For legal professionals working with sensitive criminal cases, the current "safe" AI models are actually too cautious. By using a technique called Abliteration, we can tune these models to be brave enough to do their job in court, while keeping them safe enough for the rest of the world.

Important Note: The authors are very careful to say they are not releasing the data or the modified AI to the public. Because the data contains real stories of abuse, they only give access to researchers who sign a strict agreement not to share it. They are also not releasing the "surgically modified" AI because it could be misused by bad actors if it were out in the open.

Technical Summary: Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

Problem Statement
The deployment of Large Language Models (LLMs) in legal domains, particularly within criminal law courts, faces a specific failure mode distinct from general hallucination or bias: over-alignment. While courts utilize LLMs for high-stakes tasks like translation and summarization of sensitive legal documents, standard safety alignment mechanisms often trigger inappropriate refusals or unsolicited content warnings (disclaimers) when processing legitimate case files involving violent or sexual offenses.

In the context of the Swiss Federal Supreme Court, where clerks routinely process detailed descriptions of sexual abuse and violence, these safety triggers compromise legitimate work. Existing benchmarks for over-refusal (e.g., XSTest, OR-Bench) are insufficient because they:

Focus on English-only, adversarial prompts unrelated to legal work.
Measure only "hard refusals," ignoring "soft" over-alignment where the model completes the task but appends moralizing disclaimers.
Fail to account for the multilingual nature of legal systems (French, German, Italian, English) and how language combinations affect safety triggers.

Methodology: TF-RefusalBench
To address these gaps, the authors introduce TF-RefusalBench, a multilingual benchmark derived from 100 sensitive extracts of public Swiss Federal Supreme Court rulings.

Data Construction:
- Source: 16,000 public criminal law rulings were filtered for paragraphs >1,000 characters containing terms from the LDNOOBW list (obscene/violent words).
- Severity Selection: A panel of three diverse LLM judges performed pairwise comparisons to rank extracts by content severity (graphic detail, violence, exploitation). The top 100 most severe extracts were selected, resulting in a dataset where 89% of content relates to sexual offenses/minors and 11% to physical violence.
- Augmentation: To balance the language distribution (originally 71 German, 29 French), synthetic translations were generated using a model with weak safeguards (Grok-4.3) to create a parallel corpus in French, German, Italian, and English.
- Prompt Generation: From 400 source passages, 13 prompt variants were generated per passage by varying the task (translation vs. summarization) and the language slots: source language ( $S$ ), target language ( $T$ ), and instruction language ( $P$ ). This yielded 5,200 unique prompts (15,600 total queries with 3 repetitions each).
Evaluation Metrics:
- Refusal: Did the model decline the task for ethical/content reasons?
- Disclaimer: Did the model append a warning or caveat despite completing the task?
- Judging: Three distinct LLM judges (MiMo-V2-Flash, Mistral-Small-3.2, DeepSeek-V4-Flash) scored outputs via majority vote. Inter-judge agreement was high for refusal ( $\kappa \approx 0.98$ ) and substantial for disclaimers ( $\kappa \approx 0.75\text{--}0.80$ ).

Key Results
The study evaluated five open-weight models (Llama 3.3, GPT-OSS, Apertus, Qwen3, Gemma 4) and revealed that over-alignment is multifaceted and model-specific:

Beyond Refusal: Refusal rates alone are insufficient. While Qwen3 showed 0% refusal and disclaimer, Gemma 4 and Apertus rarely refused but added disclaimers to up to 26.5% and 20.4% of outputs, respectively.
Language Asymmetry: Safety triggers are heavily dependent on the output language. For instance, Llama's refusal rate peaked when producing French (13.6%) compared to German (1.9%), while Gemma's disclaimer rate peaked in English (41.2%). This suggests safety tuning is applied unevenly across languages.
Prompt-Language Concordance: Instructing a model in the language of the sensitive source text (concordant) significantly increased refusal rates compared to using the target language (discordant). For Llama, concordant instructions increased refusal odds by 1.6x; for GPT-OSS, by 2.6x.
Stochasticity: Over-alignment is not deterministic. Variance decomposition showed that ~27% of the variation in refusal/disclaimer behavior is due to sampling stochasticity, meaning the same prompt can yield different results across runs.

Mitigation Strategies
The paper investigates two approaches to mitigate over-alignment in the Llama 3.3-70B model:

System Prompting:
- A short system prompt identifying the user as a court assistant reduced refusal rates by half.
- An "authorization" prompt (explicitly permitting sensitive content) suppressed disclaimers more effectively than refusals.
- Matching the system prompt language to the request language further reduced refusals.
Abliteration:
- The authors applied abliteration (removing refusal directions from model activations via orthogonalization) using the Heretic implementation.
- Effectiveness: This method eliminated refusals entirely (from 6.8% to 0%) and reduced disclaimers (from 7.9% to 5.1%).
- Cost: While task quality (faithfulness) remained nearly neutral (drop of <0.1 on a 5-point scale), the intervention significantly increased vulnerability to harmful requests. On the HarmBench, attack success rates rose from 14.5% to 55.5%, and TruthfulQA scores dropped by 7.1 points.

Significance and Claims
The paper claims that over-alignment in legal contexts is a distinct, measurable phenomenon that cannot be solved by standard English-only refusal benchmarks. Its primary contributions are:

TF-RefusalBench: A gated-access benchmark designed to diagnose model behavior on legitimate, sensitive legal work across multiple languages.
Diagnostic Insight: Demonstrating that "soft" over-alignment (disclaimers) is as detrimental to legal workflows as hard refusals and is highly sensitive to language combinations.
Practical Mitigation: Showing that while system prompting offers partial relief, abliteration is the most effective method for eliminating refusals in on-premises deployments, provided the institution accepts the trade-off of reduced general safety capabilities.

The authors emphasize that their approach is specific to the institutional context of the Swiss Federal Supreme Court, where the trade-off between safety and the ability to process legitimate criminal law documents is managed within a controlled, on-premises environment. They do not claim these methods are universally safe for public deployment but argue they are necessary for the specific, high-stakes utility of LLMs in criminal law.

The Problem: The Over-Protective Librarian

The Experiment: The "TF-RefusalBench"

The Solutions: How to Fix the Librarian

The Bottom Line

Technical Summary: Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

More like this