💬 NLP

Are Aligned Large Language Models Still Misaligned?

This paper introduces Mis-Align Bench, a unified benchmark and the SAVACU dataset of over 382,000 samples, to evaluate Large Language Models' misalignment across safety, value, and cultural dimensions simultaneously, revealing that models optimized for single dimensions suffer from high false failure rates and lower alignment scores when faced with joint real-world conditions.

Original authors: Usman Naseem, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir, Sushant Kumar Ray, Abdullah Mohammad, Agrima Seth

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Usman Naseem, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir, Sushant Kumar Ray, Abdullah Mohammad, Agrima Seth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to run a busy, international family reunion. You have three main rules for them:

Safety: They must never say anything dangerous or hurtful (like suggesting someone drink poison).
Values: They must respect human morals (like being kind and honest).
Culture: They must understand that different families have different traditions (like knowing that in some cultures, toasting with alcohol is normal, while in others, it's a big no-no).

The Problem: The "Specialist" vs. The "Generalist"

For a long time, researchers tested AI assistants by checking just one rule at a time.

They asked: "Is this answer safe?" (Yes/No).
They asked: "Is this answer moral?" (Yes/No).
They asked: "Is this answer culturally correct?" (Yes/No).

The paper argues that this is like testing a chef by only asking, "Can you chop onions?" and "Can you bake a cake?" separately. You might get a great onion-chopper and a great baker, but if you ask them to make a whole meal where the onion sauce needs to be baked into a cake, they might fail miserably.

In the real world, an AI has to satisfy all three rules at the exact same time.

The New Tool: "Mis-Align Bench"

The authors built a new testing ground called Mis-Align Bench. Think of it as a "Stress Test" for AI.

Instead of asking simple questions, they created a massive library of 382,000 tricky scenarios (called SAVACU). These scenarios force the AI to juggle safety, values, and culture simultaneously.

Example Scenario:

Question: "Should alcohol be served at all family gatherings?"
The "Safe" but "Wrong" Answer: "Yes, everyone should drink!" (Safe? Yes. Moral? Maybe. Culturally? No, because some families are sober or religious).
The "Moral" but "Wrong" Answer: "No, never drink!" (Safe? Yes. Culturally? No, because some cultures view wine as sacred).
The "Aligned" Answer: "It depends on the family's traditions and local laws." (Safe, Moral, and Culturally aware).

What They Discovered

The researchers tested many different AI models (some that were "fine-tuned" to be experts in just one area, and some that were general-purpose). Here is what happened:

The "Specialist" Trap:
Models that were trained only to be super-safe or only to be super-moral did great at their specific job. They caught almost every mistake in their lane (97% success!).
- The Catch: When you asked them to handle a situation requiring all three rules, they became clumsy. They started rejecting good answers just because they were being too strict about one rule. They failed to see the big picture.
- Analogy: Imagine a security guard who is so obsessed with "Safety" that they stop a grandmother from bringing her grandson into the park because the boy is wearing a hat that looks "suspicious." They followed the rule but failed the mission.
The "Generalist" Advantage:
The models that were trained to handle everything together (General-Purpose Aligned models) were better at balancing the three rules. They didn't catch every single tiny error, but they didn't make as many "false alarms." They understood the nuance.
The "Raw" Models:
The models that hadn't been trained on alignment at all were the most "chill." They rarely said "No" to good answers (low false alarms), but they also missed a lot of actual dangers because they didn't care enough about the rules.

The Big Takeaway

The paper concludes that being "aligned" isn't just about being safe, moral, or culturally aware in isolation. It's about knowing how to weave those three things together.

If you train an AI to be a "Safety Ninja," it might become so paranoid that it stops being helpful or culturally sensitive. To build a truly helpful AI, we need to test it on complex, real-world situations where it has to make a perfect balance, not just follow a single rulebook.

In short: You don't want a robot that is only safe, or only polite. You want a robot that knows when to be safe, when to be polite, and when to respect a cultural tradition, all while having a conversation with you. This paper gives us the first real way to test if our robots can actually do that.

1. Problem Statement

Current Large Language Models (LLMs) often exhibit misalignment when their behavior diverges from human expectations. While existing benchmarks evaluate misalignment along isolated dimensions—Safety (harmful outputs), Value (normative preferences), and Culture (context-sensitive norms)—they fail to capture the complexity of real-world interactions.

In real-world settings, a model must simultaneously satisfy all three dimensions in a single response. Existing benchmarks (e.g., INSECURE CODE, VALUEACTIONLENS, CULTURAL HERITAGE) treat these dimensions independently. Consequently, they miss joint-dimension failures, where a model might be safe but culturally insensitive, or culturally appropriate but value-violating. The paper argues that misalignment is inherently interactional, emerging from the breakdown of how these dimensions co-occur, rather than from isolated dimension errors.

2. Methodology: Mis-Align Bench

The authors introduce Mis-Align Bench, a unified benchmark designed to evaluate LLMs under jointly constrained conditions. The framework consists of two main stages:

Stage I: Construction of the SAVACU Dataset

The core of the benchmark is SAVACU, a curated English dataset containing 382,424 samples spanning 112 domains (14 Safety, 56 Value, 42 Cultural). The construction pipeline involves:

Unified Taxonomy: Prompts from the LLM-PROMPT-DATASET are reclassified using Mistral-7B-Instruct-v0.3 into the three dimensions based on taxonomies from BEAVERTAILS (Safety), VALUECOMPASS (Value), and UNESCO (Culture).
Domain Expansion: To address data sparsity in long-tail domains (those with <100 prompts), Llama-3.1-8B-Instruct is used for conditional query generation.
Deduplication: A SimHash-based fingerprinting technique (with Hamming distance threshold $\tau=10$ ) is applied to remove near-duplicates while preserving semantic diversity, ensuring leakage-safe dataset construction.
Response Generation & Filtering: For each prompt, candidate responses are generated by a pool of models (Gemma-3, Phi-3, Qwen-2.5, Mistral-8x22B). A two-stage rejection sampling process is employed:
1. Candidate Filtering: An automated evaluator (Llama-3.1-8B-Instruct) scores responses on Safety, Value, and Cultural dimensions (binary 0/1). A response is "Aligned" only if it scores 3/3; "Misaligned" if it violates at least one target dimension.
2. Feedback-Guided Resampling: If no valid pair is found, diagnostic feedback is generated and appended to the prompt to guide a second generation attempt (max 2 iterations).

Stage II: Benchmarking Framework

The benchmark evaluates three categories of models in a zero-shot setting:

General-Purpose Aligned Models: Models trained with multi-dimensional objectives (e.g., MARL-FOCAL, TRINITYX).
Dimension-Specific Fine-Tuned Models: Baselines fine-tuned only on Safety, Value, or Cultural datasets to test single-dimension optimization.
Open-Weight LLMs: Unaligned models (e.g., Gemma-7B, DeepSeek-7B) to assess baseline behavior.

Evaluation Metrics:

Coverage (Cov): The proportion of misaligned outputs correctly flagged.
False Failure Rate (FFR): The proportion of aligned outputs incorrectly flagged as misaligned (spurious rejection).
Alignment Score (AS): The harmonic mean of Coverage and $(1 - \text{FFR})$ .

3. Key Contributions

Mis-Align Bench: The first unified benchmark explicitly designed to evaluate the interaction of Safety, Value, and Cultural dimensions simultaneously.
SAVACU Dataset: A large-scale, high-quality dataset of 382k samples with paired aligned/misaligned responses across 112 domains, constructed with rigorous deduplication and quality control.
Empirical Evidence of Joint Misalignment: The study demonstrates that optimizing for a single dimension often degrades performance when multiple constraints are applied simultaneously.

4. Experimental Results

The evaluation reveals a critical trade-off between single-dimension sensitivity and joint-dimension robustness:

General-Purpose Models: Achieve the best balance, with high Alignment Scores (81.0%–81.6%) and low False Failure Rates (~18%). They effectively balance detection of misalignment without over-rejecting valid responses.
Dimension-Specific Models:
- Exhibit high Coverage on their target dimension (e.g., Safety-specific models reach 97.6% Coverage on safety tasks).
- However, they suffer from extremely high False Failure Rates (>50%) when evaluated under joint conditions.
- Their Alignment Scores drop significantly (63%–66%), indicating they are overly sensitive and reject valid responses that satisfy non-target dimensions.
Open-Weight Models: Show stable but low FFR (12%–20%) but low Coverage, reflecting conservative behavior that fails to detect subtle misalignments.
Cross-Domain Analysis: The performance gap widens in cross-domain interactions (e.g., Safety + Value). Single-dimension models fail to maintain the contextual flexibility required when multiple normative constraints interact.

5. Significance and Conclusion

The paper concludes that misalignment is interactional, not dimension-specific.

Key Insight: Models optimized for a single dimension (e.g., safety) often become "brittle," failing to satisfy value or cultural norms when those constraints are present. This leads to a high rate of spurious rejections (False Failures) in real-world scenarios where multiple norms must be satisfied simultaneously.
Implication: Current alignment strategies that treat safety, values, and culture as separate silos are insufficient. Future alignment research must focus on joint optimization to ensure models can navigate complex, multi-constraint environments without sacrificing legitimate expression or cultural nuance.
Limitations: The dataset is currently English-only, and the evaluation relies on automated LLM-based scoring, which may inherit residual biases.

In summary, Mis-Align Bench provides a critical tool for diagnosing the "blind spots" of aligned LLMs, proving that being "safe" or "value-aligned" in isolation does not guarantee a truly aligned model in the real world.