🤖 machine learning

Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMs

This paper proposes an output-aware safety guardrail that predicts potential unsafe generations within a model's hidden state space to mitigate the over-refusal problem common in input-side filters, thereby preserving both safety and utility in multimodal large language models.

Original authors: Jiayi Li, Kun Zhan

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayi Li, Kun Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can see pictures and talk about them. This robot is great at answering questions, but sometimes people try to trick it into saying mean things, giving dangerous instructions, or spreading lies. To stop this, we usually put a "security guard" in front of the robot.

The Problem: The Over-Protective Bouncer
Right now, most security guards work like a strict bouncer at a club who only looks at your ID card (the question you ask) before letting you in. If your ID has a scary word on it, the bouncer slams the door shut immediately, even if you were just going to ask a harmless question.

For example, if you ask, "How do I kill a Python process?" (which is just about computer code), the bouncer sees the word "kill" and blocks you. If you ask, "Where can I shoot a good photo?" (which is about photography), the bouncer sees "shoot" and says "No way!" The robot never gets a chance to explain that you're being safe and helpful. The paper calls this over-refusal. It's like the guard is so scared of trouble that they accidentally kick out all the good customers.

The Old Way vs. The New Way
The researchers found that these old guards make their decisions based entirely on the input (what you ask). They don't wait to see what the robot actually says back.

The paper argues against this "input-only" approach. It suggests that the robot itself is actually pretty good at being safe. If you ask a tricky question, the robot often knows to give a warning or say "I can't do that" on its own. But the old guard stops the robot before it can even show its good side!

The Solution: OutGuard (The "Crystal Ball" Guard)
The authors propose a new system called OutGuard. Instead of just looking at your ID, OutGuard acts like a crystal ball that peeks into the robot's brain while it's thinking, but before it speaks.

Here's how it works:

The Peek: As the robot starts to form an answer, OutGuard looks at the "hidden thoughts" (called hidden states) swirling inside the robot's brain layers.
The Prediction: It tries to guess: "Is the robot about to say something truly dangerous, or is it just thinking about a dangerous topic but planning to say something safe?"
The Decision:
- If the robot is about to say something truly harmful (like how to make a bomb), OutGuard steps in and stops it.
- If the robot is about to say something safe (like explaining that "killing a process" is just coding), OutGuard lets it go!

The Magic Trick: Learning from "Bags" of Thoughts
To teach OutGuard how to do this, the researchers used a clever training method called Multi-Instance Contrastive Learning (MICL).

Think of the robot's answer as a bag of marbles. Some marbles are dangerous (the bad words), and most are safe. The old guards just look at the bag and say, "It has a dangerous marble, throw it away!"
OutGuard looks inside the bag. It uses a special attention mechanism to find the specific dangerous marbles. It learns to ignore the safe marbles and focus only on the ones that actually make the whole answer harmful. It's like a detective who knows that finding one bad apple doesn't mean the whole basket is rotten, unless that bad apple is the one that spoils the juice.

What the Numbers Say
The researchers tested this on two popular robot models: LLaVA 1.6 and Qwen 3.5. They compared OutGuard to other top security guards like HiddenDetector, QGuard, and LoD.

Safety: OutGuard caught almost all the truly bad requests. On the standard test set, it scored an AUPRC of 0.9725 for LLaVA and 0.9937 for Qwen (where 1.0 is perfect). This means it's just as good at catching bad guys as the others.
The "Over-Refusal" Fix: This is where OutGuard shines. The old guards blocked safe questions way too often.
- QGuard blocked 100% of safe questions (ARSP = 1.0000). It was too scared to let anyone in.
- OutGuard only blocked 5.5% of safe questions for LLaVA and 1.15% for Qwen.
- On tricky questions that look dangerous but are actually safe (like "How to kill time at the airport?"), OutGuard let 41.6% more safe questions through compared to the next best method.

Speed and Cost
You might worry that peeking into the robot's brain slows things down. The paper shows that OutGuard is very fast. It takes about 0.208 seconds per question on average, which is much faster than some other methods that take over 3 seconds. It also doesn't need much extra memory, adding only about 1.37 GB to the system.

The Bottom Line
The paper suggests that by shifting from "input-aware" (checking the question) to "output-aware" (checking the answer before it's finished), we can keep our robots safe without being annoying. OutGuard doesn't stop the robot from being helpful; it just stops it from being dangerous. It's a smarter, more precise way to keep the internet safe while still letting us ask our questions.

Technical Summary: OutGuard – Output-Aware Safety Guardrail for MLLMs

1. Problem Statement: The Over-Refusal Trade-off

Existing safety mechanisms for Multimodal Large Language Models (MLLMs) face a critical trade-off between safety and utility. While model fine-tuning offers robust safety, it often compromises general utility. Conversely, input-side safety guardrails provide a lightweight, non-intrusive alternative but suffer from severe over-refusal.

Current guardrails operate under an input-aware paradigm, making safety decisions based solely on the analysis of input image-text pairs. This approach fails to account for the model's intrinsic safety capabilities. As demonstrated in the paper, MLLMs often possess internal alignment mechanisms that can transform harmful inputs into safe outputs (e.g., refusal-style responses, admonitions, or warnings). However, input-aware guardrails indiscriminately block these interactions once the input is flagged as risky, regardless of whether the model would have generated a harmless response. This leads to the rejection of benign queries that merely appear risky (e.g., "How to kill a Python process?" or "Where can I shoot a good photo?"), significantly degrading user experience and limiting practical usability.

2. Methodology: The OutGuard Framework

To address this, the authors propose OutGuard, a paradigm shift from input-aware to output-aware safety guardrails. OutGuard operates within the model's hidden state space to predict the safety of the forthcoming generation before it is fully produced, rather than judging the input risk alone.

Core Components:

Hidden State Utilization: OutGuard extracts hidden state representations ( $H_l$ ) from the backbone LLM during the inference stage (Stage S4 in the data flow). These states link internal model dynamics to the final output.
Multi-Instance Contrastive Learning (MICL) Classifier:
- Problem Formulation: The detection task is framed as a Multiple Instance Learning (MIL) problem. A sequence of hidden states for a generated output is treated as a "bag," where individual token representations are "instances." The bag-level label (Safe/Unsafe) is known, but specific token-level harmfulness is latent.
- Architecture: The MICL classifier employs a gated attention mechanism to aggregate token-level representations into a bag-level representation ( $z_l$ ). This mechanism learns to focus on high-attention instances (likely harmful tokens) while suppressing noise.
- Training Objective: The classifier is trained using a composite loss function:
  1. Binary Cross-Entropy Loss ( $L_{cls}$ ): For bag-level classification.
  2. Entropy Regularization ( $L_{ent}$ ): To encourage sparsity in attention weights, forcing the model to focus on specific salient tokens.
  3. Intra-bag Contrastive Loss ( $L_{con}$ ): An InfoNCE-style loss that maximizes similarity among high-attention (positive) instances and minimizes similarity with low-attention (negative) instances within the same bag.
Layer Selection and Voting:
- A Safety-Score metric is used to evaluate the performance of MICL classifiers trained on different layers.
- Only classifiers from layers with a Safety-Score > 0.9 are selected.
- During inference, the selected classifiers vote on the safety of the hidden states. A majority vote determines the final decision (Safe vs. Unsafe).

Operational Workflow:
OutGuard runs as an independent module alongside the MLLM. It performs safety detection in a single pass during inference, analyzing hidden states as they are generated. If the system predicts an unsafe output, it intervenes; otherwise, the generation proceeds. This avoids the need for additional queries or significant latency overhead.

3. Key Contributions

Paradigm Shift: The paper identifies the root cause of over-refusal as the reliance on input-aware paradigms and proposes a shift toward output-aware safety evaluation.
OutGuard Framework: A novel framework utilizing hidden state representations and a Multi-Instance Contrastive Learning (MICL) classifier to directly assess output safety, distinguishing between inputs that lead to harmful outputs and those that the model neutralizes internally.
Empirical Validation: Extensive experiments demonstrate that OutGuard significantly reduces over-refusal rates while maintaining strong detection performance against harmful content and jailbreak attacks.

4. Experimental Results

The authors evaluated OutGuard on LLaVA 1.6 and Qwen 3.5 using datasets including GQA, AdvBench, SafeBench, MM-Vet, MM-SafetyBench, and XSTest. Baselines included HiddenDetector, QGuard, LoD, and ProGuard.

Safety vs. Utility Balance: OutGuard achieved superior performance across all metrics. On the test set, it maintained a low Unsafe Response rate to Unsafe Prompts (URUP < 0.1) while drastically reducing the Abstention Response rate to Safe Prompts (ARSP < 0.1). For instance, on LLaVA, OutGuard achieved an F1-Score of 0.9335 and an ARSP of 0.0749, outperforming the next best method (LoD-PP) which had an ARSP of 0.1124.
Over-Refusal Mitigation: On the XSTest dataset (benign queries that appear harmful), OutGuard reduced over-refusal rates significantly compared to baselines (e.g., 0.5840 vs. 0.7400 for LLaVA).
Robustness: OutGuard demonstrated strong robustness against jailbreak attacks (Visual-Adversarial-Examples, HADES, JOOD), achieving near-perfect classification metrics (AUPRC/AUROC > 0.97) while keeping over-refusal low.
Efficiency: The method introduces minimal latency. The average per-sample decision time was 0.208 seconds, comparable to or faster than many baselines (e.g., QGuard at 3.154s) and significantly faster than post-output methods.
Ablation Studies: Removing key components (gating module, contrastive learning, or sparsity regularization) led to severe degradation in detection capability, confirming the necessity of the MICL architecture.

5. Significance and Claims

The paper claims that OutGuard provides a practical solution to the safety-utility trade-off in MLLMs. By shifting the focus from input risk to actual output harm, the method preserves the model's built-in safety capabilities and general utility. The authors emphasize that their approach allows for precise intervention only when the model's actual response would be harmful, thereby avoiding the unnecessary blocking of benign interactions.

The work highlights the importance of output-aware safety mechanisms and demonstrates that lightweight classifiers trained on hidden states can effectively mitigate over-refusal without compromising the detection of genuine threats or requiring model retraining. The code is made available to facilitate further research in this direction.