🤖 AI

Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models

This paper proposes **CR-VLM**, a novel framework that utilizes activation steering to enable configurable refusal in Vision Language Models, allowing for adaptive safety alignment that mitigates both under-refusal and over-refusal through a refusal vector, a gating mechanism, and a counterfactual vision enhancement module.

Original authors: Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart digital assistant (a Vision Language Model) that can see photos and answer questions. Usually, these assistants are trained with a "one-size-fits-all" safety manual. If you ask something slightly controversial, it might shut down and say, "I can't answer that," even if the question is perfectly fine for you.

This is like a librarian who refuses to talk to anyone about "politics" because they think it's a dangerous topic—even if you just want to know who the current mayor is! This is called over-refusal, and it makes the assistant frustrating and unhelpful.

The paper "Steering to Say No" introduces a new system called CR-VLM. Here is how it works, explained through a few simple analogies.

1. The "Customized Filter" (Configurable Refusal)

Instead of one giant, rigid rulebook, CR-VLM acts like a customizable pair of sunglasses.

Standard Model: Everyone wears the same dark sunglasses. You can't see anything clearly, and everything looks "forbidden."
CR-VLM: You can adjust the tint. If you tell the assistant, "Don't talk to me about Bitcoin," it puts on "Bitcoin-tinted" glasses. It will still answer questions about biology or art perfectly, but the moment a Bitcoin question appears, the glasses turn pitch black, and the assistant says, "Sorry, I can't discuss that."

2. The "Teacher-Forced" Training (Getting the Signal Right)

How does the model learn exactly how to say "no" without getting confused? The researchers use a trick called Teacher-Forced Extraction.

The Analogy: Imagine teaching a child to say "No" to a stranger. Instead of just telling them "Say no," you act out a play. You pretend to be the stranger, and then you immediately whisper the correct response in their ear: "Say: 'No, thank you.'"
By "forcing" the model to see what a perfect refusal looks like during training, the researchers create a very clear "map" of what a refusal "feels" like inside the model's brain.

3. The "Gating Mechanism" (Preventing the Grumpy Librarian)

One big problem with making a model refuse things is that it often becomes "too safe" and starts refusing everything (the Grumpy Librarian problem). To fix this, they added a Gate.

The Analogy: Think of a security guard at a concert.
- If you have a ticket (an in-scope question), the guard sees you and steps aside to let you through.
- If you don't have a ticket (an out-of-scope question), the guard steps in front of the door.
The "Gate" ensures the model doesn't accidentally treat a math question like a forbidden question.

4. "Vision Enhancement" (Looking at the Evidence)

Sometimes, a model might refuse a question just because of the words used, even if the picture is totally innocent. The researchers added a module to make sure the model actually looks at the photo before deciding to refuse.

The Analogy: Imagine a security guard who refuses to let you into a building because you are wearing a "suspicious" hat, without even looking at your ID.
CR-VLM forces the guard to check the ID (the image). It ensures that if the model is going to say "No," it’s because it actually saw something in the image that violates your specific rules.

Summary: Why does this matter?

In short, this paper moves us away from "dumb" safety (where the model is just scared of everything) toward "smart" safety. It allows users to set their own boundaries, making AI assistants that are both highly respectful of your rules and incredibly helpful for everything else.

Technical Summary: Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models (CR-VLM)

1. Problem Statement

Current safety alignment in Vision Language Models (VLMs) typically relies on a "one-size-fits-all" refusal mechanism. This approach is rigid and fails to account for diverse user needs, local regulations, or specific contextual constraints. This leads to two primary failure modes:

Under-refusal: The model answers queries that should be declined based on specific user constraints (e.g., a user requesting the model to refuse all financial advice).
Over-refusal: The model becomes "too safe," declining legitimate, in-scope queries because the refusal signal is applied too broadly across the representation space.

Existing solutions like system prompt engineering are fragile and easily bypassed, while fine-tuning is computationally expensive and difficult to scale for personalized, evolving constraints.

2. Methodology: The CR-VLM Framework

The authors propose CR-VLM, a robust and efficient framework that uses activation steering to induce configurable refusal without modifying the backbone VLM's weights. The framework consists of three core technical innovations:

Teacher-Forced Activation Extraction: To overcome the issue of weak refusal signals in models lacking inherent safety alignment, the authors use a "teacher" mechanism. Instead of relying on system prompts (which may fail to trigger refusal), they append a predefined refusal string (e.g., "Sorry,") to out-of-scope samples during the extraction phase. This forces the model to compute refusal activations, allowing for the calculation of a clean, high-quality steering vector via the difference-in-means of these activations.
Selective Gating Mechanism (Over-refusal Mitigation): To prevent the model from applying the refusal shift to all inputs, the authors train a lightweight calibrated model. This model predicts how to adjust activations. It employs a gating mechanism and an orthogonality regularizer ( $L_{ortho}$ ). The regularizer ensures that the update direction for out-of-scope samples is mathematically decoupled from the direction for in-scope samples, preventing the "global shift" that causes over-refusal.
Counterfactual Vision Enhancement: To ensure the model doesn't ignore visual evidence in favor of textual steering, the authors introduce a vision-aware calibration objective. They calculate a "counterfactual" shift by comparing activations of in-constraint images vs. out-of-constraint images. This forces the learned steering to align with how visual representations should change when a refusal is visually warranted.

3. Key Contributions

First Systematic Study: It is the first work to investigate configurable (user-adaptive) refusal specifically within the multimodal context of VLMs.
End-to-End Lightweight Solution: Unlike hard-steering methods, CR-VLM provides a trainable, lightweight calibration module that is efficient and does not require full model fine-tuning.
Multimodal Alignment: It bridges the gap between textual steering and visual grounding, ensuring refusal is driven by both modalities.

4. Experimental Results

The authors evaluated CR-VLM on multiple VLMs (LLaVA-1.5, Idefics3, InstructBLIP) across two datasets (ScienceQA and MMMU).

High Refusal Accuracy (EQ1): CR-VLM significantly outperformed baselines (Prompt-based, Fine-tuning, and Persona) in refusing out-of-scope queries, often achieving refusal rates $>90\%$ .
Minimal Over-refusal (EQ2): While fine-tuning and persona-based steering often caused high over-refusal (rejecting valid questions), CR-VLM maintained very low over-refusal rates, effectively preserving the model's utility for in-scope queries.
Superior Trade-off (MB-Score): Using the MB-Score (a metric balancing refusal effectiveness and acceptance), CR-VLM consistently achieved the highest scores, proving it offers the best balance between safety and utility.
Ablation Success: Ablation studies confirmed that the orthogonality loss is critical for preventing over-refusal and that the vision loss is essential for maintaining visual grounding during the refusal process.

5. Significance

This work moves VLM safety from a static, global constraint to a dynamic, user-adaptive capability. By enabling "configurable refusal," the research paves the way for personalized AI agents that can respect individual user boundaries, professional ethics, or regional legal requirements without the prohibitive cost of continuous fine-tuning. It provides a scalable path toward more responsible and context-aware multimodal AI deployment.