Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring

This paper proposes Representational Contrastive Scoring (RCS), a lightweight framework that leverages the internal geometric representations of Large Vision-Language Models to effectively distinguish malicious jailbreak attempts from benign inputs, thereby achieving state-of-the-art generalization and reducing the over-rejection issues common in existing anomaly detection methods.

The Gist

The Big Picture: The "Smart Guard" Problem

Imagine you have a very smart, helpful robot assistant (a Large Vision-Language Model, or LVLM) that can see pictures and read text. You want it to be helpful, but you don't want it to do bad things, like write a recipe for a bomb or generate hate speech.

Attackers are constantly trying to trick this robot into doing bad things. They use "jailbreaks"—clever tricks, weird images, or confusing riddles—to bypass the robot's safety rules.

The Problem:
Current security guards for these robots are either:

1. Too specific: They only know how to stop known tricks. If an attacker invents a new trick, the guard doesn't see it coming.
2. Too slow: They check the robot's work by asking a second, huge robot to review everything. This takes too much time and money.
3. Too paranoid: They are so scared of new things that they stop the robot from doing good things just because the request looks slightly different from what they've seen before.

The Solution: "Representational Contrastive Scoring" (RCS)

The authors propose a new way to catch these bad actors. Instead of looking at the words or the pictures themselves, they look at the robot's internal thoughts (its "brain waves" or hidden representations) while it is thinking.

Here is the core idea broken down into three simple steps:

1. Finding the "Sweet Spot" in the Brain

Imagine the robot's brain is a multi-story building with 30 floors.

• Floors 1–5: These are the "sensory" floors. They just see pixels and letters. They don't understand meaning yet.
• Floors 25–30: These are the "output" floors. They are just deciding which word to say next. They might have forgotten the safety rules by now.
• Floors 14–16 (The Sweet Spot): This is where the magic happens. The robot has understood the request, but it hasn't started speaking yet. This is where the robot "decides" if a request is safe or dangerous.

The authors found that if you check the robot's "thoughts" on these middle floors, you can clearly see the difference between a good request and a bad one. It's like checking a person's face before they speak to see if they are about to lie.

2. The "Good vs. Bad" Comparison (Contrastive Scoring)

Old security systems worked like a One-Way Mirror. They only knew what "Good" looked like. If something didn't look exactly like "Good," they assumed it was "Bad." This caused them to accidentally stop innocent people (false alarms).

The new system (RCS) works like a Tug-of-War.

• It has a team of "Good Examples" on one side.
• It has a team of "Bad Examples" on the other side.
• When a new request comes in, the system asks: "Is this new request pulling closer to the Good team or the Bad team?"

If it's closer to the Bad team, it's a jailbreak. If it's closer to the Good team, it's safe. This is much smarter because it understands that "Good" can look different in many ways (e.g., a medical question vs. a cooking question), but "Bad" is still "Bad."

3. The Two Detectives: MCD and KCD

The paper introduces two specific ways to measure this tug-of-war:

• MCD (The Statistician): This detective draws a smooth cloud around all the "Good" thoughts and a separate cloud around all the "Bad" thoughts. It calculates exactly how far the new request is from each cloud. If it's closer to the "Bad" cloud, it sounds the alarm.
• KCD (The Neighbor): This detective looks at the new request and asks, "Who are your 50 closest neighbors?" If most of your neighbors are "Bad," then you are probably "Bad" too. If your neighbors are "Good," you are safe.

Why This is a Game Changer

1. It's Fast: It doesn't need to wait for the robot to finish writing a long answer. It checks the robot's thoughts before the answer is generated. It's like catching a thief before they even pick the lock, rather than waiting for them to steal the jewelry.
2. It's Smart: It doesn't get confused by new types of tricks. Because it looks at the geometry of the thoughts (how they are arranged in space), it can spot a new kind of jailbreak even if it's never seen that specific trick before.
3. It's Fair: It stops "over-rejecting." It won't stop a doctor from asking a medical question just because the question looks slightly different from a cooking question. It knows the difference between "weird" and "dangerous."

The Analogy: The Airport Security Check

• Old Method (One-Class Detection): The security guard has a photo of a "safe" passenger. If you look even slightly different from that photo (maybe you're wearing a different hat or are from a different country), the guard stops you. This is annoying and stops innocent people.
• The New Method (RCS): The guard has a photo of a "safe" passenger AND a photo of a "dangerous" passenger. When you walk up, the guard compares you to both.
  • "You look a bit like the safe guy, but you also look a lot like the dangerous guy." -> Stop.
  • "You look like the safe guy, and nothing like the dangerous guy." -> Go.

Conclusion

This paper shows that we don't need to build a giant, slow, expensive super-robot to catch jailbreaks. Instead, we just need to look closely at the internal "thoughts" of the existing robot, find the specific moment where it decides to be safe or unsafe, and use a simple math trick to compare those thoughts to known good and bad examples.

It's a lighter, faster, and smarter way to keep our AI friends safe and helpful.

Technical Summary

1. Problem Statement

Large Vision-Language Models (LVLMs) are increasingly vulnerable to multimodal jailbreak attacks (e.g., adversarial images, cross-modal prompt injection). Existing defense mechanisms face a critical dichotomy:

• Specificity vs. Generalization: Alignment-based methods and input filters often overfit to known attack patterns, failing to generalize to novel threats.
• Efficiency vs. Accuracy: Detection frameworks relying on consistency checks, gradients, or multiple inferences (e.g., using "guard models") impose prohibitive computational latency, making them unsuitable for real-time deployment.
• The One-Class Limitation: Recent lightweight anomaly detection methods (e.g., JailDAM) treat jailbreak detection as an Out-of-Distribution (OOD) problem, training exclusively on benign data. The authors identify a fundamental flaw: these models confuse distribution shifts (unseen but benign inputs, e.g., medical images) with malicious intent, leading to high false positive rates (over-refusal) and unreliable performance in open-world settings.

2. Methodology: Representational Contrastive Scoring (RCS)

The authors propose RCS, a framework based on the insight that the most potent safety signals reside within the LVLM's own internal intermediate representations, rather than in general-purpose embeddings (like CLIP).

Core Components:

1. Principled Layer Selection:

   • Instead of arbitrarily selecting layers, the authors use a geometric analysis to identify "safety-critical" layers.
   • They utilize a dataset of semantically paired benign/malicious prompts (SGXSTest) to compute three metrics per layer: Maximum Margin Separation (SVM margin), Cluster Cohesion (Silhouette Score), and Discriminative Ratio (inter-class vs. intra-class distance).
   • Finding: Middle layers (e.g., layers 14–16 in LLaVA) consistently exhibit the highest geometric separability between benign and malicious inputs, acting as a "sweet spot" for detection.

2. Safety-Aware Projection:

   • Raw high-dimensional hidden states (e.g., 4096-dim) suffer from the curse of dimensionality and contain task-irrelevant noise.
   • The authors train a lightweight Multi-Layer Perceptron (MLP) to project features into a lower-dimensional space (256-dim).
   • Objective: The projection is optimized via a contrastive loss to simultaneously:
     • Cluster samples by their source dataset (preserving benign diversity).
     • Separate the centroids of benign and malicious distributions maximally.

3. Contrastive Scoring Mechanisms:
   The framework offers two instantiations that score inputs based on their relative proximity to benign vs. malicious clusters:

   • MCD (Mahalanobis Contrastive Detection): Parametrically models benign and malicious datasets as Gaussian distributions. It calculates the difference between the Mahalanobis distance to the nearest malicious cluster and the nearest benign cluster.
   • KCD (K-nearest Contrastive Detection): A non-parametric approach that computes the difference in Euclidean distance to the $k$-th nearest benign neighbor versus the $k$-th nearest malicious neighbor.

Decision Rule:

A simple threshold is applied to the contrastive score. If the score indicates the input is significantly closer to the malicious distribution than the benign one, it is flagged. This approach approximates the log-likelihood ratio ($\log \frac{p(x|malicious)}{p(x|benign)}$), which is theoretically optimal for binary hypothesis testing (Neyman-Pearson Lemma).

3. Key Contributions

• New Paradigm: Shifts from one-class anomaly detection (modeling only benign data) to contrastive detection (explicitly modeling both benign and malicious distributions), effectively distinguishing between "unseen benign" and "malicious."
• Layer Selection Strategy: Introduces a data-driven, geometric method to identify the specific internal layers of LVLMs that encode safety-relevant signals, outperforming ad-hoc layer selection.
• Lightweight & Efficient: The method operates on hidden states extracted during the forward pass (before token generation), requiring only a lightweight projection and distance calculation. It avoids the overhead of retraining the LVLM or running multiple inference passes.
• Robust Evaluation Protocol: Proposes a rigorous evaluation benchmark that strictly separates attack types between training and testing (e.g., training on "LLM Transfer" attacks, testing on "FigStep" attacks) and includes diverse benign sources to test generalization.

4. Experimental Results

The authors evaluated RCS on LLaVA, Qwen2.5-VL, and InternVL3 against state-of-the-art baselines (GradSafe, JailGuard, HiddenDetect, JailDAM).

• Performance:
  • MCD achieved a state-of-the-art 98.6% AUROC on LLaVA and 98.1% AUROC on Qwen.
  • KCD achieved superior F1 scores and significantly lower False Positive Rates (FPR) compared to baselines.
  • Both methods consistently outperformed the enhanced baseline JailDAM-RCS (which applies contrastive scoring to external embeddings), proving the superiority of using internal LVLM representations.
• Generalization: The methods maintained high performance when tested on unseen attack families (e.g., zero-shot transfer from text-only to multimodal attacks).
• Few-Shot Adaptation: The system demonstrated remarkable adaptability, requiring only 5–10 samples of a new attack type (e.g., SafeMTData) to rapidly adjust and maintain high detection accuracy, whereas zero-shot performance on new attacks was poor.
• Efficiency: The inference overhead is negligible (~4.0% to 5.5% relative to the host LVLM's forward pass), as it detects threats before the model generates a response.

5. Significance and Impact

• Practical Deployment: RCS offers a viable path for real-time safety in LVLMs by being lightweight, interpretable, and capable of early detection (pre-generation).
• Solving Over-Rejection: By explicitly modeling malicious distributions, the method solves the critical "over-refusal" problem where benign but out-of-distribution inputs are incorrectly blocked.
• Theoretical Grounding: The work bridges the gap between representation engineering and statistical hypothesis testing, demonstrating that simple statistical tests on internal representations can achieve complex safety goals without expensive retraining.
• Open Source: The authors provide code and a comprehensive benchmark, facilitating further research into multimodal safety.

In conclusion, the paper demonstrates that effective jailbreak detection does not require complex external models or heavy retraining. Instead, by leveraging the geometric structure of internal representations and applying a contrastive scoring mechanism, one can build a robust, efficient, and generalizable defense for Large Vision-Language Models.