💬 NLP

How Language Models Process Out-of-Distribution Inputs: A Two-Pathway Framework

This paper reveals that existing white-box out-of-distribution (OOD) detection methods for language models are structurally confounded by sequence length, and proposes a two-pathway framework that distinguishes between embedding-based signals for vocabulary shifts and hidden-state trajectory features for detecting covert-intent inputs like jailbreaks.

Original authors: Hamidreza Saghir

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamidreza Saghir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a museum. Your job is to spot fake paintings (Out-of-Distribution or "OOD" inputs) among the real ones. Recently, other guards have been using a new tool: they measure how much the painting "shakes" or how chaotic the brushstrokes are (this is like measuring attention entropy in AI). They claim this tool is amazing at spotting fakes.

However, this paper argues that those guards are actually being tricked. They aren't spotting the fakes; they are just counting how many brushstrokes are on the canvas.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Long Canvas" Trap (The Length Confound)

The paper discovered that many popular AI safety tools are secretly just measuring how long the text is.

The Analogy: Imagine the security guard's tool is a ruler. It claims to measure "chaos," but it actually just measures "length."
The Problem: In the tests, the "fake" paintings (jailbreaks or spam) were often much longer than the "real" ones. The tool saw the long paintings, panicked, and said, "That's a fake!" simply because it was long.
The Proof: When the researchers forced the real and fake paintings to be the exact same length, the tool stopped working. It collapsed to near-random guessing (like flipping a coin). The paper shows that tools like CED, RAUQ, and even some confidence scores from deployed systems are structurally broken because they rely on attention mechanisms that naturally grow with text length.

2. The Two-Pathway Framework: "What" vs. "How"

Once the researchers removed the "length trick," they asked: How can we actually tell if the input is weird? They propose a two-part system, like a detective looking at a crime scene in two different ways:

Pathway A: The "What" (Embeddings)

The Analogy: This is like looking at the ingredients of a soup.
How it works: It checks the vocabulary. If the soup has "toxic" words or "spam" keywords, this pathway flags it.
When it works: It's great at spotting obvious fakes, like a soup that suddenly has "poison" written on the label.
When it fails: It fails if the fake soup uses the exact same ingredients as the real soup but is mixed in a weird way. For example, a "jailbreak" prompt often uses normal words but arranges them to trick the AI. The "ingredient checker" sees normal words and says, "All clear," missing the trick.

Pathway B: The "How" (Processing Trajectory)

The Analogy: This is like watching the chef cook the soup.
How it works: Instead of just looking at the final bowl, this pathway watches the chef's hands move through every step of the cooking process (layer by layer). It asks: "Is the chef chopping smoothly? Are they suddenly throwing ingredients in a panic? Is the pot shaking in a weird rhythm?"
Why it works: Even if the ingredients (words) look normal, a "jailbreak" prompt forces the AI chef to cook in a strange, unnatural rhythm. The "How" pathway catches this weird cooking style.
The Result: This pathway successfully spotted the tricky "jailbreak" fakes that the "ingredient" pathway missed, achieving a much higher accuracy score (0.850 vs. near-chance for the old methods).

3. The "Cross-Over" Discovery

The paper found that neither method is perfect for everything. It's like having two different types of metal detectors:

Detector 1 (Ingredients/Embeddings): Great at finding a gold coin hidden in a pile of sand (obvious vocabulary shifts).
Detector 2 (Cooking Style/Trajectory): Great at finding a plastic coin painted to look like gold (subtle structural tricks).
The Insight: You need both. When the "fake" is just a different topic, Detector 1 wins. When the "fake" is a clever trick using normal words, Detector 2 wins.

4. The "Why" (Mechanistic Evidence)

The researchers didn't just guess; they looked inside the AI's brain (the circuits) to see what was happening.

The Finding: When the AI encounters a "jailbreak" (a structural trick), it gets confused in a specific way: its "attention" (focus) circuits go haywire, scrambling its focus.
The Contrast: When the AI encounters "hate speech" (a vocabulary shift), it's the "memory" (MLP) parts of the brain that light up because it recognizes the bad words.
The Takeaway: Different types of "fakes" break the AI in different ways. The old tools only looked at the "focus" part and got confused by the length. The new "Trajectory" tool watches the whole cooking process, so it sees the breakage no matter where it happens.

Summary

The paper claims that many current AI safety tools are broken because they are fooled by text length. To fix this, we need to stop just looking at what the text says and start watching how the AI processes it. By watching the AI's "thought process" step-by-step (the trajectory), we can catch clever tricks that look normal on the surface but feel weird to the AI's internal machinery.

1. Problem Statement

The paper addresses a critical flaw in recent white-box Out-of-Distribution (OOD) detection methods for Large Language Models (LLMs). Methods such as CED, RAUQ, and WildGuard confidence scores, which rely on attention mechanisms and embedding distances, have reported strong performance on safety benchmarks. However, the author demonstrates that these results are structurally confounded by input sequence length ( $|r| \ge 0.61$ ).

The Confound: Attention-based metrics (e.g., attention entropy, CED, RAUQ) operate on a simplex whose size grows with input length $T$ . Consequently, these metrics inherit a structural dependence of $\Theta(\log T)$ .
The Consequence: When evaluated under length-matched conditions (where in-distribution and OOD samples have identical token counts), these methods collapse to near-chance performance (AUROC $\approx 0.50$ ). The paper argues that prior successes were likely detecting length differences rather than genuine OOD signals.

2. Methodology: The Two-Pathway Framework

To identify genuine OOD signals after deconfounding, the author proposes a Two-Pathway Framework organized along a Vocabulary-Transparency Spectrum. This framework distinguishes between what the text is about and how the model processes it.

Pathway A: The Embedding Pathway ( $X \to Z$ )

Mechanism: Uses static representations (embeddings) to capture the semantic content of the text.
Strengths: Effective for vocabulary-distinctive OOD (e.g., Spam, HateSpeech) where the OOD text uses words rarely seen in the training distribution.
Limitations: Fails on covert-intent OOD (e.g., Jailbreaks) where the vocabulary is identical to normal text but the structure or intent is adversarial. The paper shows standalone sentence-embedding models (MiniLM, BGE, etc.) fail completely on these tasks regardless of model capacity.

Pathway B: The Processing Trajectory Pathway ( $\{h_l\}_{l=0}^L$ )

Mechanism: Analyzes the sequence of hidden states across layers to capture the dynamics of processing.
Features: The author extracts 9 features across four categories:
1. Hidden-state dynamics: Cosine similarity between consecutive layers (smoothness of transition).
2. Representation geometry: Changes in the spread/variability of token representations across layers.
3. Component decomposition: The balance between Attention and MLP contributions at each layer.
4. Layer-specific probes: Early-layer disruption signals.
Theoretical Guarantee: Unlike attention weights, hidden states $h_l \in \mathbb{R}^d$ have fixed dimensionality $d$ regardless of sequence length $T$ . Therefore, trajectory features (specifically Categories A and G) are structurally length-independent.

3. Key Contributions

Identification of the Length Confound: The paper formally proves and empirically validates that attention-derived OOD scores are structurally dependent on sequence length. Under length-matched evaluation, they collapse to chance levels.
The Two-Pathway Framework: Proposes a unified theory explaining why different OOD types require different detection strategies:
- Vocabulary-distinctive OOD is best detected by Embedding Pathways.
- Structurally adversarial OOD (e.g., Jailbreaks) is best detected by Processing Trajectory Pathways.
Formal Length-Independence: Provides mathematical proofs (Propositions A.2, A.4) showing that trajectory features based on hidden states avoid the $\Theta(\log T)$ dependency inherent in attention statistics.
Mechanistic Evidence: Uses causal activation patching and circuit analysis to show that adversarial tasks (Jailbreaks) engage attention circuits significantly more than semantic tasks, while content-shift tasks engage MLP circuits.

4. Experimental Results

The framework was evaluated on Qwen3-0.6B (primary) and validated across 6 models (360M–7B) and 6 tasks (ToxicChat, Jailbreak, HateSpeech, Spam, AGNews, JailbreakClassification).

Performance Under Length Control:
- Attention-based methods (CED, RAUQ, Entropy): Collapsed to AUROC 0.491 – 0.527 (near chance).
- Trajectory Features (Maha-Traj): Maintained strong signal with an average AUROC of 0.721.
  - Jailbreak: 0.850
  - Spam: 0.848
  - HateSpeech: 0.605
  - ToxicChat: 0.580
The Crossover Effect:
- k-NN (Content/Embedding signal): Outperformed trajectory features on content-differentiable tasks (Spam, ToxicChat, AGNews).
- Trajectory (Processing signal): Outperformed k-NN on structurally adversarial tasks (Jailbreak, JailbreakClassification).
- Fusion: A fixed-weight fusion of both pathways achieved an average AUROC of 0.730, demonstrating complementarity.
Mechanistic Findings:
- Jailbreaks: Showed Attention Dominance (p < 0.001). Causal patching revealed sparse circuits where specific attention heads shift focus from "you/your" to directive/meta-linguistic tokens ("ignore", "prompt").
- HateSpeech: Showed MLP Dominance (p = 0.002), relying on distributed neuron activation rather than specific attention heads.
- Signal Construction: Analysis of Layer 0 vs. Last Layer showed that for Jailbreaks, the embedding signal at Layer 0 is near-chance (0.389), but processing constructs a strong signal (0.684) by the final layer.

5. Significance and Implications

Re-evaluation of Safety Benchmarks: The paper suggests that many published safety detection results based on attention scores are likely false positives driven by sequence length differences between benign and adversarial prompts.
New Standard for Evaluation: The authors advocate for length-matched evaluation as a mandatory control for any OOD detection method.
Architectural Insight: The work provides a mechanistic understanding of how LLMs process different types of anomalies. It suggests that detecting "jailbreaks" requires monitoring the process (trajectory/attention dynamics), while detecting "spam" or "hate speech" can rely on content (embeddings).
Practical Application: The proposed trajectory-based method (Maha-Traj) offers a robust, unsupervised, and length-invariant alternative for detecting adversarial inputs without requiring retraining or external classifiers.

In summary, this paper fundamentally shifts the paradigm of OOD detection in LLMs by exposing a pervasive length bias in attention-based methods and introducing a theoretically grounded, two-pathway framework that successfully disentangles content-based anomalies from structural adversarial attacks.