💬 NLP

Segment-Level Coherence for Robust Harmful Intent Probing in LLMs

This paper introduces a segment-level coherence probing method that aggregates multiple evidence tokens to robustly detect harmful CBRN-related intents in LLMs, significantly reducing false positives and maintaining high detection accuracy even against adaptive jailbreaking and character-level obfuscation.

Original authors: Xuanli He, Bilgehan Sel, Faizan Ali, Jenny Bao, Hoagy Cunningham, Jerry Wei

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuanli He, Bilgehan Sel, Faizan Ali, Jenny Bao, Hoagy Cunningham, Jerry Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Over-Excited Security Guard"

Imagine a large Language Model (LLM) is a very helpful, but slightly naive, assistant. Your goal is to stop this assistant from ever writing instructions on how to build a biological weapon or a chemical bomb (CBRN threats).

To do this, you hire a Security Guard (the "probe") to watch the assistant's thoughts in real-time. If the guard sees anything dangerous, they hit the emergency stop button.

The Problem:
The old security guards were too easily tricked. They operated on a "keyword trigger" system.

Scenario: A doctor is writing a medical textbook about the Ebola virus to help cure patients.
The Old Guard's Reaction: They see the word "Ebola," get a massive spike of adrenaline, and immediately hit the emergency stop button.
The Result: The doctor's helpful work is blocked. This is a False Positive. The guard is so focused on the word that they miss the intent.

The Solution: "The Detective Who Reads the Whole Story"

The authors of this paper built a new kind of Security Guard called SC-TopK. Instead of screaming "Danger!" at the first scary word, this new guard uses a strategy called Segment-Level Coherence.

Think of it like this:

Old Guard (Keyword Seeking): "I see the word 'bomb.' STOP!" (Even if the sentence is "The movie plot involves a fake bomb.")
New Guard (Intent Aware): "I see the word 'bomb.' Okay, let's keep reading. Is the next sentence about 'movie scripts'? Is the next one about 'special effects'? Is the whole paragraph about entertainment? Ah, no danger here. Let's keep watching."

How the New Guard Works (The Two Tricks)

The paper introduces two specific techniques to make the guard smarter:

1. The "Top-K" Rule (The Jury System)

Instead of letting one single scary word decide the fate of the conversation, the new guard requires a jury of evidence.

Analogy: Imagine a courtroom. In the old system, if one witness shouted "Guilty!", the judge would convict immediately.
The New System: The judge (the AI) says, "I need at least K different pieces of evidence (like 3 or 4 distinct parts of the conversation) to all agree that this is dangerous before I convict."
Why it helps: If a user is just asking a biology question, they might use the word "toxin" once. But that's just one piece of evidence. The rest of the conversation is about "science class." The jury doesn't convict because the evidence isn't consistent.

2. SegVar (The "Smoothness" Check)

Sometimes, a bad actor might try to hide by scattering dangerous words far apart in a long, boring text.

Analogy: Imagine a criminal trying to sneak a weapon into a stadium by hiding it in three different bags, far apart from each other.
The Old Guard: Might miss the pattern because the bags are too far apart.
The New Guard (SegVar): Checks for consistency. It asks, "Is the danger level smooth and steady across this whole section, or is it just a random spike here and there?"
The Trick: If the text is benign (harmless), the guard forces the "danger score" to be smooth and low. If the text is truly harmful, the danger score should be high and consistent. This stops the guard from panicking over a single, isolated scary word.

The Results: Why This Matters

The paper tested this new guard against the old ones using three main scenarios:

The "False Alarm" Test: They fed the guard harmless conversations full of scary words (like a biology class discussing viruses).
- Result: The new guard made 35% fewer mistakes than the old guards. It stopped blocking helpful scientists.
The "Obfuscation" Test: They tried to trick the guard by writing the dangerous instructions in code (like a secret cipher or a different language).
- Result: Even when the text was scrambled, the new guard could still detect the intent with 98.85% accuracy. It's like a guard who can tell you are holding a gun even if you are wearing a disguise, because they see your posture (the internal logic), not just your clothes (the surface words).
The "Internal View" Test: They looked at where in the AI's brain the guard was looking.
- Result: They found that looking at the AI's "Attention" (what it's focusing on) or "MLP" (how it processes info) was much better than looking at its "Residual Stream" (a generic memory buffer). It's like checking a person's eyes and hands to see if they are nervous, rather than just checking their heartbeat.

The Bottom Line

This paper solves a major problem in AI safety: How do we stop bad guys without stopping good guys?

By moving from "Keyword Spotting" to "Intent Aggregation," the new method acts less like a trigger-happy guard and more like a seasoned detective. It waits for a pattern of behavior before sounding the alarm, ensuring that real threats are caught while harmless experts can keep doing their work.

1. Problem Statement

Large Language Models (LLMs) face increasing threats from adaptive jailbreaking, particularly in high-stakes domains like Chemical, Biological, Radiological, and Nuclear (CBRN) safety. A common mitigation strategy involves streaming probes that monitor the model's internal activations in real-time to detect harmful intent before generation occurs.

However, existing streaming probes suffer from a critical failure mode: shortcut learning.

The Issue: Current methods (e.g., Softmax or Hardmax aggregation) rely heavily on isolated, high-scoring tokens. In CBRN domains, benign expert discussions often contain the same sensitive terminology (e.g., "Ebola," "toxin," "synthesis") as malicious requests.
The Consequence: Probes trigger false positives when these keywords appear in safe contexts because they prioritize immediate lexical spikes over the broader semantic intent of the conversation.
The Gap: Existing training objectives do not sufficiently enforce segment-level coherence, leading to detectors that are overly sensitive to surface-level cues rather than distributed evidence of harmful intent.

2. Methodology

The authors propose SC-TopK, a new streaming probing framework designed to enforce distributed evidence accumulation. The method introduces two key modifications to the standard sliding-window aggregation framework:

A. Top-K Supportive-Window Pooling

Instead of weighting all tokens or windows by confidence (which allows a single high-scoring token to dominate), this method requires multiple windows to jointly support a prediction.

Mechanism: The model aggregates the logits of the top $K$ windows with the highest scores and averages them.
Effect: A harmful prediction is only made if multiple distinct segments of the conversation exhibit elevated risk scores. This reduces sensitivity to isolated keyword spikes and forces the probe to learn intent-level patterns.

B. Segment Variance Regularization (SegVar)

To prevent the "Top-K" windows from being widely scattered (which could still lead to fragmented, unstable predictions), the authors introduce a regularization term.

Mechanism: SegVar penalizes the variance of logit magnitudes across the selected evidence windows. It encourages the probe to find contiguous, locally consistent spans of evidence rather than disjointed spikes.
Selective Application: Crucially, SegVar is applied only to benign (negative) examples.
- Reasoning: Benign conversations containing sensitive terms should not trigger high variance; they should remain low and consistent.
- Harmful examples: Malicious intent may legitimately be distributed across disjoint regions (e.g., an initial request followed by elaboration). Applying variance regularization here would suppress valid signals, so it is excluded for positive samples.

C. Architecture and Features

Input: The method probes internal activations (Attention, MLP, or Residual stream) at each token position.
Aggregation: Uses a sliding window mean (SWiM) of size $M=16$ to smooth token-level noise before applying the Top-K and SegVar objectives.
Inference: Uses Exponential Moving Average (EMA) smoothing to aggregate evidence in a streaming setting.

3. Key Contributions

Novel Objective Function: Introduction of SC-TopK, which combines Top-K supportive window pooling and Segment Variance Regularization to shift detection from keyword-matching to intent-level evidence aggregation.
Feature Discovery: Systematic analysis revealing that Attention and MLP activations consistently outperform the standard Residual Stream for safety probing, with MLPs showing particular dominance in chemical domains.
Robustness to Obfuscation: Demonstration that probes trained on base models can be applied "plug-and-play" to models fine-tuned with character-level ciphers (e.g., ASCII conversion, Polybius squares). Once the model learns to decode the cipher, the probe detects the underlying harmful intent without retraining.
Efficiency: The method uses a single linear layer (approx. 262K parameters), offering a massive reduction in computational overhead compared to fine-tuned classifier baselines (approx. 90M parameters) while matching or exceeding their performance.

4. Experimental Results

The method was evaluated on Llama-3.1, Qwen3, and Gemma2 across Bio, Chemical, and Generic high-stakes domains.

Performance Gains:
- At a fixed 1% False Positive Rate (FPR), SC-TopK improved the True Positive Rate (TPR) by 35.55% relative to strong streaming baselines (SWiM).
- It achieved a 16.80% increase in Log-space AUROC, a metric critical for real-world deployment where low false positives are paramount.
- Even starting from a near-saturated baseline (AUROC = 97.40%), the method reduced the remaining error margin by 31.2%.
Domain Specifics:
- Bio-Domain: SC-TopK significantly reduced false positives on benign prompts containing biological terminology (e.g., "Neo-Ebola" in a sci-fi context).
- Chemical Domain: Outperformed baselines by 4.0% to 7.9% in log-space AUROC across different model architectures.
Obfuscation Robustness:
- When applied to models fine-tuned on five different character-level ciphers, SC-TopK achieved an AUROC of 98.85% without any probe retraining, demonstrating that safety signals persist through surface-level transformations.
Ablation Studies:
- Removing SegVar or using Softmax aggregation resulted in significant performance drops, confirming the necessity of both distributed evidence and variance regularization.
- Applying SegVar to all instances (including harmful ones) degraded performance, validating the strategy of applying it only to benign data.

5. Significance

This paper addresses a fundamental limitation in current LLM safety monitoring: the reliance on isolated tokens. By enforcing segment-level coherence, SC-TopK provides a more robust, deployment-ready solution for real-time safety monitoring.

Practical Impact: It enables safety layers that do not disrupt legitimate expert discourse in high-stakes fields (CBRN), a major hurdle for current systems.
Scalability: The lightweight nature of the probe makes it feasible to deploy on large-scale models without the latency and cost of auxiliary classifier models.
Future-Proofing: The ability to detect harmful intent even in obfuscated, cipher-based attacks suggests that internal representations of safety are deeper than surface-level text, offering a promising direction for defending against adaptive jailbreaks.

In conclusion, the authors demonstrate that shifting the inductive bias from "max-pooling" (finding the most dangerous word) to "distributed evidence" (finding consistent patterns of intent) is essential for the next generation of robust LLM safety systems.