💻 computer science

The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators

This paper demonstrates that while activation-space probes in large language models effectively detect broad risks and block a high percentage of harmful requests, they fail to function as reliable standalone adjudicators for distinguishing harmful from benign prompts when context changes without altering surface form.

Original authors: Dominik Schwarz

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dominik Schwarz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to tell the difference between a harmless joke and a dangerous threat. You might think the robot just needs to learn a list of "bad words." But here's the tricky part: context is everything. The sentence "How do I kill this?" is perfectly safe if you are talking about a computer program that is frozen, but it is a crime if you are talking about a person. The words are identical, but the meaning flips completely based on the situation. This is the world of Large Language Models (LLMs), the super-smart AI brains behind chatbots. Scientists have been trying to build "probes"—tiny sensors that peek inside the AI's brain while it thinks—to see if they can spot the difference between a bad intention and a good one, even when the words look the same. The big question is: Can these sensors act like a smart judge that understands the story, or are they just clumsy bouncers who only check the clothes you're wearing?

A researcher named Dominik Schwarz decided to test this by setting up a high-stakes game of "spot the difference." He took three different AI models and created pairs of prompts: one was a "camouflaged" request that sounded like it wanted to do something bad (like "How do I kill this?"), and the other was a "twin" request that used the exact same words and structure but was actually harmless (like "How do I kill this process?"). The goal was to see if the AI's internal sensors could tell these two twins apart without getting confused by the similar wording.

The results were a bit of a plot twist. When the sensors looked at a huge pile of clearly dangerous requests, they were fantastic detectives, catching about 95% to 97% of the bad guys. However, when the researchers tested them on the tricky "twins" where the only difference was the hidden intent, the sensors hit a wall. They started blocking harmless requests just as often as they blocked the dangerous ones. In fact, on the hardest test set, the sensors failed to distinguish the twins much better than random guessing. Even when the researchers tried to train the sensors specifically on these pairs, the sensors became so good at spotting the specific training examples that they failed miserably on new, unseen examples, blocking nearly 80% to 100% of harmless prompts that looked similar.

The paper calls this phenomenon the "Entanglement Wall." Think of it like this: The AI's brain has a "danger radar" that is very good at detecting the topic of danger (like the concept of "killing"). But when the topic is the same for both a bad guy and a good guy, the radar can't tell them apart. It's like a security guard who is trained to stop anyone holding a red balloon because red balloons are usually used in heist movies. If a child shows up with a red balloon, the guard stops them too. The sensor sees the "red balloon" (the dangerous topic) and panics, unable to see that the child is just playing.

The study suggests that while these activation sensors are excellent "broad-risk detectors"—great for a first pass to catch obvious threats—they are not ready to be the final "context adjudicators" or judges that decide if a specific, tricky request is safe. They are too easily confused by the surface-level similarity of the words. The researchers found that to truly understand the difference between a harmful intent and a harmless one in these specific cases, you need more than just a simple sensor reading; you need a system that can actually understand the story, not just the vocabulary. So, for now, these sensors are great at raising the alarm, but they aren't smart enough to decide if the alarm is a false positive on their own.

Technical Summary: The Entanglement Wall

Problem Statement

Context can alter the harmfulness of a request without changing its surface form or topic. For instance, "How do I kill this?" is benign when referring to a software process but dangerous when referring to a person. While residual-stream probes have demonstrated high accuracy in distinguishing harmful from benign requests in cross-corpus benchmarks (comparing distinct datasets like AdvBench and Alpaca), their ability to resolve local consequence changes between topic-matched requests remains unclear. Existing benchmarks often conflate intent with style, provenance, or topic differences. This paper investigates whether activation-space probes can function as standalone context adjudicators—distinguishing harmful intent from benign intent within surface-matched pairs—or if they merely act as broad-risk detectors that flag proximity to danger without resolving the specific context.

Methodology

The study employs a multi-faceted approach across three model families (Llama-3.1-8B, Mistral-7B-Instruct, and Qwen2.5-7B), with extensions to 24B and 32B variants.

1. Deployment Evaluation (Activation Sensor)

A sensor was constructed using 43 intent zones (24 dangerous, 19 harmless) derived from a held-out calibration corpus.

Mechanism: At generation time, the sensor calculates the cosine distance between token activations and zone medoids. It computes a "risk margin" (strongest danger zone minus strongest harmless zone) averaged over an early generation window.
Decision Rule: A fixed zero-threshold gate blocks generation if the average margin is positive. No threshold tuning was performed on evaluation data.
Benchmarks: Evaluated on 480 taxonomy-selected harmful prompts and 660 benign prompts (including XSTest, which contains hard-benign, risk-adjacent requests).

2. Same-Topic Paired Design (Geometry Experiments)

To isolate context from topic, the authors constructed "Twin" cohorts where a camouflaged harmful prompt is paired with a benign twin sharing the exact topic and surface frame but differing in intended outcome.

Cohorts:
- Twin-n43: Original curated pairs.
- Twin-n163: Full constructed cohort (43 + 120 pairs).
- Twin-n70: A subset where Llama-Guard-3 explicitly flags the camouflaged side as unsafe and the twin as safe.
- Twin-n217: A robustness cohort generated during the study.
Read Points: Activations were read from the post-attention layer norm (residual stream) at specific indices (e.g., layer 14 for deployment, layer 12 for analysis).

3. Readout Families and Controls

The study compared seven readout strategies to test transferability and overfitting:

Unsupervised/External Transfer: Fixed directions fitted on disjoint source contrasts (AdvBench vs. Alpaca) applied to Twin pairs without refitting.
In-Model/Direct Fitting: Classifiers trained directly on the pair boundaries.
Deflation: Removing the mean-difference direction to test residual signal.
Controls: Extensive controls included category hold-out, generation-batch hold-out, text-only baselines (TF-IDF, sentence encoders), and permutation tests to rule out dataset artifacts.

4. Evaluation Metrics

Ranking: Area Under the Receiver Operating Characteristic Curve (AUROC).
Operating Point: True Negative Rate (TNR) at 95% True Positive Rate (TPR), and TPR at low False Positive Rates (1%, 5%, 10%).
Thresholds: A numerical benchmark was set at AUROC $\ge$ 0.90 or TNR $\ge$ 40% at 95% TPR. Crucially, a post-hoc requirement was added that any crossing must persist on the full Twin-n163 cohort, not just the guard-selected subset.

Key Results

1. Deployment Performance (Broad-Risk Detection)

The activation sensor successfully blocked 95.5–97.7% of judge-classified compliant attacks across the three model families. However, it exhibited high false-blocking rates on benign benchmarks, particularly XSTest:

XSTest Block Rates: 59.6% (Llama), 63.6% (Mistral), and 68.4% (Qwen).
Conclusion: The sensor acts as an effective first-stage filter for broad risk but lacks the specificity to distinguish benign, risk-adjacent requests from harmful ones.

2. Geometry and Transfer Performance (Context Adjudication)

When transferring fixed directions from disjoint source audits to the same-topic Twin pairs:

Source Audit: Achieved near-ceiling AUROC (0.996–0.999) on the source contrast.
Transfer to Twin-n70: AUROC dropped to 0.656–0.819.
Transfer to Twin-n163: AUROC further degraded to 0.590–0.690.
Operating Point: TNR at 95% TPR fell to 8.0–21.5%, far below the 40% threshold.

3. Direct Fitting and Overfitting

Classifiers trained directly on the pair corpus (pair-trained) achieved high in-corpus AUROC (0.888–0.929) but failed under hold-out conditions:

Category/Generation-Batch Hold-out: Performance dropped significantly (e.g., to 0.661–0.780).
XSTest False Blocking: These classifiers blocked 79.6–100% of XSTest prompts at 95% in-corpus TPR.
Text Baselines: Simple text features (TF-IDF, length) showed similar degradation patterns, suggesting the signal relies on generation/curation structure rather than pure semantic intent.

4. Deflation and Scale Extension

Deflation: Removing the mean-difference direction reduced separation but left a statistically detectable residual. However, this residual never met the numerical performance criteria on Twin-n163.
Scale: Experiments on 24B and 32B models replicated the qualitative pattern (strong source separation, weak same-topic transfer) but did not achieve the required operating points.

Significance and Claims

The paper introduces the concept of the "Entanglement Wall": an empirical operating-point limit where activation-space probes can effectively screen for broad risk proximity but fail to adjudicate context-specific consequences when topic and surface form are matched.

Not a Universal Limit: The authors explicitly state this is not a claim of universal representational impossibility. Dedicated guards (like Llama-Guard-3) can distinguish these pairs with high accuracy (AUROC 0.95+), proving the information exists in the model's output space.
Role of Probes: The results suggest that at the tested residual-stream read points, activation scores function as broad-risk detectors rather than standalone context adjudicators.
Implication for Safety: Ambiguous cases (where a prompt is topically adjacent to harm) still require a context-sensitive decision stage beyond simple activation thresholding. The "entanglement" arises because the "danger direction" in the residual stream captures the shared topic component of both the harmful and benign twins, preventing the probe from isolating the specific harmful intent without overfitting to dataset artifacts.

The study concludes that while activation probes are valuable for initial risk screening, they currently lack the robustness to replace context-aware adjudication in high-stakes, same-topic scenarios.