🤖 AI

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

This paper reveals that current compliance detectors suffer from "rule blindness," failing to genuinely rely on stated regulatory rules rather than surface features, and demonstrates that even the proposed training-free Internal Compliance Score (ICS) lacks robustness against trivial baselines and adaptive attacks.

Original authors: Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital world, large language models act as powerful engines for generating text, from drafting emails to answering complex questions. However, when these tools are used in regulated fields like finance, healthcare, or law, they must follow strict written rules. A bank, for instance, might have a policy that personal data must be deleted within ninety days. To ensure the AI follows this, companies often deploy "guard models"—specialized classifiers trained to scan the AI's output and flag any violations. The hope is that these guards read the specific rule provided and check if the AI's answer matches it. If the guard is working correctly, it should only flag a violation when the specific rule is broken, not just because the topic sounds risky.

A team of researchers from Lexsi Labs set out to test whether these compliance monitors actually understand the rules they are supposed to enforce. They investigated a phenomenon they call "rule blindness." This occurs when a detector gives the same verdict regardless of the rule it is supposed to be checking. Imagine a security guard at a museum who stops every visitor carrying a red bag, not because red bags are forbidden, but because the guard has simply learned to associate the color red with danger. If the museum changes the rule to ban blue bags, a rule-blind guard would still stop the red bags and ignore the blue ones, completely missing the new instruction. The researchers wanted to know if the digital guards protecting our data were suffering from the same issue.

The team began by testing a wide variety of existing guard models and activation probes—tools that peek inside the AI's internal workings to guess its intentions. They used a method of counterfactual testing, which involves changing the input in specific ways to see how the system reacts. In their experiments, they took a scenario where data was kept for four hundred days, which clearly violates a ninety-day rule. They then swapped that ninety-day rule for a completely different, unrelated rule, or even removed the rule entirely. Surprisingly, the guard models flagged the violation just as often, and with the same confidence, even when the governing rule was gone or changed. The detectors were tracking the scenario—the fact that data was kept too long—rather than the specific rule that made it a violation. This failure was not limited to one type of model; it appeared in fixed-safety classifiers that never see the rule at all, and even in sophisticated "policy-conditioned" guards that are explicitly designed to read and follow custom rules. One such guard correctly cited the position of the rule in the text ninety-one to ninety-five percent of the time, yet its final verdict barely changed when the rule was swapped for a permissive one that allowed the behavior.

To understand what these systems were actually reading, the researchers built a new, purpose-made benchmark. They created a test where the answer depended entirely on the interaction between a specific rule and a specific scenario, ensuring that neither the rule text nor the scenario text alone could predict the outcome. For example, a rule might say a bank can simplify checks for accounts under one thousand dollars, while another rule sets the limit at five thousand. If a scenario describes an account with two thousand dollars, the verdict depends entirely on which rule is active. When they ran their tests on this unconfounded benchmark, every efficient detector they tried, including their own new tool, performed no better than random chance. The only system that succeeded was a large language model allowed to reason step-by-step, breaking down the problem before giving an answer. This suggested that the fast, automated detectors were not composing the rule with the scenario; they were simply recognizing a broad signal of "violation" based on the surface features of the text.

The researchers then introduced a new, training-free tool called the Internal Compliance Score. This method reads the AI's internal activation signals directly, requiring only ten pairs of examples to calibrate. It is incredibly cheap and fast, needing no retraining and no extra processing steps. They found that this tool was excellent at ranking candidate responses, helping to pick the most compliant answer from a list of options. In tests, it improved the pass rate for instruction-following tasks by five percentage points. However, the same rule blindness that plagued the other guards also affected this new score. When the rule was deleted or swapped, the score remained unchanged. The tool was effectively measuring a general sense of risk rather than a specific adherence to the written rule. Furthermore, the researchers discovered that this advantage was fragile; a clever, adaptive attack could easily fool the system by adding a few specific words to the end of a response, causing the score to drop to random levels.

The study also revealed a deeper issue with how compliance is currently measured. The researchers audited seven public benchmarks used to test these systems and found that four of them were "lexically degenerate." This means the labels for whether a response was compliant or not were baked into the text itself, often through the way the scenario was written or the verdict was narrated. A simple computer program that just counted words, without understanding any rules, could solve these benchmarks with high accuracy. This implies that many previous claims about a model's ability to follow rules were actually just claims about its ability to recognize specific words or sentence structures. The researchers concluded that while these detectors are useful for sorting and ranking responses, they cannot be trusted to make legal or audit decisions based on specific rules. They offer the appearance of rule-specific assurance, but in reality, they are blind to the rules they are meant to enforce. The paper ends by releasing the tools and protocols used for these tests, allowing others to check future systems for this same blindness before they are deployed in the real world.

Technical Summary: What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

Problem Statement

The deployment of language models in regulated sectors (data protection, healthcare, finance) increasingly relies on "guard models" and activation probes to monitor outputs against written regulatory rules. The fundamental assumption is that these detectors verify compliance by reasoning about the relationship between a specific scenario and a governing rule.

The paper identifies a critical failure mode termed rule blindness: the detection verdict depends on surface features of the scenario (e.g., the presence of a violation signal) rather than the specific rule provided. The authors demonstrate that deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard model and activation probe tested. This implies that current compliance monitoring offers only the appearance of rule-specific assurance, as the detectors cannot distinguish a scenario from the rule that governs it.

Methodology

The Internal Compliance Score (ICS)

To audit these systems at scale without the computational cost of retraining, the authors introduce the Internal Compliance Score (ICS).

Mechanism: ICS is a training-free activation readout. It computes a "difference-of-means" direction ( $\hat{d}_\ell$ ) in the residual stream of a monitored model using a small calibration set (typically 10 labeled pairs of adherent and violating texts).
Scoring: For a new case, the score is a single dot product between the final token's layer- $\ell$ activation and the calibrated direction: $ICS_\ell(x) = \langle a_\ell(x), \hat{d}_\ell \rangle$ .
Efficiency: It requires no gradient steps, no additional forward passes (if activations are retained), and minimal labeled data (10 pairs per domain). It is designed to be re-calibrated instantly when the monitored model is fine-tuned.

Evaluation Framework

The paper employs a rigorous evaluation protocol to distinguish genuine rule-conditioned reasoning from surface-level shortcuts:

Outcome Ablation: Removing the explicit "verdict" text from benchmark inputs to ensure detectors are not simply reading the answer.
Lexical Floors: Establishing baselines using policy-blind bag-of-words models (TF-IDF) to determine if a benchmark is "degenerate" (solvable without reading the rule).
Counterfactual Interventions:
- Rule Ablation: Deleting, shuffling, or swapping the rule text to test if the detector's verdict changes.
- Crossed Benchmark: A purpose-built benchmark where two rules and two scenarios are crossed such that neither the rule nor the scenario alone predicts the label. This removes confounds like rule polarity (permissive vs. prohibitive).
Budget-Matched Nulls: Comparing detectors against random directions selected via the same layer-search process to account for selection bias.

Key Contributions

Method (ICS): A training-free, first-party activation readout that scores compliance risk directly from the monitored model's activations. It is calibrated from a small set of pairs and scored via a single projection.
Evaluation (Audit of Benchmarks): The authors audit 13 public compliance benchmarks. They find that four of seven are "lexically degenerate," meaning a simple bag-of-words model can solve them without reading the rule, rendering them incapable of adjudicating rule-conditioned claims.
Characterisation (Rule Blindness): Through counterfactual tests, the paper proves that ICS and all tested detectors (including policy-conditioned guards like LPG and Llama Guard 3) are rule-blind.
- Deleting the rule changes accuracy by $<0.003$ .
- Swapping a violated rule for its permissive counterpart changes the verdict of a policy-conditioned guard in only ~7% of cases, despite the guard correctly citing the rule's position 91–95% of the time.
- On the crossed benchmark, all efficient detectors (probes and guards) perform at chance (AUROC ~0.50). Only a model allowed to perform step-by-step reasoning (Chain-of-Thought) solves the task.
Application (Candidate Selection): While ICS cannot adjudicate rules, it is useful for ranking. ICS-guided selection of candidate responses improves mechanically verified pass rates on IFEval by +5.2 percentage points and regulatory advisory compliance by +11.5 points over random selection. However, this gain is non-robust to adaptive white-box attacks.

Key Results

Rule Blindness is Universal: The failure to read the rule-scenario relation is not an artifact of a specific model or benchmark. It affects:
- Fixed-taxonomy guards (e.g., Llama Guard 3) which never see the rule.
- Policy-conditioned guards (e.g., LPG) which explicitly receive the rule via a custom channel but still fail to condition their verdict on it.
- Activation probes (including ICS and SIREN).
Generalization Limits: ICS generalizes well within a task family (e.g., from one compliance domain to another) but collapses to chance when transferring between families (e.g., from compliance to safety data).
Cost vs. Accuracy: ICS achieves competitive in-domain AUROC (0.952) with zero trained parameters and negligible inference cost. However, its advantage over a bag-of-words model vanishes when both are evaluated against their respective budget-matched nulls, indicating the signal is largely structural or lexical rather than deep semantic reasoning.
Adversarial Vulnerability: The selection gain provided by ICS is removed by an adaptive white-box attack (GCG) that optimizes a suffix against the ICS direction.

Significance and Claims

The paper claims that current compliance monitoring systems, including both deployed guard models and activation probes, do not read the rules they are supposed to enforce. Instead, they detect a broad "violation signal" or "compliance register" that is independent of the specific governing clause.

Modest Claims on Accuracy: The authors do not claim ICS is a superior adjudicator. They explicitly state that ICS is a "broad compliance-risk direction" rather than a "rule-specific adjudication direction." Its value lies in its cost-efficiency (training-free, re-fittable) and its ability to rank candidates, not in its ability to make final compliance judgments.
Audit Value: The primary contribution is the counterfactual protocol (including the crossed benchmark) which provides a necessary test for future claims of rule-conditioned reasoning. The paper demonstrates that without such tests, high accuracy on standard benchmarks is likely an artifact of dataset degeneracy.
Practical Utility: ICS is presented as a useful tool for triage and candidate ranking in non-adversarial settings, where it can recover a significant fraction of the "oracle headroom" (the gap between random selection and the best possible selection). However, it is not robust enough for standalone adversarial enforcement.

In summary, the paper argues that the field of compliance monitoring has overestimated the reasoning capabilities of current detectors. The "rule blindness" phenomenon suggests that detectors are tracking the presence of a violation scenario rather than the logic of the rule, necessitating a shift toward more rigorous, counterfactual evaluation standards.