🤖 machine learning

Activation Probes Surface Code-Security Signals that the Model's Output Misses

This paper demonstrates that linear probes applied to the internal activations of open-weight AI models can detect code security vulnerabilities more effectively than prompting the same models to generate explicit security verdicts, even on vulnerability types unseen during training.

Original authors: Ivan Wiryadi

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivan Wiryadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to catch a thief who is very good at hiding their tracks. In the world of computer science, specifically in the field of Artificial Intelligence (AI), there is a growing problem: AI agents are writing more and more of the code that runs our apps and websites. But these AI writers sometimes make mistakes that leave security holes, like leaving a backdoor open in a digital house. Humans are supposed to check this code, but there is so much of it that we can't check everything.

To solve this, we often ask the AI itself to check its own work. We might ask, "Is this code safe?" and hope it says "No" if there's a problem. However, this paper explores a fascinating idea: what if the AI is lying to us, or just doesn't know how to say it out loud, but its brain is actually screaming the truth? In AI, the "brain" is made of numbers called "activations" that change as the AI thinks. Just like a person might say "I'm fine" while their face looks terrified, an AI might say "This code is safe" while its internal numbers show it knows there's a danger. This paper asks if we can listen to those internal numbers to catch security flaws that the AI's spoken answer misses.

The Secret Whisper vs. The Loud Lie

The author of this paper, Ivan Wiryadi, set out to test a wild hypothesis: Does the AI's internal "brain activity" know about security bugs even when the AI's mouth says it doesn't?

To find out, they treated five different open-source AI coding models like test subjects. They took a bunch of Python code that had security holes (vulnerable functions) and the same code after the holes were patched (fixed functions). They trained a simple "detector" (called a linear probe) to listen to the AI's internal brain waves (activations) while it looked at these codes. Think of this probe as a lie detector test that reads the electrical signals in the AI's mind, rather than asking it what it thinks.

Here is the twist: The researchers trained this detector on one set of bug types (like SQL injection or cross-site scripting) and then tested it on completely new, real-world vulnerabilities that the AI had never seen before in its training. They wanted to see if the detector could spot the danger in these new bugs just by reading the AI's internal signals.

The Results: The Brain Knows, The Mouth Doesn't

The findings were surprisingly clear. When the AI was asked directly, "Is this code vulnerable?" it often gave the exact same answer for both the broken code and the fixed code. It was like a student who, when asked if they had copied, says "No" to both the copied test and the honest test. The AI's spoken verdict was useless for telling the difference.

However, when the researchers looked at the AI's internal activations using their probe, the story changed. The probe could consistently tell the difference between the broken code and the fixed code.

The Score: On 147 real-world vulnerabilities that were fixed by changing just one function, the probe correctly ranked the vulnerable code as "riskier" than the fixed code 61% to 67% of the time across all five models.
The Comparison: This beat the AI's own spoken answers every single time. Even when the researchers tried to trick the AI into thinking harder by asking it to explain its reasoning step-by-step (a technique called "Chain of Thought"), the AI still failed to distinguish the bad code from the good code in its written output.
The Significance: The paper notes that while 61–67% isn't a perfect score (it's not 100%), it is statistically significant. It is better than a random guess (which would be 50%), and it proves that a "security signal" exists inside the model that the model's own output hides.

What This Means (and What It Doesn't)

The paper explicitly rules out the idea that simply asking the AI to be a security reviewer is enough. The author shows that even with advanced prompting tricks, the AI's written "Yes/No" verdicts are often wrong or tied, making them unreliable for spotting subtle differences between broken and fixed code.

The paper suggests that the AI's internal state carries a "code-security signal" that is currently being ignored. The author is careful not to call this a solved problem or a magic bullet. They describe it as a "modest but consistent" effect that holds up across different types of AI models. They also note that this was tested on Python code and specific types of bugs, so we don't know yet if it works perfectly for every programming language or every kind of security flaw.

The Takeaway

In simple terms, this paper discovered that AI coding agents might be "lying" to us when they say their code is safe, but their internal "brain waves" are telling the truth. By listening to those internal signals instead of just reading the AI's output, we might be able to build better safety nets for the massive amount of code AI is now writing. It's a bit like realizing that while a suspect might say "I didn't do it," their heartbeat is racing in a way that tells a different story. The author hopes that in the future, we can use these "heartbeat" signals to automatically flag dangerous code before it ever reaches a human reviewer.

Technical Summary: Activation Probes Surface Code-Security Signals that the Model's Output Misses

Problem Statement

AI coding agents (e.g., Claude Code, Cursor, OpenAI Codex) are increasingly authoring production code, with empirical audits indicating that 24–29% of AI-generated Python and JavaScript snippets contain security weaknesses. Human security review cannot scale to match the rate of code generation, necessitating automated triage. However, current automated checks are insufficient: static analyzers suffer from high false-positive rates, and prompting Large Language Models (LLMs) for binary security verdicts is non-robust. Trivial code rewrites can flip LLM verdicts, and models often emit identical "YES/NO" answers for both vulnerable and fixed code pairs, rendering their stated output useless for distinguishing risk.

Furthermore, the most widely used coding agents are closed-weight, preventing deploying teams from inspecting their internal states. Teams can instead deploy open-weight models as reviewers, but standard prompting of these reviewers fails to detect vulnerabilities that the model's own hidden states might encode.

Methodology

The author investigates whether linear probes applied to the residual stream activations of open-weight code LLMs can recover security signals that the models' prompted outputs miss.

Models and Data

Reviewer Models: Five open-weight code LLMs across four architecture families were evaluated: Qwen2.5-Coder (7B, 14B), DeepSeek-Coder (33B), Devstral-Small, and Llama-3.1 (8B).
Training Data (SVEN-Python): The probes were trained on paired vulnerable-and-fixed Python functions from the SVEN dataset, covering four specific Common Weakness Enumerations (CWEs): 022, 078, 079, and 089.
Evaluation Data (PATCHEVAL): The probes were tested zero-shot on 234 real-world, disclosed Python vulnerabilities (CVEs) from the PATCHEVAL benchmark. Crucially, the CVEs used for evaluation contained CWE types unseen during training (no overlap with the four training CWEs).
Data Splitting: To prevent data leakage, splits were grouped by project/repository, ensuring no code from a specific repository appeared in both training and evaluation.

Probe Fitting

Architecture: A single linear probe was fitted per model.
Input: Activations were extracted from the input of layer normalization (layers[L].input_layernorm) using nnsight.
Optimization: The author swept across layers, token-pooling strategies (mean, max, last, sliding-window-max), and L2 regularization strengths. The configuration maximizing 5-fold GroupKFold AUC was selected.
Task: The probe learns to distinguish between a vulnerable function and its fixed counterpart based solely on hidden state activations.

Evaluation Metrics

Paired Win-Rate: For each CVE, the probe scores the vulnerable function ( $\hat{p}_{vul}$ ) and the fixed function ( $\hat{p}_{fix}$ ). The win-rate is the fraction of cases where $\hat{p}_{vul} > \hat{p}_{fix}$ .
Comparators: The probe was compared against the same model's prompted output under three conditions:
1. No-shot: Direct query with no examples.
2. Few-shot: Query with examples of held-out weakness types.
3. Chain-of-Thought (CoT): Query requiring step-by-step reasoning before a verdict.
Readout Methods: Comparators were evaluated using both logit readout (renormalized YES/NO probability) and token readout (literal "yes"/"no" string).

Key Results

1. Probes Detect Signals Missed by Prompting

On the unseen, real-world vulnerabilities (single-function CVEs), the linear probe consistently ranked the vulnerable function higher than its fix, achieving a paired win-rate of 61–67% across all five models. This performance is statistically significant ( $p < 0.05$ ) and consistently above the 50% chance line.

In contrast, the models' prompted outputs performed poorly:

Logit Readout: The probe's win-rate was higher than the prompted models' logit readout for every model under every prompt condition. However, while the probe significantly outperformed no-shot prompting on three of the five models and few-shot prompting on four, the advantage over Chain-of-Thought (CoT) prompting read from logits was narrow and statistically significant for only one of the five models. Despite this narrow margin, the probe's win-rate remained higher than CoT for every model.
Token Readout: When reading the literal written verdict, the models failed to distinguish the pairs in 72–97% of cases, assigning the same word to both the vulnerable and fixed functions.

2. Generalization to Unseen Weakness Types

The probes, trained only on four specific CWEs, successfully generalized to 234 CVEs with entirely different weakness types. This suggests the probes capture a generalizable "code-security signal" rather than memorizing specific vulnerability patterns.

3. Internal State vs. Stated Output

The study confirms a gap between a model's internal representation and its generated text. While the model's stated answer (token) often collapses to a tie, the internal probability (logits) and the activation probe retain a signal capable of ranking risk. The probe effectively "surfaces" a security signal that the model's own output mechanism discards.

4. In-Distribution Performance

On the training distribution (SVEN-Python), the probes achieved an AUC of 0.77–0.85 and accuracy of 0.72–0.77 at a 0.5 threshold, significantly outperforming the prompted models (accuracy ~0.50).

Significance and Claims

The paper claims that model activations carry a code-security signal that prompting the same model misses.

Scalable Review: The findings suggest a pathway for self-hosted security controls where an open-weight model acts as a reviewer. By reading the hidden states of this reviewer via a linear probe, teams can flag changes worth human re-review without needing access to the internals of the closed-weight coding agent that generated the code.
Superiority to Prompting: The probe outperforms the same model's no-shot and few-shot prompting strategies on unseen vulnerabilities. While the probe also outperforms Chain-of-Thought (CoT) prompting in terms of win-rate across all models, the advantage over CoT is narrow and not always statistically significant. The clear gap remains over no-shot and few-shot prompting.
Modest but Robust: The author characterizes the effect as "modest" (win-rates in the low 60s) but consistent across architectures and weakness types. They explicitly state that wiring the probe into a coding agent's post-write hook and calibrating it for absolute thresholds are necessary future steps for deployment.

The work extends previous research on linear probes for safety and vulnerability detection by demonstrating transferability to unseen weakness types and real-world CVEs, and by directly comparing probe performance against the model's own prompted reasoning capabilities.