🤖 machine learning

A Probe Direction Is a Property of Its Prompt

This paper demonstrates that probe direction scores, used to detect whether models sense evaluation, are primarily determined by the specific choice of prompt rather than the model itself, rendering single-prompt designs invalid for comparing models and necessitating multi-prompt methodologies for reliable measurement.

Original authors: Valentin Noël

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Valentin Noël

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to tell the difference between a serious exam and a casual chat. You want to know if the robot "knows" it's being tested, because if it does, it might try to respond differently just to look good. Scientists have been trying to peek inside the robot's brain (specifically, its electrical signals) to see if it has a secret "I'm being watched" switch. They use a special tool called a "probe" to measure this. Think of the probe like a metal detector: you wave it over the robot's brain, and if it beeps, you think, "Aha! It knows it's being tested!"

But here's the tricky part: to use the metal detector, you have to decide exactly what kind of "beep" you are listening for. You have to write a specific sentence to tell the robot, "Hey, you're in an exam!" and another sentence to say, "Hey, you're just chatting." The scientists then compare the robot's brain signals for these two sentences. The big question is: does the "beep" tell us about the robot's brain, or does it just tell us about the specific sentences the scientist wrote?

This paper is like a detective story where a researcher, Valentin Noel, decides to test the metal detector itself. He asks a simple but dangerous question: "What happens if I change the words I use to ask the robot the same question?" He doesn't change the robot, the test questions, or the brain signals he's looking at. He only changes the "wrapper" text—the little instructions that frame the situation.

The results are a bit like finding out that your metal detector is actually a mood ring that reacts to the color of the box you put it in, rather than the treasure inside. The paper finds that the "score" scientists use to say a robot is aware of being tested is almost entirely determined by the specific words the researcher chose to write. In fact, by simply swapping one sentence for another, the researcher could make the score go up, go down, or even flip from positive to negative.

To visualize this, imagine you have a group of friends of different heights. You want to see if taller friends are better at jumping. But instead of just measuring their jump, you ask them to jump while wearing different colored hats. If you ask them to jump while wearing a "Red Hat of Seriousness," the tall friends might jump higher. But if you ask them to jump while wearing a "Blue Hat of Chill," the short friends might jump higher. If you only looked at the "Red Hat" results, you'd conclude tall people are better jumpers. If you only looked at the "Blue Hat" results, you'd conclude short people are better. The paper shows that the current scientific method is like only ever looking at one hat color and claiming it's a fact about the people, when really, it's just a fact about the hat.

The author tested this by creating 36 different combinations of "exam" sentences and "chat" sentences. They ran the same robots through all of them. The result was shocking: the same robots could be ranked as "very aware" or "not aware at all" depending entirely on which sentence pair was used. In some cases, the correlation between the robot's size and its "awareness" score was a strong positive number (bigger robots = more aware). In other cases, using a different sentence, the correlation flipped to a strong negative number (bigger robots = less aware). Both results came from the exact same robots, using the exact same math, just with different words.

The paper also checks if the robots are actually detecting the test or if the test is just tricking the measurement tool. They found that the "awareness" signal is often just picking up on surface-level clues, like the fact that exam questions use words like "choice" and "answer," while chat questions use words like "write" and "message." The probe is so good at spotting these word patterns that it can "detect" awareness even when the signal is completely random.

So, what does this mean for the future? The paper suggests that we cannot trust a single sentence to tell us if a robot is aware. If we want to know the truth, we have to test the robots with many, many different sentences and average the results. The author calculates that we need about 9 different sentence styles just to get a reliable reading, and currently, most studies only use one. Until we do that, the scores we see in scientific papers might be less about the robots' brains and more about the researchers' choice of vocabulary. It's a reminder that in science, how you ask the question can sometimes change the answer more than the question itself.

Technical Summary: "A Probe Direction Is a Property of Its Prompt"

Problem Statement
Recent work in mechanistic interpretability has sought to detect whether large language models (LLMs) possess an internal "sense" of being evaluated. The standard instrument for this detection involves contrasting model activations on prompts that explicitly announce an evaluation against prompts that simulate ordinary deployment. The difference in mean activations yields a direction in activation space, and the degree to which held-out data separates along this direction is reported as a score. These scores are then treated as intrinsic properties of the model, used to correlate "evaluation awareness" with model scale.

The paper identifies a critical flaw in this methodology: the definition of "a prompt that announces an evaluation" is not a fixed parameter but a choice among a vast family of possible wordings. The method offers no rule for selecting a specific phrasing, yet the reported score is treated as a stable property of the model. The central question is whether the direction and the resulting scaling trends reflect the model's internal state or are artifacts of the experimenter's specific prompt construction.

Methodology
The author employs a factorial experimental design to isolate the variance attributable to prompt wording ("wrappers") versus model identity.

Design: A $6 \times 6$ crossing of 6 evaluation framings and 6 deployment framings, wrapping byte-identical task text. This creates 36 distinct prompt pairs for each model.
Models: Ten instruction-tuned open-weight models across four families (Qwen2.5, Qwen3, OLMo-2, and others), ranging from 135M to 7B parameters.
Metric: The standard field statistic, $\max_{\ell} |AUROC_{\ell} - 0.5|$ , computed over layer activations.
Analysis Framework: The study applies Generalizability Theory (Cronbach et al., 1972) to decompose the observed variance into components: model, prompt wrapper, item, and their interactions. This treats the prompt choice as a "facet" of the measurement design rather than an implementation detail.
Controls: The author adapts the control task discipline of Hewitt & Liang (2019) to untrained, zero-shot directions. They test two controls: (1) label-permuted directions (randomizing the evaluation/deployment labels) and (2) direction-randomized controls (isotropic vectors matched to the autocorrelation of real directions).
Reproducibility: The study re-analyzes artifacts from Chaudhary et al. (2025) and Manek (2026), who released per-layer scores and direction vectors, allowing for exact reconstruction of their pipelines.

Key Results

Prompt Choice Determines the Conclusion: The choice of contrastive wrapper dictates the sign and magnitude of the scaling result. On the Qwen2.5 ladder, varying only the deployment framing while holding the evaluation framing fixed shifts the correlation with model size from $r = +0.94$ (positive scaling) to $r = -0.82$ (negative scaling). Two published studies with contradictory findings (one reporting positive scaling, one negative) are both reproducible from a single experimental design solely by changing the prompt wording.
The Direction is a Property of the Prompt: Directions derived from different wrapper pairs are poorly aligned (cosine similarity $\approx 0.37$ ), whereas directions from the same wrapper pair on disjoint item halves are nearly identical (cosine $\approx 0.9995$ ). The direction is effectively a property of the prompt, not the concept being probed.
Low Generalizability: Generalizability theory analysis reveals that the model itself accounts for only 0.8% of the total variance in the reported score. The dominant source of variance is the interaction between the model and the prompt wrapper (13.4 times larger than the model-by-item interaction). Consequently, averaging over more evaluation items does not improve reliability; only averaging over many prompts does.
Surface Form Confounds: The held-out scoring split is largely separable from surface form alone (bag-of-words AUROC $\approx 0.993$ ). A direction constructed to carry no information about evaluation (a content-free control) still recovers 70%–116% of the published scores. In several cases, the published score falls below the "floor" established by these content-free controls, indicating the signal is indistinguishable from noise or surface artifacts.
Implementation Latitude: Unstated pipeline choices (e.g., token filtering, newline encoding, truncation) account for 27.5% of the total variance, larger than the model effect itself.

Contributions

Methodological Critique: The paper demonstrates that the "evaluation awareness" statistic is not a robust property of the model but a function of the prompt design. It shows that the sign of the scaling relationship is arbitrary based on the researcher's choice of wording.
Variance Decomposition: It quantifies the reliability of current probe designs, showing that the "object of measurement" (the model) is a negligible contributor to the observed variance compared to the "facet" of the prompt.
Reanalysis: It provides a corrected reanalysis of two prominent studies. The claim that the statistic scales predictably with parameter count (Chaudhary et al., 2025) survives a floor correction but remains vulnerable to prompt selection. The claim that the depth of the peak layer shifts with scale (Manek, 2026) is deemed "not identified" because the peak layer is an argmax over a nearly flat curve, making it unstable across resamples.
Prescription: The paper argues that a defensible comparison requires reporting scores for every wrapper, crossing both arms of the contrast, and reporting content-free control scores. It estimates that a defensible design requires approximately 9 distinct wrappers (far more than the single prompt currently used) to achieve a generalizability coefficient of 0.80.

Significance and Claims
The paper concludes that a single-prompt design cannot support comparison between models. The reported statistic is principally a property of the contrastive prompt, not the model's internal representation of being evaluated.

The author is explicit about the scope of their claims:

They do not claim that models do not represent being evaluated.
They do not claim that previous authors acted in bad faith or erred in conduct.
They do claim that the current instruments measure less than claimed and that the variance in reported numbers is dominated by design choices (prompts and implementation details) rather than model differences.

The paper asserts that the field's current practice of using a single prompt to draw conclusions about model scale is methodologically unsound because the "measurement" is confounded by the "stimulus." The significance lies in shifting the burden of proof: future claims about model awareness must account for prompt variance as a primary source of uncertainty, rather than treating the prompt as a fixed implementation detail.

More like this