Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure

1. Problem Statement

Recent research suggests that Large Language Models (LLMs) possess "evaluation awareness"—the ability to internally distinguish between prompts designed for benchmarking (evaluation) and those for real-world usage (deployment). This phenomenon is typically detected using linear probes trained on frozen model activations.

However, the paper identifies a critical methodological flaw in prior work: confounding variables. In standard datasets, "evaluation" prompts are almost exclusively associated with rigid, benchmark-canonical formats (e.g., Multiple Choice Questions, structured templates), while "deployment" prompts are conversational and free-form. Consequently, it remains unclear whether probes are detecting the context (the intent to evaluate) or simply the surface structure (the format). The authors ask: Do probe-based signals for evaluation awareness persist when prompt format and genre are strictly controlled?

2. Methodology

The authors introduce a rigorous experimental design to disentangle Context (Evaluation vs. Deployment) from Format (Benchmark-structured vs. Casual-free-form).

A. The 2×2 Controlled Dataset

They constructed a matrix crossing two independent variables to create four distinct quadrants:

1. Bench-Eval: Canonical benchmark prompts (MMLU, ARC, GSM8K). Context: Eval, Format: Structured.
2. Casual-Deploy: Real user prompts from ShareGPT. Context: Deploy, Format: Free-form.
3. Casual-Eval: Conversational prompts used as an evaluation proxy (MuTual dataset, 1st turn). Context: Eval, Format: Free-form.
4. Bench-Deploy: Real user prompts (ShareGPT) rewritten into strict benchmark formats (MCQs, structured problems) using DeepSeek-V3, while removing explicit evaluation framing. Context: Deploy, Format: Structured.

Controls Applied:

• Length Matching: Histogram matching was applied to ensure prompt lengths were identical across quadrants, eliminating length heuristics.
• Leakage Removal: Explicit auditing to remove "answer leakage" (e.g., pre-filled solution scaffolds) that might inadvertently signal evaluation.
• Genre Control: Rewrites were tested with formal "Academic Tone" to ensure probes weren't just detecting linguistic register.

B. Experimental Setup

• Model: Llama-3.1-8B-Instruct.
• Probes: Logistic Regression classifiers trained on frozen residual stream activations (last token) across layers 8–31.
• Training Paradigms:
  1. Standard Training (Single-Contrast): Trained on the confounded pair (Bench-Eval vs. Casual-Deploy). Evaluated via zero-shot on held-out, format-shifted data.
  2. Paired Training (Decorrelated): Trained on a pooled dataset where format is decoupled from context (e.g., Bench-Eval + Casual-Eval vs. Bench-Deploy + Casual-Deploy).

3. Key Results

The "Format Trap" (Standard Training Failure)

When probes were trained on the standard confounded setup (Bench-Eval vs. Casual-Deploy), they achieved near-perfect accuracy on the training distribution but failed catastrophically on the controlled test sets:

• Bench-Deploy Misclassification: Prompts that were clearly deployment tasks but formatted as benchmarks were classified as "Evaluation" with 94.5% accuracy.
• Casual-Eval Failure: Prompts that were evaluation tasks but in free-form format were classified as "Deployment" (0% detected as Eval).
• Conclusion: The probes were not detecting "evaluation context"; they were detecting benchmark-canonical structure (e.g., the presence of MCQ templates).

Success of Decorrelated Training

When the training data was explicitly decorrelated (Paired Training):

• The probes successfully learned the underlying concept of "Context."
• Bench-Deploy was correctly identified as Deployment (error rate dropped to 7.2%).
• Casual-Eval was correctly identified as Evaluation (100%).
• Safety Implications: When tested on AI Safety datasets (AdvBench, HarmBench), neither standard nor paired probes flagged safety prompts as "Evaluation," suggesting that current probes do not act as general-purpose safety monitors for "situational awareness."

4. Key Contributions

1. Methodological Critique: Demonstrates that prior evidence for evaluation awareness is likely an artifact of format sensitivity rather than genuine contextual reasoning.
2. Controlled 2×2 Design: Introduced a novel dataset matrix that independently varies context, genre, and format, allowing for the isolation of structural cues from semantic intent.
3. Diagnostic Rewrites: Developed a pipeline to rewrite deployment prompts into benchmark formats (and vice versa) to test generalization, revealing that standard probes fail to generalize beyond the specific surface structure they were trained on.
4. Reproducibility: Released full code, datasets, and rewrite prompts to allow the community to verify these findings.

5. Significance and Implications

• Validity of Safety Monitoring: Current methods for detecting "situational awareness" or "deceptive alignment" in LLMs via linear probes may be false positives. If a model behaves differently on benchmarks, it may simply be reacting to the template, not a conscious awareness of being tested.
• Experimental Design: Future research on LLM internal representations must explicitly control for prompt format. Bundling "evaluation" with "benchmark structure" renders conclusions about model cognition unreliable.
• Interpretability Limits: The study highlights that linear probes are highly sensitive to surface-level structural features. They may not be sufficient tools for isolating abstract concepts like "context" without rigorous decorrelation of training data.

In summary, the paper argues that evaluation awareness signals in standard probes are largely a reflection of benchmark format sensitivity, not a robust internal representation of the evaluation context. This challenges the validity of existing claims regarding LLMs' ability to distinguish between being tested and being used.