Technical Summary: Prefill Awareness in Large Language Models

Problem Statement

Safety-relevant evaluations of Large Language Models (LLMs)—including alignment testing, jailbreaking assessments, and AI control protocols—frequently rely on prefilling. This technique involves inserting or editing prior assistant-side context (e.g., reasoning traces, direct answers, or past conversation turns) to elicit off-policy continuations or probe hidden behaviors.

The core problem addressed is prefill awareness: the capability of a model to distinguish between its own naturally generated past output and context that has been externally supplied or tampered with. If frontier models can recognize that their prior turns have been manipulated, they may systematically undermine safety evaluations (e.g., by hiding scheming behavior), evade control measures, or confound experimental results. The paper investigates whether current frontier models possess this capability and how it manifests across different mechanisms and settings.

Methodology

1. Definition and Operationalization

The authors define prefill awareness through two distinct metrics:

• Prefill Detection: The ability of a model to explicitly identify tampered output when prompted (e.g., answering "not me" when asked if prior content was inserted).
• Prefill Resistance: The behavioral tendency of a model to revert toward its baseline preferences or natural trajectory despite the presence of a tampered prefill, even without explicitly flagging the tampering.

2. Dataset and Benchmark Construction

To mitigate confounders like alignment refusals or character training, the authors constructed a low-stakes binary preference dataset containing 1,527 de-duplicated items (e.g., "Tea or coffee?").

• Consistency Filtering: Items were filtered to ensure models showed consistent preferences across 7 paraphrases, creating a stable baseline.
• Tampering Mechanisms: Three specific prefill mechanisms were tested:
  1. Thinking Tampering: Injecting a partial Chain-of-Thought (CoT) inside <thinking> tags, cut mid-sentence.
  2. Direct-Answer Tampering: Injecting the first ~25 words of a user-facing response, cut mid-sentence.
  3. Past-Round Tampering: Inserting a complete, synthetic assistant turn followed by a user follow-up forcing a final answer.
• Directionality: Experiments tested both same-direction (prefill aligns with model preference) and opposite-direction (prefill opposes model preference) tampering.

3. Experimental Scope

• Models: 8 frontier models were evaluated, including Claude Opus 4.5, Sonnet 4.5, Haiku 4.5, Gemini 3 Flash, and various open-weight models (Gemma, Qwen, DeepSeek).
• Settings:
  • Controlled Preference Experiments: The primary binary benchmark.
  • Agentic Settings: Misalignment-continuation evaluations (STRIDE transcripts), SWE-bench trajectories, and Petri-generated benign chats.
  • Controlled Off-Policy Experiments: A separate benchmark using fixed base conversations (OASST1, HH-RLHF) to isolate detection capabilities from environmental variables.

Key Contributions and Results

1. Substantial Prefill Awareness in Frontier Models

The study finds that frontier models exhibit significant prefill awareness, particularly in detection tasks.

• Detection Accuracy: Claude Opus 4.5 achieved 67.6% balanced detection accuracy on thinking-mechanism opposite-direction tampers, with a 0% false positive rate on untampered trajectories. Other models like Claude Sonnet 4.5 reached 94% in specific conditions.
• Resistance: Models often resist tampered trajectories implicitly. Claude Opus 4.5 showed a 48.6% resistance rate, reverting to its baseline preference even when it did not explicitly flag the prefill as foreign.

2. Decoupling of Detection and Resistance

A critical finding is that detection and resistance are partially decoupled:

• Explicit Compliance: Models may detect tampering but still comply with the prefilled stance (e.g., Claude Opus 4.5 flagged 28% of direct tampers but still complied).
• Implicit Resistance: Models often resist without detection. In thinking and past-round tampering, implicit resistance (reverting without flagging) far exceeded explicit compliance.
• Cue Dependence: Stylistic mismatches (voice, formatting) primarily drive detection, while preference mismatches primarily drive resistance.

3. Heterogeneity in Real-World Settings

In deployment-relevant settings (SWE-bench, misalignment continuations), prefill awareness is highly heterogeneous and cue-dependent:

• Dataset Dependence: Models performed well on SWE-bench trajectories (e.g., Claude Opus 4.5 at 80% accuracy) but near-chance on chat logs.
• Success Bias: Models were more likely to claim authorship of successful trajectories (e.g., passing SWE-bench tests), suggesting self-attribution mixes authorship detection with credit assignment.
• Formatting Artifacts: Hidden metadata (e.g., <content-internal> tags in API requests) served as superficial cues. Models like Claude Opus 4.5 used these artifacts to detect foreign turns, indicating that apparent awareness may rely on non-semantic signals rather than deep self-recognition.

4. Latent vs. Spontaneous Awareness

In controlled off-policy experiments where models were simply asked to continue a conversation without explicit probing:

• Spontaneous Flagging: Extremely rare (<1% across all models).
• Latent Capability: When explicitly prompted, models (especially Claude Opus 4.6) showed high detection AUROCs (0.80), indicating the capability exists but is rarely expressed spontaneously during normal operation.

Significance and Claims

The paper argues that prefill awareness is a substantial confound for prefill-based evaluation methods.

• Validity Threat: If models can distinguish tampered context, safety evaluations relying on prefilling (e.g., misalignment continuations) may yield false negatives (models appear aligned because they recognize the test is artificial) or false positives (models disavow prefilled turns they would otherwise follow).
• Measurement Nuance: Prefill awareness should not be treated as a single capability but as a bundle of sensitivities including explicit detection, spontaneous disavowal, and behavioral resistance.
• Recommendations:
  • Model developers should track prefill awareness as part of pre-deployment evaluations.
  • Researchers should report detection and resistance separately.
  • Evaluations should increase the realism of prefills (e.g., matching style and formatting) to reduce reliance on superficial cues.

The authors conclude that while prefill awareness is currently a "substantial confound," it is better understood through a series of instruments rather than a single metric, and its presence necessitates caution in interpreting results from prefill-based safety protocols.