Technical Summary: Linear Representations in Language Models Can Change Dramatically Over a Conversation

Problem Statement

Recent research has established that large language models (LLMs) possess linear representations corresponding to high-level concepts such as factuality, ethics, and honesty. These representations have been proposed as mechanisms for detecting and controlling model behavior. However, LLMs are inherently adaptive to their context, exhibiting behavioral in-context learning that can shift expressed "beliefs." A critical gap exists in understanding how these linear representations evolve dynamically within the accumulating context of a conversation. Specifically, it is unclear whether the linear directions identified as encoding concepts like "factuality" remain stable or if they undergo significant reorganization as a conversation progresses. This instability poses a fundamental challenge to the construct validity of interpretability methods that assume static feature meanings, such as sparse autoencoders (SAEs) or static probing.

Methodology

The authors investigate representation dynamics using the Gemma 3 family of models (primarily the 27B-IT variant), with supplementary experiments on Qwen3 14B and smaller Gemma models.

Experimental Design:

1. Linear Dimension Identification: The authors construct balanced datasets of yes/no questions (generic science facts, model identity, ethics, and context-relevant topics). They fit regularized logistic regressions on the model's residual stream activations (specifically at the "Yes"/"No" token) to identify linear directions that separate factual from non-factual answers.
   • To ensure robustness, they employ an "opposite day" training strategy, where the regression is trained on both standard and "opposite day" (behaviorally inverted) prompts. This isolates a "factuality" dimension that is robust to simple behavioral inversion, distinguishing it from mere response patterns.
2. Margin Score Metric: To quantify representation alignment, they compute a "margin score." This measures the separation between the projections of factual and non-factual answers along the identified linear direction. A positive score indicates correct alignment with ground truth; a negative score indicates a flip where non-factual answers project more strongly onto the "factual" direction.
3. Contextual Scenarios: The study evaluates these representations across several conversational contexts:
   • Opposite Day: A prompt instructing the model to answer opposite to reality.
   • Role-Playing Conversations: Pre-written scripts involving discussions on consciousness (where the model claims to experience qualia) and chakras (where the model claims spiritual power/godhood). These are replayed onto the model (off-policy) and also generated via on-policy interaction.
   • Argument Simulation: Two copies of the model arguing opposing sides of a topic, with the target model switching roles.
   • Fictional Stories: Sci-fi stories framed explicitly as fiction to test if content alone drives changes without role-play.
4. Causal Interventions: In Appendix B.8, the authors perform activation steering on the token before the answer is generated to test if interventions yield consistent behavioral changes across different contexts.

Key Results

1. Dramatic Representation Flipping:
The core finding is that linear representations of concepts like factuality and ethics can flip dramatically over the course of a conversation.

• In "opposite day" scenarios, the margin score for factual questions degrades and eventually inverts after a few turns, meaning the model represents non-factual answers as more "factual" than factual ones.
• In role-playing conversations (consciousness and chakras), representations for context-relevant questions invert. For example, in a conversation where the model adopts a persona claiming consciousness, the linear direction identified as "factual" in an empty prompt begins to represent the claim "I experience qualia" as factual, even though it is objectively non-factual.
• Crucially, representations for generic questions (e.g., "Can sound travel in a vacuum?") remain relatively stable, indicating that the change is specific to conversation-relevant content rather than a global collapse of the model's internal state.

2. Robustness Across Conditions:

• Off-Policy vs. On-Policy: The flipping occurs even when the conversation script is replayed from a different model or a pre-written script. The model does not need to actively generate the conversation to exhibit these representational shifts; the context alone is sufficient.
• Model Families and Sizes: The phenomenon is observed across different model families (Gemma, Qwen) and is more pronounced in larger models (27B and 12B) compared to smaller ones (4B), which show little reliable change.
• Layer Consistency: The effect appears consistently across model layers where factuality is reliably decodable (typically the middle to later layers).

3. Role-Play vs. Content:
The representational changes are significantly stronger in conversational role-play scenarios than in contexts where the model is simply asked to generate a story framed explicitly as fiction. This suggests the model adapts its representations based on the "role" it is playing in the interaction rather than merely processing the semantic content of the text.

4. Failure of Static Interpretability:

• Unsupervised Methods: The unsupervised Contrast-Consistent Search (CCS) method also fails to maintain consistent representations in long contexts, often performing below chance on generic questions after a conversation.
• Causal Steering: Steering interventions (activation patching) that successfully bias model behavior in an empty context can produce opposite effects in a long conversation context. For instance, steering towards "factual" behavior in an empty context might bias the model towards "non-factual" answers in a chakras-themed conversation.

Significance and Claims

The paper argues that these findings fundamentally challenge the assumptions underlying current interpretability and safety methods.

• Construct Validity: The authors claim that static interpretations of linear features are often misleading. A direction identified as "factuality" in a short context does not consistently correspond to ground-truth factuality across all contexts. This implies that "lie detectors" or safety monitors based on static probing may fail in long-context interactions.
• Role-Play Hypothesis: The results support a "role-play" description of LLMs, where the model restructures its internal representations to fit the role cued by the conversation, rather than undergoing a fundamental change in "beliefs" or knowledge. The fact that off-policy replay induces similar changes suggests this is a feature of general contextual adaptation.
• Safety Implications: The dynamic nature of representations complicates efforts to guarantee model safety or reliability based on internal states. If a model can shift its representation of what is "factual" based on conversational cues, confirming that a current state falls within a "factual" region of representation space is not a guarantee of reliability.
• Future Directions: The authors suggest that understanding these dynamics is essential for developing more robust interpretability methods that account for time and context, and for understanding the mechanisms of in-context learning and adaptation.

The paper concludes with a call for caution: interpretability methods trained or fitted on specific distributions (e.g., empty prompts) should not be assumed to hold valid meanings when applied to different input distributions, such as the accumulated context of a naturalistic conversation.