Technical Summary: How Context Shapes Truth

Problem Statement

While prior research has established that Large Language Models (LLMs) encode the truthfulness of statements as linear directions (vectors) within their residual stream activations, the geometric evolution of these "truth vectors" when contextual information is introduced remains unexplored. Existing studies demonstrate that truth can be separated from falsehoods in activation space and that probes can generalize across settings, but they do not characterize how the geometry of these representations—specifically their direction and magnitude—transforms when a model processes a statement alongside context. This gap is critical for understanding how LLMs integrate external information, particularly in Retrieval-Augmented Generation (RAG) and in-context learning scenarios.

Methodology

The authors propose a framework to measure the geometric transformations of truth vectors induced by context. The experimental setup involves four instruction-tuned LLMs (LLaMA-3.1-8B, Mistral-Nemo-12B, Qwen3-4B, and SmolLM3-3B) and four datasets spanning diverse domains (fact-checking, legal, movie synopses, and synthetic knowledge conflicts).

Core Procedure:

1. Prompt Construction: For each statement, four prompts are generated by varying the [Selected Choice] (supporting or refuting the statement) and the presence of [Context] (with or without). The model is instructed to generate a completion starting with a specific token (e.g., ")") to ensure fair comparison.
2. Vector Extraction: Truth vectors are extracted from the residual stream activations (post-MLP) at the position of the first generated token. For a statement $k$ in layer $l$, the truth vector without context ($v_{k,nc}^{(l)}$) and with context ($v_{k,c}^{(l)}$) are defined as the difference between activations for true and false completions:
   $$v_{k,nc}^{(l)} = a_{k,True, nc}^{(l)} - a_{k,False, nc}^{(l)}$$
   $$v_{k,c}^{(l)} = a_{k,True, c}^{(l)} - a_{k,False, c}^{(l)}$$
3. Geometric Metrics: Two properties are measured to quantify the change:
   • Directional Change ($\theta$): The angle between the truth vector with context and without context, calculated via the cosine similarity.
   • Relative Magnitude: The ratio of the L2 norm of the truth vector with context to the norm without context ($||v_{k,c}^{(l)}||_2 / ||v_{k,nc}^{(l)}||_2$). Values $>1$ indicate an amplification of the separation between true and false representations.

The study also compares relevant context against five types of random/irrelevant context (random characters, random words, incoherent sentences, random Wikipedia text, and shuffled dataset contexts) to assess sensitivity to relevance.

Key Results

1. Three-Phase Pattern of Directional Change

Across all models and datasets, the directional change ($\theta$) follows a consistent three-phase pattern across layers:

• Phase 1 (Early Layers): Truth vectors with and without context are roughly orthogonal (high $\theta$), suggesting distinct representations in early processing stages.
• Phase 2 (Middle Layers): Vectors converge sharply, reaching a minimum $\theta$. This aligns with the known role of middle layers in semantic encoding and knowledge retrieval. Notably, $\theta$ never reaches zero, indicating that context always leaves a distinct geometric signature.
• Phase 3 (Later Layers): Behavior varies. For some datasets, vectors stabilize. For others (specifically when context conflicts with parametric knowledge, e.g., ConflictQA-Counter), $\theta$ increases again, suggesting continued processing of conflicting signals in deeper layers.

2. Amplification of Truth Separation

Adding context generally increases the relative magnitude of truth vectors (values $>1$). This indicates that context amplifies the separation between true and false representations in the activation space. This effect is most pronounced in the middle layers, where relative magnitude often peaks before stabilizing in later layers.

3. Model Size and Sensitivity Mechanisms

A significant divergence exists between larger and smaller models regarding how they encode context relevance:

• Larger Models (LLaMA, Mistral): Distinguish relevant from irrelevant context primarily through directional change ($\theta$). Relevant context induces significantly higher directional shifts compared to random context.
• Smaller Models (Qwen, SmolLM): Distinguish relevant from irrelevant context primarily through magnitude scaling. While directional changes are often negligible or negative, the relative magnitude of truth vectors is significantly higher for relevant context. The authors hypothesize this is due to representational capacity constraints in lower-dimensional spaces, making magnitude a more feasible encoding strategy than distinct directions.

4. Impact of Knowledge Conflict

Context that contradicts parametric knowledge (ConflictQA-Counter) produces larger geometric changes (both higher $\theta$ and magnitude) than context aligned with parametric knowledge. This suggests that conflicting information triggers stronger, more persistent geometric shifts, potentially reflecting competition between memory and context pathways.

5. Domain Specificity

In complex, domain-specific datasets like LegalBench (Corporate Lobbying), models often fail to show significant geometric differences between relevant and random context. This correlates with low Flesch readability scores, suggesting that highly technical language may overwhelm the model's ability to extract meaningful geometric signals distinguishing relevance.

Significance and Claims

The paper claims to provide the first geometric characterization of how context transforms truth representations in LLMs. The findings offer a theoretical lens for understanding internal context processing and have practical implications for:

• RAG Systems: Layer-wise $\theta$ values could serve as diagnostic signals to detect when a model fails to integrate retrieved context (lack of middle-layer convergence) or when knowledge conflicts exist (abnormal late-layer divergence).
• Steering Smaller Models: The discovery that smaller models rely on magnitude rather than direction suggests that activation steering techniques for these models should focus on magnitude scaling rather than purely directional interventions.

The authors maintain a modest stance, noting that while these geometric changes are significant, they do not always translate linearly to output probability changes. They also acknowledge limitations, including the focus on the first token, the use of English-only datasets, and the specific nature of the instruction-tuned models studied.