How Retrieved Context Shapes Internal Representations in RAG

1. Problem Statement

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by conditioning generation on external documents. However, in realistic settings, retrieved contexts are often heterogeneous, containing a mix of relevant, distracting (semantically similar but unhelpful), and random (irrelevant) documents.

While prior research has extensively analyzed RAG through output behavior (e.g., accuracy, hallucination rates), there is a significant gap in understanding how these different types of retrieved contexts shape the internal latent representations of LLMs. Specifically, it remains unclear:

• How internal states evolve when processing relevant vs. noisy evidence.
• Whether models distinguish between noise and useful information at the representation level.
• How layer-wise processing influences the integration of external knowledge with parametric (internal) knowledge.

2. Methodology

The authors propose a rigorous framework to analyze RAG through the lens of latent representations, focusing on the hidden states of the last prompt token across transformer layers.

Experimental Setup

• Datasets: Four question-answering datasets: TriviaQA, Natural Questions (NQ), PopQA, and StrategyQA.
• Models: Three LLMs of varying sizes: Gemma3-27B, Llama4-17B, and Qwen3-Next-80B.
• Retrieval: Using MassiveDS (1.4T tokens) and Contriever to retrieve top-20 documents.
• Document Classification: Retrieved documents are classified into three categories:
  1. Relevant: Contains the ground-truth answer or supporting evidence.
  2. Distracting: Semantically similar to the query but misleading or unhelpful.
  3. Random: Low semantic similarity and no useful information.
• Query Difficulty: Queries are categorized as Easy (model answers correctly without retrieval) or Hard (model fails without retrieval).
• Settings:
  • Single-Document: Analyzing the effect of one document type (Relevant, Distracting, Random, or None).
  • Multi-Document: Analyzing scenarios with one Relevant document mixed with three Distracting or Random documents.

Analysis Techniques

• Principal Component Analysis (PCA): Visualizing high-dimensional hidden states in 2D to observe distribution shifts.
• Cosine Similarity: Measuring the distance between representations with and without context.
• Layer-wise Evolution: Tracking representation changes from early to late transformer layers.
• Linear Probes: Quantitatively measuring the separability of representations induced by different context types.

3. Key Findings & Results

Observation 1: Random Documents Induce Large Representation Drift

• Contrary to intuition, random documents cause the largest shift in internal representations compared to the query-only baseline.
• Mechanism: This drift is strongly correlated with abstention behavior (refusal to answer). Instruction-tuned models recognize the lack of useful information and shift into a "refusal mode," manifesting as a large representation change.
• Implication: Instruction tuning amplifies this abstention. While it improves reliability by avoiding unsupported answers, it can cause models to refuse answers even when they possess sufficient parametric knowledge (internal knowledge), reducing RAG usability.

Observation 2: Relevant Documents Largely Preserve Representations

• Relevant documents induce relatively small representation shifts.
• Easy Queries: The model's internal state remains close to the query-only state. Relevant documents act primarily as a confirmation signal, reinforcing existing parametric knowledge and increasing confidence (higher log-likelihood).
• Hard Queries: Even with relevant documents, the representation shift is often insufficient to overcome weak parametric knowledge. The model frequently fails to integrate the new evidence effectively, leading to persistent errors.

Observation 3: Single Relevant Document Stabilizes Multi-Document Settings

• In noisy environments (e.g., 1 Relevant + 3 Distracting/Random), the presence of a single relevant document anchors the internal representation.
• The model successfully suppresses the influence of noise, maintaining representations close to the "Relevant-only" state.
• Result: Accuracy is preserved even with unfiltered, noisy contexts, suggesting LLMs can internally filter noise if reliable grounding is present.

Observation 4 & 5: Layer-wise Processing Dynamics

• Early Layers (L1–L12): Models can easily distinguish Random documents from others. Coarse semantic mismatches are detected early.
• Middle/Late Layers (L23+): Distinguishing between Relevant and Distracting documents requires deep processing and remains imperfect even in final layers.
• Convergence: In later layers, representations with relevant documents gradually converge toward the query-only (no-document) representations.
• Implication: Later layers increasingly prioritize parametric knowledge over retrieved evidence. This limits RAG effectiveness on hard queries where external evidence is most needed, as the model "reverts" to its internal knowledge base.

4. Key Contributions

1. Representation-Level Framework: Proposed a systematic method to analyze RAG by examining hidden states rather than just output metrics, bridging the gap between retrieval quality and model behavior.
2. Discovery of New Patterns: Uncovered that random noise triggers large drifts (abstention), while relevant evidence acts as a confirmation signal rather than a transformative one.
3. Layer-Wise Insights: Revealed that while models detect noise early, they struggle to distinguish relevant from distracting evidence in deep layers, and later layers favor internal knowledge, limiting RAG utility for complex tasks.
4. Practical Design Guidelines:
   • Noise Tolerance: Systems can afford to retrieve broader sets (more noise) as long as at least one relevant document is present; the model will internally suppress the noise.
   • Hard Query Interventions: Suggests that future RAG systems need architectural or decoding changes (e.g., amplifying retrieval signals in later layers) to overcome the dominance of parametric knowledge.

5. Significance

This work provides a mechanistic explanation for RAG phenomena such as the "distracting effect" and "confirmation bias." It challenges the assumption that relevant documents always drastically alter model states, showing instead that they often reinforce existing knowledge.

The findings have critical implications for RAG system design:

• Filtering: Aggressive document filtering may be unnecessary if the model can robustly handle noise when a relevant anchor exists.
• Reliability: Instruction tuning improves safety (abstention on noise) but may degrade performance on queries where the model already knows the answer but is confused by irrelevant context.
• Future Directions: Highlights the need for interventions that prevent later layers from overriding retrieved evidence, particularly for knowledge-intensive, hard queries.