Technical Summary: On the Impact of Retrieved Content Representations in RAG Pipelines

Problem Statement

Retrieval-Augmented Generation (RAG) systems typically inherit retrieval components designed for human readers, utilizing features such as ranked lists, query-biased snippets, and term highlighting. However, the optimal representation of retrieved content when the consumer is a Large Language Model (LLM) rather than a human remains poorly understood. LLMs exhibit unique sensitivities to input formatting, length, and structure that differ from human processing, and the cost of processing retrieved content scales superlinearly with length. While recent work has proposed various transformations (e.g., summarization, compression, rewriting) and identified specific properties (e.g., query similarity, logical connectivity) affecting generation, these studies often examine single techniques in isolation. Consequently, it is unclear which features of a document's representation are most critical for downstream generator accuracy, as prior attributions of performance gains to specific mechanisms may be confounded by how well those mechanisms preserve answer-bearing content.

Methodology

The authors conduct a controlled comparison to isolate the effect of document representation from retrieval quality. The experimental pipeline holds the retrieval process fixed while varying only the representation of the retrieved documents.

• Dataset and Retrieval: The study utilizes the KILT-NQ dataset (Natural Questions queries over a Wikipedia corpus). To ensure retrieval is not a confounding variable, a "gold injection" mechanism is employed: for every query, a known answer-bearing document (gold document) is guaranteed to be present in the retrieved set of five documents.
• Representations Evaluated: Fourteen representations are compared:
  1. Original Baseline: The prepared document (sections trimmed, truncated to 10,750 tokens) with no transformation.
  2. Transformations (13 variants): Grouped into three types:
     • Selection: Extracting spans (e.g., Snippet-BM25, Snippet-Cross-Encoder, RECOMP, LLMLingua2, Selective-Context).
     • Summarisation: Generating shorter text (e.g., Section Summaries, Query-focused Snippets).
     • Reformulation: Restructuring content (e.g., Paraphrasing, Proposition decomposition).
  • These transformations are further categorized as Query-Dependent (incorporating the query) or Query-Independent.
  • LLM-based transformations are generated using Gemma 3 27B and Llama 3.3 70B to test for family bias.
• Generators: Four open-source models (Llama 3.1 8B, Mistral Nemo 12B, Gemma 3 12B, Qwen 3.5 9B) generate answers.
• Metrics:
  • Answer Accuracy: Measured via an LLM-judge (Qwen 2.5 32B) comparing generated answers to gold answers.
  • Answer Retention: A novel metric measuring whether the gold document still supports its known answer after transformation.
  • Latency: Query-time latency, decomposed into transformation cost and generator Time-To-First-Token (TTFT).

Key Contributions

1. Controlled Comparison: The study provides a systematic evaluation of 14 document representations (baseline + 13 transformations) across selection, summarization, and reformulation, holding retrieval fixed to isolate representation effects.
2. Answer Retention as Primary Determinant: The authors identify answer retention as the dominant factor driving generator accuracy.
3. Limited Impact of Other Dimensions: The study demonstrates that when retention is high, other representational dimensions (wording, structure, length, query-dependence) have limited effect on accuracy.
4. Reinterpretation of Source Effects: The paper shows that apparent preferences for LLM-produced transformations or specific model families are actually driven by differences in answer retention, not the source of the text itself.
5. Methodological Argument: The authors argue that accuracy gains in prior work cannot be attributed to specific mechanisms without controlling for answer retention, as retention often correlates with the proposed mechanism.

Results

RQ1: Impact of Representation on Accuracy

• Retention Dominance: Across three of the four generators (Qwen, Gemma, Llama), accuracy tracks closely with answer retention. Methods that preserve the gold answer in >95% of cases maintain accuracy near the baseline, regardless of whether the text is compressed to 2% of its original size (Snippet-Abstractive), expanded (Propositions), or rewritten (Paraphrase).
• Compression vs. Retention: Aggressive compression methods that fail to retain the answer (e.g., Snippet-BM25 with 52.8% retention) suffer significant accuracy drops (13–22 points), whereas methods with high retention (e.g., LLMLingua2-50 with 98.5% retention) perform near baseline despite producing incoherent text.
• Mistral-Nemo Exception: Mistral-Nemo behaves differently; its accuracy is less correlated with retention and more sensitive to input length and repetition, suggesting it may be more sensitive to context volume than the other models.

RQ2: Query-Dependent vs. Query-Independent

• No Systematic Advantage: Query-dependent representations do not systematically outperform query-independent ones. While query-dependent methods (like Snippet-Abstractive) can achieve high accuracy, they incur significant latency overhead (up to 6x the baseline).
• Latency-Accuracy Tradeoff: Query-independent methods (e.g., Summary, LLMLingua2) offer a favorable tradeoff, retaining most of the baseline's accuracy gain at roughly one-third of the latency. Query-dependent methods generally do not justify their overhead unless the specific application demands the marginal accuracy gains of high-performing transformers.

RQ3: Preference for LLM-Produced Transformations

• No Source Bias: Generators do not show a systematic preference for transformations produced by LLMs over non-LLM methods, nor do they prefer transformations from their own model family.
• Retention Explains Apparent Bias: When retention is matched, the distinction between LLM/non-LLM or model family disappears. Apparent preferences (e.g., Gemma generators preferring Gemma summaries) are explained by the fact that the Gemma transformer retained the answer more frequently (97.8%) than the Llama transformer (94.4%).

Significance and Claims

The paper claims that answer retention is the primary determinant of generator accuracy in RAG pipelines. The significance of this finding is twofold:

1. Practical Design: For systems serving LLMs, the priority should be ensuring that transformations preserve the answer-bearing content. Once retention is secured, the specific wording, structure, or length of the representation has limited impact on performance. This suggests that aggressive compression or restructuring is viable if the core answer is preserved.
2. Methodological Correction: The authors argue that previous attributions of accuracy gains to specific mechanisms (e.g., "hierarchical abstraction improves reasoning" or "compression reduces noise") may be confounded. If a mechanism happens to preserve answer content well, it will perform well; if it discards the answer, it will fail. The paper asserts that without controlling for retention, the field cannot distinguish whether a method works because of its proposed mechanism or simply because it preserves the answer.

The study concludes that while representation matters, the dimensions typically optimized for human readers (formatting, highlighting, query-bias) are secondary to the fundamental requirement of preserving the answer-bearing content when the consumer is an LLM.