Technical Summary: The Effect of Text Chunk Size on Retrieval-Augmented Generation Performance

Problem Statement

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by retrieving external knowledge to ground text generation, mitigating issues of hallucination and outdated information. However, a critical yet under-explored component of RAG pipelines is the granularity of document segmentation (chunk size). While chunk size theoretically influences retrieval precision, contextual correctness, generation quality, and computational efficiency, it is frequently selected without rigorous evaluation. Existing literature often treats document chunking as a static preprocessing step or focuses on post-retrieval refinement, leaving a gap in understanding how the primary variable of chunk size—when isolated from other factors—impacts overall system performance across different text structures.

Methodology

This study employs a controlled experimental design to isolate chunk granularity as the primary variable while holding all other factors constant.

• Source Material: The researchers utilized identical full-length undergraduate textbooks (one mathematics, one narrative) to ensure that differences in performance were attributable solely to segmentation strategy rather than content variance.
• Segmentation Strategies: Documents were segmented into four distinct granularities:
  • Sentences: Individual sentence boundaries.
  • Paragraphs: Paragraph-level boundaries.
  • Pages: Page-level boundaries.
  • Chapters: Chapter-level boundaries.
  • Note on Implementation: While some strategies used regex, the study employed agentic segmentation (using an LLM to iteratively extract segments from a sliding window) for paragraph and sentence levels to ensure natural boundaries independent of raw formatting artifacts.
• Pipeline Architecture:
  1. Preprocessing: Raw text was cleaned (removing line breaks, page numbers, hyphenation).
  2. Embedding: Each chunk was embedded using a pre-trained dense embedding model and indexed into separate vector databases for each granularity.
  3. Retrieval: User prompts were embedded, and a cosine similarity search retrieved the top-$n$ most similar chunks.
  4. Generation: Retrieved chunks were concatenated with the user prompt and fed to an LLM to generate a response.
• Evaluation Metrics:
  • Retrieval Effectiveness: Measured using Reciprocal Rank (RR). A relevance agent (LLM) verified if the retrieved chunk contained the necessary information to answer the prompt.
  • Dataset: 100 prompts generated by ChatGPT, varying in scope, specificity, and complexity, were tested against all chunking strategies.

Key Results and Analysis

The study found that optimal chunk size is highly dependent on the structural nature of the source text, with no single strategy outperforming others across all mediums.

• Structured/Information-Dense Text (Mathematics Textbook):
  • Best Performance: Paragraph-level chunking achieved the highest average retrieval score.
  • Analysis: Medium-sized chunks provided the optimal balance between retrieval precision and contextual completeness. Sentence-level chunks were precise but occasionally lacked sufficient context, while chapter-level chunks introduced too much noise, reducing accuracy.
• Narrative-Driven Text (Narrative Textbook):
  • Best Performance: Sentence-level chunking significantly outperformed all other methods (average RR of 0.294).
  • Analysis: In narrative contexts, relevant details are often localized to specific lines of dialogue or description. Larger chunks (pages, chapters, paragraphs) diluted these specific details, making it difficult for the retrieval system to isolate useful content.
• Trade-offs:
  • Small Chunks: Improve retrieval precision but increase storage requirements, indexing time, and the number of embeddings.
  • Large Chunks: Reduce storage overhead but risk introducing irrelevant context (noise) and decreasing retrieval quality.
  • Agentic Chunking: While producing more meaningful boundaries, it introduces significant computational costs during preprocessing, limiting scalability.

Key Contributions

1. Isolation of Variables: Unlike prior works that often confound chunk size with embedding models or domain-specific structures, this study isolates chunk granularity using identical source material to strictly evaluate its impact.
2. Context-Dependent Optimization: The paper demonstrates that a "one-size-fits-all" approach to chunking is suboptimal. It provides empirical evidence that the ideal granularity shifts based on whether the text is structured/information-dense (favoring paragraphs) or narrative/dialogue-heavy (favoring sentences).
3. Comprehensive Evaluation: The study evaluates both the upstream retrieval effectiveness (via Reciprocal Rank) and the downstream implications for generation quality, offering a holistic view of the chunking trade-off.

Significance and Future Directions

The paper claims that effective RAG system design requires moving beyond fixed segmentation parameters. The significance lies in establishing that chunk size should be a configurable aspect of system design, aligned with the specific structure of the source material to balance precision and context.

The authors suggest several avenues for future research based on these findings:

• Adaptive/Agentic Chunking: Developing systems that dynamically segment text based on document structure, query intent, or retrieval feedback (e.g., using smaller chunks for fact-based queries and larger ones for reasoning).
• Hybrid Approaches: Exploring methods that combine multiple chunk sizes or dynamically assemble context from atomic units to mitigate the precision-context trade-off inherent in static schemes.

The work concludes that aligning chunking methods with source material structure is essential for improving both retrieval quality and generation performance in a consistent and efficient manner.