CHOP: Chunkwise Context-Preserving Framework for RAG on Multi Documents

1. Problem Statement

Current Retrieval-Augmented Generation (RAG) systems face significant challenges when processing multi-document collections (e.g., technical manuals, policy books) characterized by high lexical and semantic overlap.

• Fragmentation Issues: Standard length-based chunking disrupts discourse continuity, breaking coreference links (e.g., "this method," "Eq. 3") and local references, which reduces interpretability and recall.
• Semantic Conflicts: In multi-document settings, near-duplicate content and conflicting references create retrieval ambiguity. Similar documents often collide in the vector space, leading to:
  • Retrieval confusion (retrieving irrelevant but semantically similar chunks).
  • Hallucinations and factual errors in the generated output.
  • Inability to distinguish between different topics or product models within the same document flow.
• Independence Limitation: Most existing approaches process chunks independently, ignoring discourse-level dependencies (definitions, transitions, coreference) that span across chunk boundaries.

2. Methodology: The CHOP Framework

CHOP (Chunkwise Context-Preserving) is a framework designed to enhance retrieval by processing chunks sequentially and enriching them with context-aware metadata. It operates on the principle that chunks belonging to the same topical flow should share a consistent semantic signature, while topic shifts should trigger a new signature.

The framework consists of two primary components:

A. CNM-Extractor (Category–Noun–Model Extractor)

This module generates a compact, structured signature for each chunk to anchor its semantic position.

• Input: A text chunk $C_i$.
• Output: A triplet $M_i = \{Category, Nouns, Model\}$.
  • Category: The broad product family (e.g., "air conditioner").
  • Nouns: 1–2 key nouns capturing the core target/action, formatted strictly (e.g., " ").
  • Model: Specific model or series names (e.g., "225B").
• Mechanism: An LLM extracts this structured JSON data. If a field is ambiguous or absent, it is set to null.
• Function: This signature acts as a prefix to the chunk, stabilizing the contextual frame and clarifying references to entities and units before embedding.

B. Continuity Decision Module

This module determines whether a new chunk belongs to the same document flow as the previous one or represents a topic shift.

• Input: A pair of adjacent chunks $(C_i, C_{i+1})$.
• Decision Logic: An LLM-based classifier evaluates whether $C_{i+1}$ is a continuation of $C_i$ based on decision rules (e.g., same product category/model vs. new model or new document markers).
• State Update:
  • If Continuity = TRUE: The system inherits the previous chunk's CNM ($M_{i+1} = M_i$). This preserves semantic coherence across the flow.
  • If Continuity = FALSE: A new CNM is extracted for the current chunk ($M_{i+1} = \text{CNM}(C_{i+1})$), marking a topic shift and preventing error propagation.
• Embedding Construction: The final input for the vector database is constructed by prefixing the chunk with its determined CNM:
  $$x_i = [\text{PFX}(M_i) \parallel C_i]$$
  $$e_i = \text{Embed}(x_i)$$

3. Key Contributions

1. Context-Preserving Chunking: Unlike static or purely semantic chunking, CHOP dynamically maintains context across chunks by inheriting metadata (CNM) when topics are continuous, effectively solving the "orphaned reference" problem.
2. Semantic Regularization: By prefixing chunks with structured CNM signatures, CHOP regularizes the embedding space. This reduces semantic collisions between similar but distinct documents (e.g., different models of the same product), improving the retriever's discrimination power without retraining the retriever model.
3. Scalable Multi-Document Handling: The framework is specifically designed for long, stitched technical documents where sub-documents have high overlap, a scenario where traditional RAG pipelines often fail.

4. Experimental Results

The authors evaluated CHOP on the MRAMG-Bench dataset (derived from product manuals), comparing it against Naive-500T (fixed-length 500-token chunks) and Cosine-Chunking (semantic boundary detection).

Retrieval Performance (Table 1):

• Top-1 Hit Rate: CHOP achieved 90.77%, significantly outperforming Naive-500T (81.28%) and Cosine-Chunking (70.77%).
• Ranking Quality: CHOP showed superior performance in MRR@K and NDCG@K across all $K$ values. For example, NDCG@10 reached 0.9291 for CHOP vs. 0.8698 for Naive-500T.
• Insight: CHOP not only retrieves the correct evidence more frequently but also ranks it higher, indicating better relevance discrimination.

Generation Performance (Table 2):

• Using retrieved chunks as input for an LLM (Gemma-12B), CHOP consistently yielded higher generation quality.
• Metrics: CHOP achieved gains in F1, ROUGE-L, and BERTScore across Top-3, Top-5, and Top-10 retrieval settings.
• Impact: The improved retrieval directly translated to reduced hallucinations and more accurate answers, demonstrating that better grounding leads to better generation.

5. Significance and Conclusion

• Practical Baseline: CHOP offers a practical, scalable approach to building high-quality knowledge bases for RAG in real-world, multi-document scenarios without requiring complex retraining of the underlying retriever.
• Efficiency vs. Effectiveness: It balances computational cost (using LLMs only for metadata extraction and continuity decisions) with significant gains in retrieval accuracy and generation reliability.
• Future Directions: The authors plan to explore adaptive prefixing, dynamic continuity modeling for streaming inputs, and lightweight strategies to further reduce inference costs.

In summary, CHOP addresses the critical gap in current RAG systems where context is lost during segmentation. By enforcing contextual continuity through metadata inheritance and semantic regularization via structured prefixes, it significantly mitigates retrieval confusion in complex, overlapping document collections.