Chunking German Legal Code

Technical Summary: Chunking German Legal Code

Problem Statement
The increasing adoption of Large Language Models (LLMs) in legal services necessitates robust Retrieval-Augmented Generation (RAG) systems to mitigate hallucinations. However, effective retrieval is hindered by the "lost in the middle" phenomenon and context window limitations, which require retrieved information to be small and highly relevant. A critical challenge in legal domains, specifically with the German Civil Code (Bürgerliches Gesetzbuch, BGB), is that standard chunking strategies often sever the deliberate hierarchical structure of statutory text. Legal norms are organized into specific units (sections, subsections, sentences) where conditions and consequences are tightly coupled. Traditional fixed-size windows or semantic aggregations risk breaking these logical boundaries, leading to a loss of context essential for accurate legal question answering (QA). Furthermore, advanced chunking methods introduce significant computational overhead in terms of index build time, query latency, and storage requirements.

Methodology
The authors evaluate a comprehensive range of segmentation strategies on the BGB corpus, normalized into 2,455 hierarchical sections. The evaluation utilizes a legal QA dataset (gerlayqa-bgb-paraphrased) containing 525 layperson questions with section-level gold labels. The study compares the following approaches:

1. Structure-Preserving Baselines:
   • Fixed-Size Windows: Overlapping windows of varying token sizes (e.g., 16/4 to 256/64).
   • Base Units: Splitting text at natural legal boundaries: Sections (§), Subsections (Absätze), Sentences (Sätze), and Propositions (self-contained legal statements).
2. Advanced/LLM-Intensive Strategies:
   • Lumber-style Chunking: Uses an LLM to dynamically identify semantic boundaries, creating variable-length chunks that may span multiple legal sections.
   • Contextual Chunking: Augments base units with LLM-generated contextual prefixes (hierarchy and full section text) before embedding.
   • Semantic Clustering: Groups semantically similar base units using KMeans, pooling embeddings into single cluster vectors.
   • RAPTOR: A hierarchical retrieval method that recursively clusters and summarizes chunks into a tree structure, routing queries through summary nodes down to leaf nodes.

Evaluation Metrics
The study measures four key dimensions:

• Recall@10: The ability to retrieve the correct top-10 distinct sections matching gold labels.
• Query Latency: Online search and aggregation time (excluding initial embedding).
• Index Build Time: Offline computational cost to construct the index.
• Storage Requirements: Persistent memory footprint of the index and metadata.

Key Results

• Retrieval Performance: Strategies aligned with the inherent legal structure achieved the highest recall. Subsection retrieval performed best (Recall@10 ≈ 0.47), followed closely by Section retrieval (≈ 0.46). Sentence and proposition retrieval formed the next tier. In contrast, methods that override the legal structure performed worse: Lumber-style and Fixed-Size chunking yielded the lowest recall, with the coarsest fixed-size setting (256/64) performing the worst. Semantic clustering and RAPTOR occupied a middle tier, generally underperforming the structural baselines.
• Latency: RAPTOR was the fastest method (approx. 2.0–2.3 ms) due to hierarchical pruning of the search space. Semantic clustering and Lumber were the slowest (13–16 ms) due to extensive post-retrieval consolidation. Among flat baselines, latency scaled as Section ≈ Proposition < Sentence < Subsection, as finer units map to fewer distinct parent sections, reducing aggregation overhead.
• Build Time: Deterministic flat indexes (Fixed-Size, Section) were the fastest (under 1 minute). RAPTOR and Lumber required significantly more time (3–11 hours) due to recursive LLM calls for summarization and boundary detection. Contextual chunking also incurred high costs (38–51 minutes) due to LLM preprocessing.
• Storage: Fixed-Size (256/64) and Semantic Clustering were the most compact (approx. 15 MB) due to vector reduction. Storage increased monotonically for flat baselines as granularity increased (Section < Subsection < Sentence < Proposition). RAPTOR required more space than flat baselines due to the storage of both leaf and summary nodes.

Significance and Claims
The paper demonstrates a critical trade-off between semantic enrichment and operational efficiency. The authors claim that for codified statutory text like the BGB, preserving domain-specific structure is more effective than attempting to semantically enrich or reorganize the text via complex LLM-driven methods.

• Structural Integrity: The deliberate boundaries drawn by lawmakers (sections and subsections) naturally group conditions and consequences. Retrieval at this granularity aligns with both the legislator's intent and the evaluation gold labels, whereas methods that pool provisions based on thematic similarity (Semantic Clustering) or predicted coherence (Lumber) tend to blur specific legal distinctions, reducing recall.
• Efficiency vs. Complexity: Simpler, structure-preserving methods offer superior or comparable retrieval accuracy with significantly lower computational costs (build time, latency, and storage) compared to LLM-intensive techniques like RAPTOR and Lumber.
• Limitations: The study acknowledges that results are specific to the BGB and layperson queries. It does not address cross-reference resolution (explicit or implicit links between sections), which remains a potential avenue for future improvement. The evaluation focuses on retrieval correctness rather than downstream generative performance.

In conclusion, the authors argue that for legal information retrieval on structured statutes, the "simplest" approach—respecting the native hierarchy of the code—is often the most robust and efficient, challenging the assumption that more complex semantic segmentation is always superior.