Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation

Technical Summary: Lost in a Single Vector (DICE)

Problem Statement: Document-Side Early Compression
The paper identifies a specific failure mode in dense retrieval for long documents, termed document-side early compression. In standard dense retrieval, a document is encoded into a single vector before any query comparison occurs. When a document is long (e.g., thousands of tokens) but the decisive evidence is confined to a short, local span, the encoding process compresses this critical information together with a large volume of irrelevant context. Consequently, the resulting document vector may be "diluted," causing the document to receive a low similarity score despite containing the answer. This degradation occurs even when modern encoders support large context windows (4k–32k tokens), suggesting the issue is representational rather than a capacity limitation.

Methodology: DICE (Document Inference via Chunk Evidence)
To address this without altering the standard retrieval interface (one query vector vs. one document vector), the authors propose DICE, a training-free, document-side strategy. The method operates as follows:

1. Chunking: The long document is split into smaller, non-overlapping chunks (default size: 1024 tokens) in token space.
2. Local Encoding: Each chunk is encoded independently by a frozen encoder. Crucially, local position indices are used, resetting the position embedding to zero for each chunk rather than inheriting the absolute position from the original document. This ensures each chunk is processed as a self-contained context window.
3. Query-Independent Aggregation: The resulting chunk embeddings are aggregated back into a single document vector using a query-independent pooling function. The authors evaluate mean pooling, max pooling, and top-k pooling (based on embedding norm). The default and most effective strategy is mean pooling, which preserves evidence from all chunks equally.
4. Retrieval: The aggregated vector is used in the standard cosine similarity ranking against the query vector. The query encoding and retrieval interface remain unchanged.

Key Contributions

• Diagnostic Metric (EDI): The authors introduce the Evidence Dilution Index (EDI) to quantify the failure of single-vector encoding. EDI measures the gap between the document-level similarity score and the strongest chunk-level similarity within the same gold document. A high EDI indicates that the document vector has fallen significantly below the strongest local evidence due to compression.
• Training-Free Remedy: DICE offers a practical solution that improves retrieval performance without requiring model retraining or changes to the query encoder.
• Empirical Validation: The study demonstrates that document-level encoding is an underexplored lever for improving long-document retrieval, distinct from changing the retrieval unit (e.g., passage retrieval) or the scoring mechanism (e.g., late interaction).

Experimental Results
The method was evaluated on the LongEmbed benchmark across four backbone architectures (Dream, Mistral, Llama3, Qwen) and on the FollowIR benchmark.

• Performance Gains: DICE consistently outperforms single-vector baselines, with the most significant improvements occurring on long-context slices (>4k tokens).
  • On the Dream backbone, Passkey retrieval (>4k) improved from 30.0% to 90.0%, and Needle-in-a-Haystack (>4k) improved from 23.3% to 74.0%.
  • Similar substantial gains were observed across Mistral, Qwen, and Llama3, particularly on synthetic tasks and real-world tasks like NarrativeQA.
• Ablation Insights:
  • Chunk Granularity: Larger chunk sizes (up to 1024 tokens) yielded better results than smaller ones (e.g., 128 tokens), suggesting that overly small chunks fail to preserve sufficient local context.
  • Aggregation Strategy: Simple mean pooling proved most robust. Max pooling and top-k pooling often degraded performance on real-world tasks, indicating that conservative aggregation is superior to selective chunk filtering in this setting.
  • Position Encoding: Resetting positions locally within each chunk was found to be superior to preserving absolute document offsets.
• EDI Analysis: Across 12,779 filtered samples, DICE achieved a lower EDI (indicating less evidence dilution) than the single-vector baseline in 92.8% of cases. The improvement was most pronounced when evidence was highly localized (low Evidence Concentration).
• Cost Trade-off: DICE increases document-side encoding time by approximately 3–4× due to processing multiple chunks. The authors note this is acceptable for offline indexing scenarios where documents are indexed once and queried repeatedly.

Significance and Claims
The paper claims to establish document-side encoding as a practical and underexplored degree of freedom for long-document retrieval. By framing the problem as "early compression" and providing a training-free method to delay this compression until after local evidence is captured, DICE improves retrieval fidelity while maintaining the scalability and simplicity of the standard single-vector interface. The authors emphasize that the gains stem from preserving sufficient local context before aggregation, rather than from sophisticated chunk selection mechanisms.

Limitations
The authors acknowledge that DICE incurs higher encoding costs, relies on query-independent aggregation (foregoing dynamic emphasis of relevant chunks), and still compresses the document into a single vector, meaning it cannot match the granularity of multi-vector or passage-level retrieval methods.