💬 NLP

Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents

This paper demonstrates that replacing full-corpus injection with structured retrieval methods, particularly LLM navigation over a compact index (NAVINDEX), achieves comparable accuracy to baseline injection on transactional legal document benchmarks while significantly reducing token consumption, context size, and operational costs.

Original authors: Mahmoud Hany, Mourad ElSheraey, Mahmoud Said, Peter Naoum

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahmoud Hany, Mourad ElSheraey, Mahmoud Said, Peter Naoum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a lawyer trying to solve a complex puzzle. You have a massive stack of legal documents (contracts, amendments, side letters) and a very smart but expensive assistant (an AI) who can read them.

The paper asks a simple question: What is the best way to give these documents to the AI so it can answer your questions accurately without wasting money or time?

Here is the breakdown of the three methods they tested, explained with everyday analogies.

The Problem: The "Whole Library" Approach

The Baseline (Inject):
Imagine you have a question about a specific clause in a contract. The simplest way to answer it is to hand the AI the entire library of documents every single time you ask a question.

Pros: The AI sees everything, so it can't miss anything.
Cons: It's incredibly expensive and slow. It's like hiring a librarian to read 500 books to you just to find the price of a single item on page 12 of Book #42. Also, if the library gets too big, the AI gets "confused" and forgets the middle parts of the story.

The Contenders: Two Smarter Ways to Search

The researchers tested two "smart retrieval" methods that try to find only the relevant pages before showing them to the AI.

1. The "Semantic Search" Method (NavEmbed)

The Analogy: The Keyword Index.
Instead of reading the whole book, the AI looks at your question, understands the meaning of the words, and searches a digital index to find the 10 most similar paragraphs.

How it works: It uses "vectors" (mathematical representations of meaning) to guess which pages are relevant.
The Result: This method was very good at saving money. It reduced the amount of text the AI had to read by 17 to 30 times compared to the "Whole Library" method.
The Catch: Sometimes, legal documents rely on specific structures (like "Section A refers to Section B"). Pure meaning-matching sometimes misses these structural links, leading to a few missed answers.

2. The "Structured Navigation" Method (NavIndex)

The Analogy: The Table of Contents with a Map.
This method doesn't just guess based on meaning. It builds a special map of the documents that explicitly notes: "This paragraph defines a term," "This clause refers to that schedule," and "This amendment overrides the original."

How it works: The AI scans this compact map (which is tiny compared to the full text), picks the right "node" (page), and then fetches only that specific page and the pages it explicitly links to.
The Result: This was the champion. It matched the "Whole Library" method perfectly on accuracy (getting 18/18 questions right) but cost 25% less and required the AI to read 56 times less text when formulating the answer.

The "Crossover" Rule: When to Use Which?

The paper discovered a mathematical tipping point for cost, which they call the Caching Crossover.

The Rule: If your document stack is small (less than about 10 times the size of the specific answer you are looking for), it is actually cheaper to just dump the whole stack into the AI's memory every time, because the "discount" for re-reading cached text makes it cheap.
The Reality: Legal deals are usually huge. Once the documents get bigger than that threshold, the "Whole Library" method becomes a money pit.
The Verdict: For small, one-off tasks, dump the whole thing in. For large, complex legal deals, use the "Structured Navigation" (NavIndex) method.

Key Takeaways (The "Lessons Learned")

Don't be greedy: Limiting the AI to a maximum of 10 relevant pages saved money without hurting accuracy.
Structure beats keywords: Mixing standard keyword searches (like Google) with meaning-search actually made things worse and more expensive because it pulled in too much irrelevant text.
What matters most: It didn't matter which "search engine" (embedding model) they used; what mattered was what information was attached to the search results. Giving the AI a map of the document's structure was far more important than the search algorithm itself.

Summary

The paper proves that for analyzing complex legal documents, you don't need to feed the AI the entire book. By building a smart, structured map of the documents and letting the AI navigate that map, you can get perfect accuracy while paying significantly less and processing much less data.

Small documents: Feed the whole thing in (it's cheap enough).
Big documents: Use the smart map (NavIndex) to save money and keep the AI focused.

Technical Summary: Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents

Problem Statement

The paper addresses the challenge of answering questions over a corpus of transactional legal documents (e.g., agreements, amendments, side letters, schedules). The baseline approach, termed inject, involves placing the entire corpus into the Large Language Model's (LLM) context window for every query. While this maximizes retrieval recall by ensuring no information is missed, it suffers from two critical limitations as the corpus grows:

Token Footprint: The cost scales with the corpus size rather than the question, making it inefficient for large document sets.
Long-Context Degradation: LLMs exhibit uneven attention across long inputs, often losing information positioned in the middle of the context window.
Context Window Limits: Legal matters often exceed the maximum context window of current models, rendering full injection impossible.

Existing benchmarks (e.g., CUAD, ContractNLI) typically operate on single documents and fail to capture the cross-instrument dependencies (e.g., defined terms, cross-references, amendments) inherent in transactional legal analysis.

Methodology

The authors evaluated two structured retrieval strategies against the inject baseline using a proprietary, structure-aware chunking method developed at Syntheia. The evaluation was conducted on a benchmark of 20 questions (18 document-bound with verified ground truth, 2 out-of-scope controls) across six legal documents.

Evaluation Protocol

Judge: A position-bias-controlled, reference-anchored pairwise judge (using claude-opus-4-7) compared answers from the baseline and the retrieval method. A "tie" was awarded only if both directions of comparison agreed, certifying that both answers were equally consistent with the ground truth.
Metrics: The study tracked token footprint (total tokens the model attends to, including cached context) and dollar cost (operational spend, accounting for prompt caching discounts). These were treated as separate metrics because caching lowers cost without reducing the token count the model must process.

The Three Modes

Inject (Baseline): The full corpus is placed in the context window for every query.
Navigate-Embeddings (navembed): Documents are pre-embedded into a tree of nodes. At query time, the top- $k$ most similar nodes are retrieved via cosine similarity (with optional cross-encoder reranking) and placed in the context.
Navigate-Index (navindex): A vectorless approach where an LLM navigates a compact, structured index. The index contains:
- Boolean semantic flags: (e.g., isDefinition, hasMoney, hasLiabilityLimit).
- Structural metadata: Headings, clause references, and explicit cross-reference/defined-term graphs.
- Navigation Process: The LLM scans the index (S1) to select up to 10 node IDs based on boolean filters and summaries, then deterministically fetches the verbatim text for those nodes (S2) to generate the answer (S3).

Key Results

Inject vs. navembed

Performance: The strongest configuration (Semantic retrieval + reranking, RUN-012) was judged tied with inject on 16 of 18 document-bound questions (inject preferred on 2).
Efficiency: This configuration reduced the input token footprint by 17.3× (188K tokens vs. 3.26M tokens).
Cost: It achieved a lower dollar cost than inject in all validated runs.
Failure Mode: A hybrid BM25 configuration (RUN-011) suffered from "length bias," inflating the payload and failing to reduce costs effectively.

Inject vs. navindex

Performance: The token-optimized configuration (Config 6) was judged tied with inject on all 18 document-bound questions and both out-of-scope controls. This represents the highest single-run tie rate in the study.
Efficiency:
- Token Footprint: Reduced total input tokens by 1.61×.
- Answering Context: Reduced the context the model attends to during answer generation by ~56× (2.9K tokens vs. 163K tokens).
- Cost: Recorded a 25% lower dollar cost ($1.21 vs. $1.61) compared to inject.
Mechanism: The success of navindex is attributed to its explicit encoding of structural signals (cross-references and defined terms) which cosine similarity alone cannot capture, allowing the model to retrieve the exact context needed without relying on semantic proximity alone.

Cost Analysis: The Caching Crossover Rule

The paper derives a closed-form rule for when cached inject is cheaper in dollars than retrieval:
$C < 10 \times R$
Where $C$ is the corpus size and $R$ is the retrieval payload.

Implication: Cached inject is only cheaper if the corpus is roughly 10 times smaller than the retrieval payload.
Reality: For the 163K-token corpus tested, both navembed and navindex were cheaper than cached inject at steady state. However, for very small corpora (<35K tokens), inject remains the cheaper option.
Caveat: This assumes a "warm" session where the cache is valid (5-minute TTL). In asynchronous workflows, the cache write cost may be incurred repeatedly, eroding the advantage of inject.

Key Contributions

Paired Evaluation: A rigorous, position-bias-controlled evaluation of full-corpus injection against structured retrieval on a legal QA set with verified ground truth.
navindex Design: The design of a navigation descriptor for legal instruments that encodes typed boolean flags, cross-reference graphs, and defined-term resolution, operationalized via specific prompts.
Caching-Crossover Rule: A closed-form economic rule separating token footprint from dollar cost, highlighting that token reduction is a structural benefit while cost savings are session-dependent.
Failure Mode Catalog: Identification of specific failure modes to avoid, such as BM25 length bias, title-only over-selection, and uncapped node selection.

Significance and Claims

The paper claims that structured retrieval is a viable and often superior alternative to full-corpus injection for legal document analysis, provided the retrieval strategy respects the structural dependencies of legal texts.

Quality Preservation: Structured retrieval can match the answer quality of full injection (as measured by a reference-anchored judge) while drastically reducing the token footprint and answering context.
Structural Fidelity: The navindex approach demonstrates that encoding legal structural signals (cross-references, defined terms) directly into the index is more effective than relying solely on vector similarity, particularly for questions requiring the reconciliation of amendments and schedules.
Economic Viability: While full injection is cheaper for very small corpora or single-session queries, structured retrieval becomes the economically superior choice as corpus size grows or when the retrieval payload is significantly smaller than the full corpus.

The authors remain modest about their claims, noting that the evaluation is limited to a single model family, a specific 20-question set, and does not directly measure retrieval recall (precision/recall) due to the lack of ground-truth node labels. They emphasize that the "tie" verdict certifies equivalence in consistency with the ground truth, not absolute correctness, and that the results are specific to the evaluated corpus and prompt engineering.