Technical Summary: Vector RAG vs. LLM-Compiled Wiki

Problem Statement

The paper addresses a gap in quantitative comparisons between two distinct architectures for answering questions over a research corpus: Vector RAG (Retrieval-Augmented Generation) and LLM-Compiled Wikis. While informal commentary (e.g., Andrej Karpathy's "agentic markdown wiki") suggests that compiling research into a cross-linked wiki at ingest time may offer superior multi-paper synthesis and cost amortization, no preregistered, head-to-head quantitative comparison against standard chunk-vector RAG had been published.

The study investigates three specific tradeoffs:

1. Synthesis: Does the wiki architecture better connect findings across multiple papers?
2. Fidelity: Does the process of rewriting papers into wiki pages degrade source fidelity or point-source grounding?
3. Cost: Does the upfront cost of building a wiki result in cheaper query-time costs, creating a break-even point after a certain number of questions?

Methodology

The study is a preregistered, blinded, two-judge comparison conducted on a fixed corpus of 24 peer-reviewed papers across three domains (AI ethics & law, climate science, precision medicine).

• Systems Compared:
  • Vector RAG: A single-round retrieve-then-generate pipeline. It uses document-aware markdown chunking, multi-query expansion, hybrid retrieval (dense+sparse), Cohere reranking, and CRAG-inspired corrective validation. It does not perform retrieval during generation.
  • LLM Wiki: An offline compilation process where an LLM converts papers into a persistent, cross-linked markdown wiki (entities, concepts, sources). At query time, a tool-using agent browses this wiki (listing pages, reading content) to generate answers.
• Evaluation:
  • Questions: 13 evaluation questions across six difficulty tiers (chronological, conflict, multi-hop, emergence, policy, bias-check).
  • Models: Both systems used Claude Opus 4.7 (xhigh) for answer generation.
  • Judging: Answers were scored by GPT-5.4 (primary) and Gemini 2.5 Pro (inter-rater reliability) using a blinded, randomized order.
  • Rubric: Four 1–10 criteria: groundedness (claims trace to source), structural_integrity (unified narrative), conflict_awareness, and inter_paper_mapping (multi-hop synthesis).
• Hypotheses:
  • H1 (Synthesis): Wiki > RAG by ≥2.0 points on synthesis criteria for multi-hop/emergence questions.
  • H2 (Point-Source): RAG ≥ Wiki on groundedness for bias-check questions.
  • H3 (Cost): Wiki ingest cost > RAG ingest cost AND Wiki query cost < RAG query cost.

Key Contributions

1. Preregistered Comparison: The first quantitative, blinded comparison of an LLM-compiled wiki against vector RAG.
2. Evaluation Methodology Findings: Demonstrated that LLM judges behave differently based on rubric concreteness. Judges agreed closely on inter_paper_mapping (concrete definition) but showed significant "ceiling-judge" drift on holistic criteria like structural_integrity, where one judge (Gemini) frequently saturated scores near 10/10 for the wiki.
3. Exploratory Decomposition Ablation: A post-hoc analysis of a Decomposition-RAG variant (breaking questions into sub-questions) to isolate whether the wiki's advantage stems from retrieval coverage or representation alignment.

Results

1. Synthesis and Organization (H1)

• Wiki Advantage: The wiki significantly outperformed single-round RAG on inter_paper_mapping (cross-paper synthesis), clearing the preregistered threshold (+6.625 vs. +2.0).
• Organization: The advantage in structural_integrity was weaker after Inter-Rater Reliability (IRR) adjustment (+1.625), falling just short of the +2.0 threshold.
• Decomposition Effect: A decomposition-based RAG variant recovered ~88% of the wiki's synthesis advantage, reducing the gap to below the registered threshold. This suggests the wiki's synthesis advantage is largely due to better retrieval coverage across documents, which can be mitigated by decomposing queries in RAG.

2. Groundedness and Fidelity (H2)

• Rubric Score: RAG met the preregistered test for single-fact grounding on the bias-check tier (RAG > Wiki), satisfying H2.
• Claim-Level Analysis (Post-hoc): A granular analysis of atomic claims revealed a reversal in mechanism. While RAG scored higher on the holistic rubric, the wiki's cited claims were ~2x more likely to be strictly supported by the cited text and ~4–5x less likely to be unsupported compared to RAG.
• Interpretation: RAG tends to retrieve a chunk and then synthesize/extrapolate beyond it (leading to holistic coherence but lower strict alignment). The wiki pre-positions evidence in "claim-shaped" artifacts, leading to higher citation precision, though this does not guarantee original PDF fidelity (as the wiki itself is a compilation).

3. Cost Asymmetry (H3)

• Query Cost: The expected cost advantage for the wiki failed completely. The wiki consumed 21x more tokens per query than RAG (1.65M vs. 78k tokens).
• Amortization: Because the per-query cost is higher, the "break-even" point for amortizing the ingest cost is mathematically impossible (negative N). The cost intuition that "pre-compilation pays a fee for cheaper queries" was refuted under the tested setup.
• Ingest Cost: The ingest cost ratio could not be adjudicated due to a prompt-caching accounting issue in the telemetry (aggregating cached and uncached tokens), but the query-side result alone refutes the conjunctive H3 hypothesis.

4. Decomposition-RAG Ablation

• Synthesis: Decomp-RAG closed the synthesis gap with the wiki but at a 3.4x higher LLM token cost than single-round RAG (though still 3.4x cheaper than the wiki).
• Citation Alignment: Decomp-RAG did not recover the wiki's advantage in claim-by-claim citation support (19.2% supported vs. Wiki's 40.2%). This indicates that retrieval coverage and representation alignment are separable mechanisms.

Significance and Conclusions

The paper concludes that grounded research synthesis is not a single capability. No single architecture excelled in all three dimensions: organization, citation alignment, and cost.

• Single-Round RAG: Best for cost-sensitive, point-source retrieval.
• Decomposition-RAG: A viable middle ground for multi-paper synthesis where structure matters, offering ~88% of the wiki's synthesis benefit at a fraction of the query cost, though it still lags in strict citation alignment.
• LLM Wiki: Superior for evidence-artifact claim-citation alignment (the cited page directly supports the claim), but at a prohibitive per-query token cost (~21x RAG) and pending validation of source fidelity.

The study emphasizes that evaluations must report synthesis structure, claim-citation alignment, and cost separately rather than collapsing them into a single "groundedness" verdict. The findings also highlight the fragility of LLM-as-judge evaluations on holistic criteria, suggesting that concrete operational definitions are necessary for reliable inter-judge agreement. Future work is needed to validate wiki source fidelity against original PDFs and to explore hybrid architectures that combine iterative retrieval with claim-grounded citation repair.