🤖 AI

ScalableRAG: High-Quality RAG at Zero Ingestion Cost

The paper introduces ScalableRAG, a high-quality Retrieval-Augmented Generation approach that achieves superior accuracy across multiple datasets with zero ingestion costs by utilizing a dynamic workspace for on-the-fly aggregative reasoning, while also offering a Limited-Ingestion variant that further enhances performance at scale with minimal vector database usage.

Original authors: Hilaf Hasson, Aditya Chakravarty, Jayant Thomas, Krishna Gogineni

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hilaf Hasson, Aditya Chakravarty, Jayant Thomas, Krishna Gogineni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive mystery by reading a library of a million books. In the world of artificial intelligence, this is called Retrieval-Augmented Generation (RAG). Think of a standard AI as a brilliant detective who has read the entire internet but can't remember specific details from a dusty archive. To help the detective, we usually build a "card catalog" or a "knowledge graph" before they start working. This involves reading every single book, summarizing it, and organizing it into a complex database so the AI can find the right page quickly. It's like hiring a team of librarians to spend weeks indexing the library before the detective even arrives. This process is expensive, slow, and requires a lot of computing power just to get started. But what if the detective could walk into the library, look at the books, and figure out the answers on the fly without needing a pre-made catalog? That is the big question this paper tackles: Can we get the same brilliant results without paying the high "ingestion" cost of building that massive database first?

The researchers at Cohesity, led by Hilaf Hasson and their team, say the answer is a resounding "yes." They introduce a new system called ScalableRAG, which comes in two flavors: Zero-Ingestion and Limited-Ingestion. Their main discovery is that you don't need to pre-process the entire library to answer complex questions. Instead, their AI agent acts like a super-organized detective who carries a magical, expandable clipboard. When the detective needs to find a specific fact, they don't just search for a keyword; they create a temporary "group" of relevant documents, filter them, and even do math on the results right there in the moment.

Here is how their "Zero-Ingestion" method works, using a simple metaphor: Imagine you have a pile of mixed-up receipts from a year of shopping. A traditional system would require someone to sit down, read every receipt, and sort them into labeled folders (Groceries, Gas, Electronics) before you ask a question like, "How much did I spend on gas in June?" This sorting takes forever. ScalableRAG, however, skips the sorting entirely. When you ask the question, the AI looks at the pile, finds all the receipts that mention "gas," then filters that smaller pile to find only the ones from "June," and finally adds up the numbers. It does this by creating and managing "sets" of documents on the fly. It's like having a detective who can instantly draw a circle around the relevant receipts, count them, and do the math, all without ever needing to file them away first.

The paper shows that this "on-the-fly" approach is incredibly powerful. In tests across six different collections of documents (ranging from government hearing transcripts to financial reports and movie scripts), their Zero-Ingestion system handily out-performed all baselines (including knowledge graph approaches) in three out of the six datasets (MuSiQue, Transcripts, and Hotels). On the other three datasets (2Wiki, FinanceBench, and ComplexTR), it only marginally missed maximum performance, losing by an average of just 1.63% compared to the best-performing baseline for each (which was A-RAG). Despite these narrow losses on half the datasets, the system achieved an average accuracy across all six datasets that was about 7.36% higher than the next most competitive baseline. This is a big deal because it suggests that for many types of questions, the expensive step of pre-building a database is actually unnecessary.

To make the system even better for trickier questions, they also built a "Limited-Ingestion" version. This is like giving the detective a small, pre-made index for the most common types of clues, but still letting them do the heavy lifting on the fly. This version uses a tiny bit of pre-processing (just a sample of the documents to find patterns) and a small vector database (a tool that helps find similar ideas, not just exact words). This hybrid approach pushed the accuracy even higher, especially in datasets where the information was hidden in complex structures, like finding a specific facility in a hotel description or a specific clause in a legal contract.

The paper explicitly argues against the idea that you must build a massive, pre-processed knowledge graph or a full SQL database to answer complex, multi-step questions. They show that the "ingestion-heavy" methods, which require an LLM to read every document and extract data before the user even asks a question, are often overkill. While those methods are good at grouping data, ScalableRAG proves that an agent can do the same grouping and math instantly during the conversation. The authors are very confident in these results, having measured them across diverse datasets with a rigorous testing method where a separate AI acts as a judge to grade the answers. They found that their system not only saves money and time but often gets the right answer more often than the expensive alternatives, or comes very close with significantly less effort.

In short, ScalableRAG changes the game by showing that you don't need to build a giant warehouse of organized knowledge to find the needle in the haystack. Instead, you can send a smart agent into the haystack with a set of magical tools to grab, sort, and count the needles right where they are. Whether you need a system that costs nothing to set up (Zero-Ingestion) or one that adds a tiny bit of pre-work for extra precision (Limited-Ingestion), this research suggests that the future of AI question-answering might be much simpler, faster, and cheaper than we thought.

Technical Summary: ScalableRAG

Problem Statement

Recent advancements in Retrieval-Augmented Generation (RAG) have increasingly focused on optimizing performance by incurring high ingestion costs. These methods typically involve building complex knowledge structures, such as knowledge graphs (e.g., GraphRAG, HippoRAG) or extracting structured SQL tables (e.g., SRAG, AutoSchemaKG), which require at least one Large Language Model (LLM) call per document, often significantly more. While these approaches enable sophisticated aggregative reasoning (e.g., grouping by keys to compute counts or averages), they are computationally expensive and difficult to scale. Conversely, standard dense retrieval RAG systems often struggle with aggregative questions because they lack persistent state or structured indexing.

The authors posit that many questions requiring aggregation rely on a primary key that is in one-to-one correspondence with a subset of the document corpus. They argue that such grouping operations can be replicated efficiently at inference time without the need for heavy pre-processing or vector databases.

Methodology

The paper introduces ScalableRAG, a stateful agentic RAG framework designed to achieve high-quality retrieval and reasoning with minimal to zero ingestion costs. The system operates through two variants:

1. Zero-Ingestion ScalableRAG

This variant requires no preprocessing whatsoever—no knowledge graphs, no schema extraction, and no vector embeddings.

Stateful Workspace: The agent maintains a persistent registry of document sets ( $S$ $S$ ) and value sets ( $V$ $V$ ).
- Document Sets: Named subsets of the corpus (e.g., has_Erik_pos) created via filtering operations.
- Value Sets: Mappings from document IDs to extracted scalar values (numbers, dates) or lists of strings.
Tool-Based Interaction: The agent interacts with the corpus using a suite of deterministic tools:
- apply_filter: Applies regex to partition a named set into positive and negative subsets.
- search_filenames: Filters documents based on filename patterns.
- set_operation: Performs intersection, union, and difference on named sets.
- extract_field / extract_list: Extracts values or lists from documents into value sets.
- aggregate: Computes statistics (sum, avg, count, min, max) over value sets.
- read_docs / find_in_doc: Reads specific document sections or navigates within large files.
Diagnostic Feedback: A core innovation is the enrichment of tool observations with automatically computed diagnostics. When a filter is applied, the system provides:
- Token decomposition: Which keywords in the regex are too strict or too broad.
- Selectivity gaps: Alerts if the combined pattern over-constrains the set compared to individual keywords.
- Negative anchor checks: Identifies if excluded documents still contain key question entities, prompting the agent to refine its regex.
Reasoning Loop: The agent uses a ReAct-style loop where it plans actions, executes tools, observes results (including diagnostics), and iteratively refines its document sets until it can answer the query.

2. Limited-Ingestion ScalableRAG

To further improve accuracy on datasets with complex structures, this variant adds two optional, low-cost ingestion modules:

Vector Embeddings: Pre-computes embeddings for large document chunks to enable cosine_search. This serves as a fallback for semantic retrieval when lexical matching fails. It also supports training a lightweight classifier on a small sample of labeled documents to classify the rest of the corpus.
Automatic Pattern Discovery: A preprocessing step that analyzes a sample of documents to discover extraction patterns:
1. Deterministic Regex: Scans for recurring "Label: Value" structures.
2. LLM-Generated Regex: Uses an LLM to propose multi-line structural patterns.
3. Semantic Patterns: Identifies prose-embedded concepts.
  These patterns are validated, scored for utility, and exposed to the agent as a pattern_search tool, allowing it to leverage pre-discovered structures without full corpus indexing.

Key Contributions

Zero-Ingestion ScalableRAG: A RAG system that achieves state-of-the-art performance without any ingestion costs (no vector DB, no knowledge graph). It utilizes inference-time set persistence and set-algebraic reasoning to handle aggregative tasks.
Limited-Ingestion ScalableRAG: An extension that incorporates minimal vector embeddings and a one-time, sample-based pattern discovery process to further boost accuracy on structurally complex datasets while maintaining a constant number of LLM calls relative to corpus size.
Diagnostic Feedback Mechanism: The introduction of automated, relative diagnostics (e.g., scope checks, selectivity gaps) that guide the agent's planning and regex refinement, significantly reducing the need for manual tuning or external validation.

Experimental Results

The authors evaluated ScalableRAG on six diverse corpora: MuSiQue, 2WikiMultiHopQA, Transcripts (US Congressional hearings), FinanceBench, ComplexTR, and Hotels.

Performance:
- Zero-Ingestion ScalableRAG outperformed all baselines (including heavy-ingestion methods like HippoRAG2, SRAG, GraphRAG, and AutoSchemaKG) in 3 out of 6 datasets (MuSiQue, Transcripts, Hotels).
- On the remaining three datasets, it missed the maximum performance by an average of only 1.63%.
- Across all six datasets, the average accuracy was 7.36% higher than the next most competitive baseline.
- Notably, on the Hotels dataset (challenging for aggregative reasoning), it outperformed SRAG by 36.23% with zero ingestion.
Efficiency:
- ScalableRAG uses a similar number of steps per question as agentic baselines (e.g., A-RAG) but consumes significantly fewer tokens during inference.
- The Limited-Ingestion variant provided additional accuracy lifts on datasets with rich structure (Hotels, Transcripts) where pattern discovery was effective.

Significance and Claims

The paper claims that ScalableRAG demonstrates that the benefits of heavy knowledge ingestion (specifically the ability to perform aggregative reasoning and group by primary keys) can be replicated with zero or minimal ingestion costs.

Cost Efficiency: By avoiding the "ingestion tax" of one LLM call per document, ScalableRAG offers a scalable alternative that is particularly effective for corpora where the primary key of interest corresponds to a subset of documents.
Stateful Reasoning: The work challenges the paradigm of stateless retrieval, showing that maintaining a workspace of document and value sets allows for iterative refinement and complex multi-hop reasoning that standard vector retrieval cannot achieve.
Practicality: The system achieves high accuracy without the need for pre-computed vector databases or complex schema induction, making it a viable solution for scenarios where preprocessing resources are constrained.

The authors conclude that while knowledge graph approaches may still excel for questions violating the one-to-one correspondence hypothesis, ScalableRAG provides a highly competitive, cost-effective framework for a broad range of aggregative and multi-hop question-answering tasks.