🤖 AI

Dr-DCI: Scaling Direct Corpus Interaction via Dynamic Workspace Expansion

The paper introduces DR-DCI, a framework that scales Direct Corpus Interaction by dynamically expanding a local workspace via retriever-steered document retrieval, thereby combining the scalability of traditional retrieval with the precision of direct corpus operations to achieve superior accuracy and efficiency across large-scale corpora.

Original authors: Yi Lu, Zhuofeng Li, Ping Nie, Haoxiang Zhang, Yuyu Zhang, Kai Zou, Wenhu Chen, Jimmy Lin, Dongfu Jiang, Yu Zhang

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi Lu, Zhuofeng Li, Ping Nie, Haoxiang Zhang, Yuyu Zhang, Kai Zou, Wenhu Chen, Jimmy Lin, Dongfu Jiang, Yu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery, but instead of a few files on your desk, you have a library containing 10 million books.

The Old Way: The Overwhelmed Detective

Traditionally, when a detective (an AI agent) needs to find a clue in such a massive library, they ask a librarian (a Retriever) to hand them a short list of the "most likely" books. The detective then reads snippets from those books.

The Problem: If the clue is hidden in a weird sentence or requires comparing three different books, the short list isn't enough. The detective is stuck.
The Alternative (Raw DCI): Some detectives try to ignore the librarian and just start shouting commands like "Search the whole library for the word 'dragon'!"
The Result: In a small library, this works great. But in a 10-million-book library, shouting "Search everything!" takes forever, crashes the system, or returns so much junk that the detective gets lost.

The New Solution: DR-DCI (The "Smart Filing Cabinet")

The paper introduces DR-DCI, a new way for the detective to work. Instead of searching the whole library or just reading a short list, the detective uses a dynamic workspace.

Think of the detective's desk as a small, expandable filing cabinet.

The "Pull" Action: When the detective needs a clue, they don't search the whole library. They ask the librarian: "Bring me the top 50 books that might have the answer."
The Workspace: The librarian brings those 50 books and places them on the detective's desk (the workspace). The rest of the 10 million books stay on the shelves, out of the way.
The Investigation: Now, the detective has a manageable pile of papers. They can use powerful tools (like a magnifying glass or a highlighter) to:
- Search within those specific 50 books.
- Compare Book A with Book B.
- Read the fine print.
- Verify if a clue is real.
The Loop: If the detective realizes they need more info, they ask the librarian for another batch of books, which get added to the desk. If they find the answer, they stop.

Why This Works (The Magic Ingredients)

The paper found that this "Smart Filing Cabinet" approach is a winner for three main reasons:

It's Fast and Cheap: Because the detective only searches the small pile on the desk (not the whole library), they finish the job much faster and use less energy (money).
It's Smarter: The detective isn't just reading a list; they are actively comparing and cross-checking the books on their desk. This helps them find answers that a simple list would miss.
It Scales: Whether the library has 100,000 books or 10 million, the detective's desk stays the same size. They only pull in what they need. The old "shout at the whole library" method breaks down completely when the library gets too big, but this method keeps working smoothly.

The "Reset" Button

The paper also mentions a clever safety net. Sometimes, the detective gets confused or follows a wrong lead, but the useful books are still sitting on the desk. Instead of giving up, the system can wipe the detective's memory of the confusion but keep the books on the desk. A fresh, clear-minded detective can then pick up right where the books are and solve the mystery.

The Bottom Line

DR-DCI is like giving a detective a smart, expanding desk instead of forcing them to either read a short list or search the entire library at once. It combines the best of both worlds: the ability to find the right books quickly (Retrieval) and the ability to dig deep into those specific books without getting overwhelmed (Direct Corpus Interaction).

The experiments showed that this method solved difficult questions better than the old ways, worked faster, cost less, and didn't break even when the "library" grew to massive sizes.

Technical Summary: DR-DCI

Problem Statement

Agentic search over large corpora faces a fundamental tension between scale and control. Traditional Retrieval-Augmented Generation (RAG) systems rely on retriever-mediated interfaces (e.g., BM25, ColBERT) to provide ranked lists of documents. While scalable, these interfaces limit agent capabilities by exposing evidence only as snippets or bounded views, hindering the ability to reorganize material, verify constraints across documents, or perform flexible cross-document operations.

Conversely, recent Direct Corpus Interaction (DCI) approaches allow agents to interact with corpora via shell-executable commands (e.g., grep, rg, find). This offers fine-grained control and precision. However, applying DCI directly to massive corpora is inefficient and unstable. As corpus size increases, terminal commands become slow, prone to timeouts, and generate excessive irrelevant output, making global exploration infeasible. The core bottleneck is not local precision, but the lack of a scalable mechanism for focused corpus-level exploration.

Methodology: DR-DCI

The authors introduce DR-DCI (Retriever-Steered Direct Corpus Interaction), a framework that treats retrieval not as a final evidence interface, but as an agent-callable action for expanding a local workspace.

Core Mechanism

Dynamic Workspace Expansion: Instead of operating directly on the full corpus $C$ , the agent maintains an evolving local workspace $W_t \subseteq C$ .
The pull Action: The agent invokes a pull(query, topK) action to materialize ranked candidate documents from the hidden corpus into the workspace. The environment handles retrieval, deduplication, and materialization (via hard links), returning new documents $\Delta W_t$ , a ranked preview $P_t$ , and statistics.
Local DCI Operations: Once documents are materialized, the agent uses terminal-style DCI tools to search, filter, compare, and verify evidence within the bounded workspace $W_t$ .
Separation of Concerns:
- Retrieval handles corpus-level candidate discovery (scaling).
- DCI handles workspace-level document interaction (precision).

Key Interface Components

Inter-document DCI: Commands like rg and grep allow the agent to search across multiple materialized documents to combine constraints and follow bridge entities.
Intra-document DCI: Commands like read with offsets allow for precise span-level verification within a single document.
Ranked Previews: The pull action returns a ranked preview of newly added documents to guide the agent's investigation without replacing the need for local verification.
Workspace-Preserving Context Reset: A recovery mechanism where, if a reasoning trajectory fails (e.g., low confidence), the system discards the unreliable reasoning history but preserves the materialized workspace $W_t$ . A fresh agent instance then attempts to solve the query using the existing evidence.

Implementation Details

Terminal-Aware Interface: Documents are materialized in a root-flat, deduplicated workspace with shell-safe filenames to ensure reliable local operations.
Retriever Backend: The framework is agnostic to the retriever backend, supporting both sparse (BM25) and dense (e.g., Qwen3 8B) retrieval, though dense retrieval showed superior semantic discovery in experiments.

Key Contributions

Retrieval as Workspace Management: The paper recasts retrieval from a one-shot context selection module into an agent-callable operation that modifies the agent's environment. Retrieved documents persist as workspace state, enabling subsequent tool calls to search and verify across materialized candidates.
A Scalable Interface for DCI: DR-DCI instantiates the insight that retrieval should support corpus-level exploration while DCI supports local evidence interaction. This separation allows DCI to preserve precision without requiring every search command to scan the full corpus.
Empirical Validation: The authors validate the approach across three large-scale settings, demonstrating that workspace expansion improves fixed-corpus search and remains effective as corpus size increases.

Experimental Results

1. BrowseComp-Plus (Fixed Corpus)

Performance: DR-DCI achieved 71.2% accuracy on the 830-query BrowseComp-Plus benchmark, an improvement of 8.3 points over Raw-DCI (62.9%).
Efficiency: It reduced average tool usage (30.94 vs. 37.53), wall time (146s vs. 3139s), and estimated cost ($34.91 vs. $88.13).
Context Reset: With the workspace-preserving context reset mechanism, accuracy further improved to 73.25%.
Cost-Accuracy Trade-off: DR-DCI demonstrated a superior cost-accuracy trade-off compared to external systems like o3 and Claude Sonnet 4.6.

2. Controlled Corpus Scaling (BCP-100)

Setup: The corpus was expanded from 100K to 10M documents by adding FineWeb distractors while keeping the same 100 questions and gold evidence.
Results:
- DR-DCI: Accuracy degraded gracefully from ~80% to ~70% as corpus size increased 100x. Tool timeouts and costs remained bounded.
- Raw-DCI: Became unstable and infeasible beyond smaller scales due to repeated full-corpus terminal searches hitting timeouts.
- BM25 Baseline: Remained stable but significantly underperformed DR-DCI because it lacked access to materialized documents for local verification.

3. 20M-Scale Wiki-18 QA

Setup: Evaluated on a 20M-scale file-per-document corpus across six QA datasets (NQ, TriviaQA, etc.).
Results: DR-DCI achieved an average score of 63.0, outperforming retrieval-based and trained search-agent baselines (e.g., R1-Searcher-7B, Search-R1-32B).

Ablation Studies

Dynamic vs. Static: Dynamic Pull (interleaved retrieval) outperformed Single Pull (static workspace) in accuracy and efficiency.
Ranked Previews: Providing ranked previews significantly improved performance (82/100) compared to hidden or shuffled previews, indicating the importance of steering.
Inter-document DCI: Blocking cross-document search caused accuracy to plummet from 82/100 to 40/100, proving that ranked previews alone are insufficient; the ability to compare documents is critical.
Retriever Backend: Both BM25 and Dense retrieval worked, but Dense retrieval yielded the best results (82/100).

Significance and Claims

The paper claims that DR-DCI offers a practical interface for scaling Direct Corpus Interaction to massive corpora while retaining precise local evidence interaction. Its significance lies in resolving the tension between scale and control by:

Decoupling Discovery from Verification: Using retrieval for scalable candidate discovery and DCI for precise verification.
Enabling Robustness: Demonstrating that agents can operate effectively on corpora orders of magnitude larger than those used in standard RAG settings without incurring prohibitive costs or timeouts.
Defining Interface Requirements: Establishing that effective agentic search requires not just retrieval, but specific interface features like ranked previews and inter-document search capabilities to guide agent investigation.

The authors position DR-DCI as a complementary approach to existing agentic search systems, shifting the focus from "how to train agents to search" to "how to design the corpus-access interface itself" to support durable workspace states and scalable interaction loops.