Technical Summary: GR EPSEEK: TRAINING SEARCH AGENTS FOR DIRECT CORPUS INTERACTION

Problem Statement

Large Language Model (LLM) search agents typically rely on pre-computed retrieval indices (using sparse or dense embeddings) to fetch ranked lists of documents before reasoning. This paradigm restricts information seeking to fixed text chunks and relevance scores, often failing in scenarios requiring exact entity matching, symbolic pattern search, or multi-hop reasoning where semantic embeddings cause "conflation" of similar entities. Furthermore, contemporary approaches that bypass indices to search raw corpora via shell commands (Direct Corpus Interaction, or DCI) often rely on prompting large, proprietary models at inference time, resulting in high computational costs and operational inefficiencies (e.g., query times exceeding an hour).

This work addresses the challenge of training compact, open-weight models to perform Direct Corpus Interaction (DCI) efficiently. The goal is to enable search agents to treat the corpus as an executable environment, issuing shell commands (e.g., grep, rg) to find, filter, and compose evidence directly from raw text, while overcoming the instability of Reinforcement Learning (RL) on large corpora and the latency bottlenecks of sequential shell execution.

Methodology

The authors introduce GrepSeek, a DCI search agent trained via a two-stage pipeline and supported by a specialized execution engine.

1. Two-Stage Training Pipeline

To stabilize learning and prevent degenerate behaviors (such as overly broad commands or excessive context retrieval), the authors propose a cold-start strategy followed by RL optimization:

• Cold-Start Data Generation:

  • Backward Phase (Tutor): An answer-aware Tutor LLM decomposes a question and gold answer into a chain of sub-queries. It constructs a retrieval trajectory in reverse (from answer to question), proposing shell commands that retrieve documents entailing the current target. Crucially, the Tutor enforces an answer-leak rule, forbidding the use of the target entity or its aliases in the search command to ensure the agent learns genuine information-seeking behavior rather than exploiting known answers.
  • Forward Phase (Planner): An answer-blind Planner LLM generates reasoning traces and commands based solely on the observable history, simulating inference-time behavior.
  • Alignment: The Tutor edits the Planner's reasoning to logically motivate the verified backward commands while remaining strictly grounded in the causal history.
  • Quality Filtering: Trajectories are filtered to ensure they contain sufficient information for answer generation and pass a coherence check to prevent future-state leakage.

• Optimization:

  • Supervised Fine-Tuning (SFT): The agent is first fine-tuned on the synthetic cold-start dataset to learn stable, causally grounded search commands.
  • Group Relative Policy Optimization (GRPO): The policy is further refined using GRPO. For each query, a group of trajectories is sampled, and rewards are computed based on token-level F1 overlap with the gold answer, normalized within the group. A binary format reward ensures structural validity (proper XML tags for reasoning and tool use).

2. Efficient Execution Engine

To make DCI practical at scale over multi-gigabyte corpora, the authors developed a semantics-preserving sharded-parallel execution engine:

• Sharding: The corpus is partitioned into line-aligned shards.
• Parallel Classification: Shell pipelines are dynamically classified. Stateless pipelines (e.g., rg, grep, cut, tr) are executed in parallel across shards. Pipelines with global state or cross-line dependencies fall back to sequential execution.
• Reduction Strategies: The engine aggregates partial outputs using deterministic rules (e.g., concatenation for stateless, local head followed by global head for top-N, scalar summation for counts, and k-way merge for sorted outputs) to ensure byte-exact equivalence with sequential execution.
• Persistent Daemon: A long-running daemon keeps the corpus in memory (or memory-mapped), eliminating process startup overhead for repeated tool calls.

Key Contributions

1. GrepSeek Agent: A compact, trained search agent that learns to interact directly with raw text corpora via executable shell commands, shifting DCI from an inference-time prompting strategy to a learned capability.
2. Stable Training Pipeline: A novel two-stage approach combining backward-verified cold-start data generation with GRPO, effectively stabilizing RL training on large corpora where naive application fails.
3. Semantics-Preserving Parallel Engine: An execution engine that accelerates shell-based retrieval by up to 7.6× (reducing latency from 5.39s to 0.71s) while maintaining strict byte-exact equivalence with sequential execution, making DCI viable for interactive agents.
4. Comprehensive Evaluation: Extensive experiments across seven open-domain QA benchmarks (including single-hop and multi-hop datasets) demonstrating that GrepSeek outperforms index-based RAG and other agentic baselines.

Results

Experiments were conducted on seven benchmarks: Natural Questions (NQ), TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle, using a 21M document Wikipedia corpus.

• Performance: GrepSeek achieved the strongest overall token-level F1 score (0.5691) and Exact Match (0.4948), significantly outperforming the best dense retrieval baseline (Search-R1 with Qwen3-4B, 0.5441 F1) and other agentic frameworks.
• Multi-Hop Reasoning: The model showed particularly strong gains on multi-hop datasets (e.g., +0.0631 F1 on HotpotQA, +0.0879 on 2WikiMultihopQA), where it successfully isolated rare symbolic patterns and exact entity names that dense retrievers often conflate.
• Efficiency: While GrepSeek has higher end-to-end inference latency (8.67s) compared to dense retrieval (4.77s) due to longer reasoning trajectories, it eliminates the need for offline indexing (saving 3.2–62.4 A100-hours) and reduces memory footprint to the raw corpus size (14GB vs. 70–221GB for dense indices).
• Ablation: Removing either the SFT stage or the GRPO optimization led to severe performance degradation, confirming the necessity of both structured initialization and policy refinement.

Significance and Claims

The paper claims that Direct Corpus Interaction (DCI) is a practical and competitive alternative to traditional index-based retrieval for search agents. By treating the corpus as an executable environment, agents can perform "surgical" information retrieval that preserves fine-grained lexical distinctions, handles exact entity matching, and avoids the semantic conflation common in dense embedding spaces.

The authors highlight that GrepSeek demonstrates that compact models can learn complex search behaviors through RL when provided with a stable, causally grounded initialization. While the approach has limitations regarding surface-form variation (e.g., diacritics) and lacks semantic ranking, the authors argue that the precision, scalability, and elimination of expensive indexing make DCI a valuable paradigm for knowledge-intensive tasks, particularly those requiring multi-step reasoning and exact evidence composition. The work establishes a foundation for future hybrid architectures combining direct corpus interaction with index-based methods.