Technical Summary: Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

Problem Statement

While Retrieval-Augmented Generation (RAG) is widely adopted in agentic systems, existing literature lacks a systematic comparison of how retrieval strategy choices interact with agent architecture and tool-calling paradigms. Current research often evaluates lexical (e.g., grep, BM25) and dense (semantic vector) retrieval in isolation or within fixed pipelines where retrieved documents are simply concatenated into a prompt. This approach ignores the iterative, tool-mediated loops characteristic of modern agents, where models decide what to search, how many queries to issue, and whether results are sufficient.

Furthermore, two critical practical dimensions remain under-explored:

1. Tool Output Presentation: How the delivery method of search results (inline injection vs. file-based reading) affects model performance.
2. Noise Robustness: How retrieval strategies degrade as the ratio of irrelevant surrounding text (distractors) increases relative to relevant material.

Additionally, the emergence of provider-native CLI agents (e.g., Claude Code, Codex, Gemini CLI) introduces a new class of agentic systems that differ fundamentally from custom-built harnesses, yet the effectiveness of retrieval strategies across these architecturally distinct environments has not been studied.

Methodology

The authors conducted an empirical study organized into two experiments using a 116-question subset of the LongMemEval benchmark, which tests agents on long-horizon conversational memory tasks spanning six categories (e.g., knowledge updates, temporal reasoning, preference recall).

Experimental Setup

• Corpus: Localized conversation turns and structured temporal events extracted via the Chronos preprocessing pipeline.
• Retrieval Strategies:
  • Lexical: Grep-based regex/substring matching over raw text.
  • Semantic: Vector search using approximate nearest-neighbor (ANN) with a reranking step.
• Agent Harnesses:
  • Custom: Chronos, a LangChain-based harness with dynamic prompting and fine-grained control over the tool loop.
  • Provider-Native CLIs: Claude Code, Codex, and Gemini CLI, where agents execute bash commands directly.
• Tool-Calling Architectures:
  • Standard (Inline): Search results are returned as tool response messages injected directly into the context window.
  • Programmatic (File-Based): Results are written to disk; the agent must explicitly read or search the file to access content.
• Models: Evaluated across five LLMs (Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite).
• Evaluation: An auxiliary LLM (GPT-4o) acted as a grader to assess answer accuracy against reference answers, with fixed prompts to ensure consistency.

Experiment 1: Retrieval Mode, Harness, and Delivery Method

This experiment isolated the effects of retrieval mode (grep vs. vector), harness type, and tool delivery method (inline vs. file-based) on end-to-end accuracy with the full corpus exposed.

Experiment 2: Context Scaling with Increasing Noise

This experiment tested robustness by progressively increasing the number of irrelevant "distractor" sessions surrounding the relevant "oracle" sessions. The study compared grep-only and vector-only performance across discrete session limits (s5, s10, s20, s30, and full) to observe how retrieval families diverge as noise accumulates.

Key Results

1. Lexical Search Often Outperforms Semantic Search in Agentic Loops

Contrary to the assumption that semantic search is superior for complex reasoning, grep generally yielded higher accuracy than vector retrieval across most configurations in Experiment 1.

• Inline Delivery: Lexical search was uniformly stronger than dense retrieval for every harness-model pair. The largest margin was observed with Chronos and Gemini 3.1 Flash-Lite (86.2% for grep vs. 62.9% for vector).
• File-Based Delivery: The comparison shifted. Programmatic vector retrieval exceeded programmatic grep in five of ten harness-model pairs. However, the transition to file-based delivery caused significant performance drops for some combinations (e.g., Codex with GPT-5.4 dropped from 93.1% with inline grep to 55.2% with programmatic grep), suggesting that the complexity of the file-reading workflow can negate retrieval benefits.

2. Harness Architecture Drives Performance

The choice of harness (custom vs. provider-native) had a profound impact on accuracy, often comparable to or exceeding the impact of swapping retrieval strategies.

• The same underlying model (e.g., Claude Opus 4.6) achieved 93.1% accuracy under the custom Chronos harness but only 76.7% under the Claude Code CLI harness, even when using the same retrieval method.
• This indicates that "retrieval" in agentic systems is effectively "retrieval-plus-orchestration," where system prompts, tool descriptions, and result formatting significantly influence query scheduling and termination.

3. Noise Scaling and Non-Monotonicity

In Experiment 2, retrieval performance did not degrade monotonically as noise increased.

• Crossing Points: The relative performance of grep and vector search varied depending on the session limit and the specific harness. For instance, on the Chronos harness, vector search was often stronger at lower session limits (s5–s20) but grep overtook it at higher limits for certain models. Conversely, on the Gemini CLI, vector search remained dominant across all limits.
• Inductive Biases: The study suggests that provider-specific tooling (e.g., how stdout is chunked, default hints) introduces stable inductive biases that affect whether an agent benefits more from lexical precision or semantic breadth.

Key Contributions

The paper contributes three primary insights:

1. Retrieval, Harness, and Presentation: It provides evidence that the choice between lexical and dense retrieval cannot be evaluated in isolation; it must be considered in conjunction with the agent orchestration layer and the method of surfacing tool outputs (inline vs. file).
2. Noise and Scale: It characterizes how end-to-end behavior evolves as irrelevant content grows, revealing that retrieval families interact differently with noise depending on the agent loop's specific workflow and stopping criteria.
3. Heterogeneity Across Agent Stacks: It demonstrates that retrieval effectiveness is not stable across architecturally distinct harnesses. A strategy that works well in a custom harness may fail in a provider-native CLI, even with an identical underlying corpus.

Significance and Claims

The authors modestly claim that their findings highlight retrieval mechanics, harness orchestration, and delivery paths as a single jointly evaluated system rather than independent design choices.

• Context Engineering: The paper argues that how tool results are presented (inline vs. file) is a critical context engineering decision that can invert retrieval rankings.
• Task Specificity: The authors note that their conclusions are tied to long-memory conversational QA where answers often rely on verbatim spans. They do not claim grep is universally superior, but that it can win end-to-end under specific task distributions and corpora.
• Benchmarking Implications: The study suggests that reporting only retrieval metrics (e.g., BM25 vs. ANN) in static pipelines underestimates the variance introduced by agent scaffolding. Future benchmarks must account for the full agentic loop, including tool delivery and orchestration.

The paper concludes that while lexical search is a strong default for tasks requiring literal span recovery, the optimal strategy is highly contingent on the specific agent stack, the model's capability, and the delivery mechanism of search results.