Technical Summary: Agent Retrieval Bench

Problem Definition

Modern coding agents are typically evaluated end-to-end based on whether they generate a correct patch. However, patch generation relies on an upstream context-acquisition stage: the agent must first identify and retrieve the specific repository files required to reason about a task. The authors argue that current evaluations obscure this critical layer. Unlike traditional semantic code search, where relevance is defined by query-document similarity, agentic relevance is defined by the "next useful step" in a software workflow. This relevance is often indirect, structural, or workflow-dependent (e.g., a pull request describing an implementation change may require a regression test file, not the implementation file itself).

Interactive exploration allows agents to compensate for weak initial context through additional tool calls, but this increases cost and latency. Furthermore, logged trajectories show that even interactive agents fail to touch any "gold" (necessary) file in 27–35% of cases. The paper posits that Agent Retrieval Bench is needed to isolate and evaluate this upstream retrieval problem under realistic constraints, distinct from the final patch-generation outcome.

Methodology

Benchmark Construction

Agent Retrieval Bench is a file-level code retrieval benchmark constructed from 427 samples across 25 repositories (spanning Python, Go, Rust, TypeScript, Java, and JavaScript).

• Frozen Corpus: All samples are evaluated against a repository state frozen at a base commit (pre-resolution), preventing leakage from fix commits or final patches.
• Query Sources: Queries are derived from real workflow signals: PR titles/bodies, review comments, reproduced failure traces, and anchored edits.
• Leakage Controls: The dataset rigorously removes fatal shortcuts, including exact gold paths, raw diffs, fix commit hashes, and model-generated review artifacts.

Task Definitions

The benchmark defines five distinct retrieval scenarios:

1. code2test (106 samples): Given a PR or implementation change, retrieve related test files.
2. comment2context (80 samples): Given a review comment and the reviewed file, retrieve additional context files needed to satisfy the comment.
3. trace2code (101 samples): Given a reproduced failure command and output, retrieve the root-cause source file (tests are auxiliary, not gold).
4. edit2ripple (58 samples): Given an anchored edit and change intent, retrieve additional files affected by the change (ripple files).
5. Selective Retrieval (82 samples):
   • Natural No-Gold (50): Issues where the resolution lies outside the repository (e.g., upstream dependencies).
   • Counterfactual Controls (32): Queries paired with a repository that cannot contain the answer.

Evaluation Metrics

• Ranking Metrics: Recall@k (5, 10, 20), Mean Reciprocal Rank (MRR).
• Context Budget Metrics: Budgeted Context Yield (BCY@B), which measures how much labeled gold context fits within a fixed token budget (e.g., 8k tokens) after ranking and greedy packing.
• Selective Metrics: Balanced accuracy and selective success for abstention tasks.
• Trajectory Analysis: Logged agent trajectories are analyzed for "gold-touch" rates, first-hit steps, and Potential Exploration Savings (PES).
• Seed Intervention: A controlled pilot (45 samples) where agents are seeded with different initial contexts (random, lexical, embedding, hybrid, oracle) to measure the impact on context acquisition efficiency.

Key Contributions

1. Formalization of Agentic Relevance: The paper defines agentic relevance not as semantic similarity but as the "next repository context needed," operationalized through four positive tasks and selective abstention.
2. Leakage-Controlled Dataset: A validated set of 427 samples with strict schema checks, zero fatal shortcut leakage, and evidence-backed gold labels derived from workflow artifacts.
3. Agent-Oriented Evaluation Suite: A comprehensive suite including ranked-file metrics, token-budget BCY curves, selective abstention, and a controlled-input seed intervention pilot.
4. Empirical Findings on Retrieval Families: Evidence that no single retrieval family dominates across all workflow signals or context budgets.

Results

Baseline Performance

The benchmark evaluated lexical (BM25, path-aware lexical), vectorless structural (RepoMap), and embedding-based (Qwen3-4B, Qwen3-8B, Jina, Nomic, pplx) baselines.

• Overall Leaders: Qwen3-Embedding-4B achieved the best sample-weighted MRR (0.2379), while Qwen3-Embedding-8B achieved the best Recall@20 (0.7029). RepoMap led in BCY@8k (0.3788).
• Task-Specific Winners: Performance varies sharply by task.
  • trace2code: RepoMap significantly outperformed embeddings (MRR 0.2742 vs. 0.0827 for Qwen3-4B), demonstrating that structural proximity is crucial for root-cause localization from failure traces.
  • edit2ripple: A budget-dependent reversal was observed; Qwen3-4B led at 4k tokens, while Qwen3-8B led at 8k tokens.
• Hybrid Approach: A simple Reciprocal Rank Fusion (RRF) of Qwen3-8B and RepoMap improved overall MRR to 0.2713 and Recall@20 to 0.7331, confirming the complementarity of semantic and structural signals.

Selective Retrieval & Abstention

• Calibration Gap: While raw retrieval scores could distinguish counterfactual wrong-repository controls, they failed to improve selective success on natural no-gold cases. Thresholds that maximized balanced accuracy often rejected too many positive samples, indicating that current scores are insufficient for distinguishing plausible local issues from genuinely external resolutions.

Agent Trajectories & Seed Intervention

• Miss Rates: Even with interactive exploration, logged agents failed to touch any gold file in 35.2% of OpenAI samples and 27–29% of Codex samples.
• Seed Efficiency: In the closed-tool seed intervention, retrieval-derived seeds (lexical, RRF) achieved higher File F1 and earlier gold contact with fewer post-seed tool calls and tokens compared to random non-gold seeds.
• Oracle Headroom: Oracle gold context improved File F1 by 0.3115 over the no-seed baseline, suggesting substantial remaining headroom for localization even when the correct file is provided.

Granularity Sensitivity

• File vs. Span: While file-level retrieval is the primary metric, the paper notes that gold files are large (median 515 lines) and labeled evidence often occupies only ~4.7% of the file. File-level hits do not guarantee precise within-file localization, and line-level F1 scores remain low across all methods.

Significance and Claims

The paper claims Agent Retrieval Bench provides a unified evaluation semantics and hygiene standard for the context-acquisition layer of coding agents. Its primary significance lies in:

1. Isolating Upstream Failure: It demonstrates that patch failure can be diagnosed at the retrieval stage, separate from reasoning or editing capabilities.
2. Challenging "One-Size-Fits-All": The results show that semantic embeddings, lexical matching, and structural graphs are complementary, not interchangeable. A single retriever cannot dominate all workflow signals.
3. Hybrid Necessity: The paper concludes that effective coding-agent retrieval must be hybrid and task-aware, combining semantic vectors, lexical/path signals, and repository graphs.
4. Diagnostic Utility: The benchmark serves as a diagnostic tool to measure context acquisition efficiency (CAE) and identify calibration gaps in selective retrieval, rather than claiming to predict end-to-end patch success directly.

The authors explicitly state they do not claim that retrieval is the sole cause of patch failure or that File F1 substitutes for test-passing repair. Instead, the benchmark establishes context acquisition as a measurable, distinct failure surface and cost center in the agentic workflow.