💻 computer science

From Fragments to Paths: Task-Level Context Recovery for Large Industrial Codebases

The paper introduces DeepDiscovery, a two-stage "Location–Inference" framework that effectively recovers task-relevant context from large industrial codebases by moving beyond local fragments, thereby significantly improving file retrieval and coding agent performance on complex software engineering tasks.

Original authors: Jiawei He, Weisong Sun, Mengyu Shi, Jie Jia, Tong Bian, Xikai Yang, Dong Sun

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawei He, Weisong Sun, Mengyu Shi, Jie Jia, Tong Bian, Xikai Yang, Dong Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master detective trying to solve a complex case in a massive, chaotic city (the Large Industrial Codebase). The city has millions of buildings, secret tunnels, hidden wiring, and constantly changing street signs. Your job is to find the specific clues needed to solve a crime (the Task, like fixing a bug or adding a feature).

The problem is that most current detective tools (existing AI methods) are like flashlights that only shine on the immediate street corner. They might find a few relevant documents, but they miss the crucial connections: the hidden wiring between buildings, the owner's manual for a specific machine, or the test logs from three blocks away. Without seeing the whole picture, the detective gets stuck.

This paper introduces a new detective tool called DEEPDISCOVERY. Instead of just shining a light on random spots, it uses a smart, two-step strategy to reconstruct the entire "path" the detective needs to walk to solve the case.

Here is how it works, using simple analogies:

1. The Two-Step Strategy: "Find the Anchor, Then Trace the Path"

Most tools try to search the whole city at once, which is slow and confusing. DEEPDISCOVERY works in two stages:

Stage 1: The Anchor (Location)
Imagine you are looking for a specific person in a stadium. Instead of scanning every seat, you first look for the "high-confidence" entry points: the team logos, the specific section numbers, or the VIP passes mentioned in the ticket.
- In the paper: The system quickly scans the code to find a few "anchors" (key files like a main configuration file or a specific service entry point). It uses clues like file names, folder structures, and coding patterns to guess where the action is, without reading every single line of code yet.
Stage 2: The Path (Inference)
Once the anchors are found, the detective doesn't just stop there. They start walking the "implementation path."
- In the paper: The system expands outward from those anchors. It looks for explicit links (like a door connecting two rooms) and implicit links (like a secret tunnel or a phone call between two people that isn't written in the building blueprints). It follows these connections to find configuration files, tests, and related code that the task actually needs.

2. The "Smart Budget" (Metadata-First)

Imagine you are packing a backpack for a hiking trip, but you only have room for a few items.

Old way: You stuff the whole library into the bag, hoping you find the right map. It's heavy and slow.
DEEPDISCOVERY way: You first look at the spine of the book (the metadata/summary). If the spine tells you the book is about "cooking," you leave it on the shelf. You only open the book and read the pages (load the full text) if the spine suggests it's critical to your hike.
In the paper: The system reads the "summary" of files first. It only loads the full, heavy text of a file if it's absolutely necessary for the task. This saves time and money (computing power).

3. Why It Matters: The "Freshness" Problem

Many existing tools rely on pre-made maps (offline indexes) of the city. But in a fast-growing city, the maps become outdated the moment a new building is constructed.

The Paper's Claim: DEEPDISCOVERY doesn't wait for a new map to be drawn. It explores the city right now, in real-time. This means it works perfectly even when the code is changing every minute, which is common in big companies.

4. The Results: Solving More Cases

The authors tested this new detective tool in three ways:

The Test Drive (Method Level): They gave it 27 medium-sized puzzles. DEEPDISCOVERY found the complete set of necessary clues 92.6% of the time, beating all other tools. It did this without needing to pre-build any maps, making it faster and cheaper.
The Real World (Industrial Systems): They plugged it into six different real-world AI coding assistants used by a large company. In almost every case, the assistants became much better at finding the full set of files needed to fix a problem. The improvement was huge for big, complex projects (up to a 9.2% increase in success).
The Final Exam (End-to-End): They tested it on SWE-bench Verified, a famous benchmark for software engineering tasks.
- The Result: A system using DEEPDISCOVERY solved 78.6% of the tasks.
- The Comparison: The same system without DEEPDISCOVERY only solved 70.4%.
- The Takeaway: By understanding the code better, the AI solved 41 more tasks out of 500.

Summary

Think of DEEPDISCOVERY as a detective who doesn't just read the nearest page of a manual but understands how the whole machine works. It finds the starting point, follows the hidden wires and connections, and only reads the heavy instructions when absolutely necessary.

The paper claims that by doing this, AI coding assistants can stop missing crucial pieces of the puzzle and start solving complex software engineering problems much more reliably, especially in huge, fast-changing codebases.

Technical Summary: From Fragments to Paths: Task-Level Context Recovery for Large Industrial Codebases

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong performance on various software engineering (SE) tasks, reliably understanding large industrial code repositories remains a significant challenge. Existing approaches often rely on retrieving isolated local fragments (e.g., via semantic similarity or static dependency graphs) but fail to recover the broader, task-relevant context necessary for complex, repository-level tasks.

Key challenges identified include:

Fragment vs. Path: Successful task completion often requires reasoning over a complete implementation chain spanning interfaces, business logic, configurations, tests, and cross-module dependencies, rather than just isolated code snippets.
Implicit Dependencies: Many critical links in industrial repositories (e.g., configuration registration, dependency injection, event propagation) are implicitly expressed and missed by standard semantic retrieval or explicit static analysis.
Freshness and Budget Constraints: Many repository-understanding methods rely on pre-built offline indexes (vector databases, static graphs). In fast-evolving industrial environments with frequent commits and module changes, maintaining these artifacts is costly and leads to synchronization lag, degrading retrieval quality and timeliness.

2. Methodology: DEEPDISCOVERY

The authors propose DEEPDISCOVERY, a task-level repository-understanding method designed to recover complete task-relevant context under budget constraints without relying on offline preprocessing. The system operates via a two-stage Location–Inference framework.

A. Repository Representation

The repository is modeled as a multi-relational graph $G = (V, A)$ , where $V$ represents artifacts (source files, configs, tests) and $A$ represents three types of relations:

Explicit Dependencies: Imports, calls, inheritance, references.
Implicit Task-Relevant Links: Configuration-to-code mappings, registration sites, dependency injection wiring, event bindings, and test-to-implementation bridges (extracted via a lightweight rule library).
Organizational Relations: Folder containment, module boundaries, ownership cues, and physical proximity.

B. The Two-Stage Workflow

Stage 1: Location (Task Anchor Localization)

The goal is to identify a small set of high-confidence entry points (anchors) to narrow the search space. This stage involves:

Environment-Aware Analysis: Inferring task-relevant factors (tech stack, module scope) to suppress unrelated areas.
Adaptive Repository Compression: Constructing a compressed view of the repository that adjusts granularity based on scale and budget, preserving structural cues without loading full text.
Rule-Guided Anchor Localization: Ranking candidate entities using a scoring function that combines:
- Semantic similarity (embedding + lexical matching).
- Alignment with the compressed structural view.
- Rule-template matches (naming patterns, directory structures, framework keywords).
- Task-conditioned priors over artifact roles.

Stage 2: Inference (Multi-Relational Implementation-Path Recovery)

Starting from the identified anchors, the system incrementally expands to recover the broader implementation path:

Budget-Aware Expansion: The system selects expansion actions (following explicit/implicit links or organizational proximity) based on a priority score: $Priority = \frac{Gain}{Cost}$ $P r i or i t y = \frac{G ain}{C os t}$ .
- Gain: Estimated improvement in task relevance and path completeness.
- Cost: Token usage and exploration latency.
Candidate Construction & Re-ranking: Selected actions form candidate clusters which are filtered and re-ranked before being added to the context.
Metadata-First Context Construction: To optimize budget, the system chooses between a metadata view (structural summary) and full-text content for each entity. Full text is only loaded if a promotion score indicates it is necessary for downstream reasoning.

3. Key Contributions

DEEPDISCOVERY Framework: A novel method for large industrial codebases that treats repository understanding as task-relevant context recovery rather than fragment retrieval. It utilizes a two-stage Location–Inference process combining anchor localization, multi-relational expansion, and metadata-first construction.
Controlled Method-Level Evaluation: A comparison against five representative baselines (DeepWiki, CodeWiki, RAG, GraphRAG, AST+GraphRAG) on medium-scale tasks, demonstrating superior file recovery quality without offline preprocessing.
System-Level Industrial Evaluation: A systematic evaluation across a production-scale integrated codebase ecosystem (2.67M lines of code, 25,000+ files) and multiple real AI coding systems (Cline, Cursor, Claude Code, etc.), showing consistent improvements in Full Recall Rate.
End-to-End Impact: A controlled evaluation on SWE-bench Verified showing that improved repository understanding directly correlates with higher task Solve Rates.

4. Experimental Results

Method-Level Comparison (Medium-Scale Tasks)

On 27 medium-scale tasks with 135 gold relevant files:

Recovery Quality: DEEPDISCOVERY achieved the highest Full Recall Rate (FRR) of 92.6%, outperforming the best baseline (AST+GraphRAG at 90.4%) and significantly outperforming RAG (84.4%).
Efficiency: DEEPDISCOVERY required 0 hours of offline preprocessing, whereas Graph-based baselines required ~9 hours.
Latency: At task time, DEEPDISCOVERY was significantly faster (13.2s) than RAG-style pipelines (44.8s – 63.8s).
Token Usage: It achieved high recall with fewer execution tokens than AST+GraphRAG, indicating efficiency gains beyond simply exposing more content.

System-Level Evaluation (Industrial Ecosystem)

Evaluated on 27 medium-scale and 40 large-scale tasks across six AI coding systems:

Full Recall Rate: DEEPDISCOVERY improved FRR across all systems.
- Large Subprojects: Absolute gains ranged from 1.6 to 9.2 percentage points.
- Medium Subprojects: Absolute gains ranged from 2.5 to 7.4 percentage points.
Stability: Improvements were consistent across multiple runs and systems, with the most significant gains observed in systems with weaker native repository search capabilities.

End-to-End Evaluation (SWE-bench Verified)

Solve Rate: A system equipped with DEEPDISCOVERY achieved a 78.6% Solve Rate, outperforming the corresponding baseline (70.4%) by 8.2 percentage points (41 additional tasks solved).
Statistical Significance: The improvement was statistically significant ( $p < 0.01$ ) via a paired McNemar test.
Repository Specifics: Gains were most pronounced in repositories requiring cross-file, cross-module, or configuration-sensitive reasoning (e.g., django, xarray, scikit-learn), while localized tasks saw minimal or no gains.

Ablation Study

Removing key components reduced performance:

Location-only: FRR dropped from 85.8% to 80.0%, confirming anchors alone are insufficient.
Without Adaptive Compression: FRR dropped to 81.5%.
Without Implicit Relations: FRR dropped to 83.3%, highlighting the importance of non-explicit links.
Without Metadata-First: FRR dropped to 82.2%, showing the value of selective full-text loading.

5. Significance and Claims

The paper claims that stronger task-level repository understanding is a critical factor in improving coding-agent performance on complex SE tasks.

Shift in Paradigm: The work argues for moving from "fragment retrieval" to "task-level context recovery," acknowledging that industrial tasks require bridging implicit dependencies and configuration artifacts that standard retrieval misses.
Practical Viability: By avoiding offline preprocessing, DEEPDISCOVERY addresses the "freshness" problem in industrial settings, offering a viable solution for rapidly evolving codebases where maintaining static graphs is impractical.
Complementarity: The results suggest that repository understanding and model reasoning are complementary; even strong foundation models benefit significantly from improved context recovery mechanisms.

The authors maintain a modest tone, noting that DEEPDISCOVERY is a "recall-oriented" enhancement that may not benefit all tasks equally (e.g., highly localized tasks) and that failure modes can occur with ambiguous task descriptions or inconsistent naming conventions. They position the work as a significant step toward robust, task-level repository understanding rather than a universal solution for all retrieval scenarios.