💻 computer science

ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation

ProjAgent is a repository-level code generation system that enhances performance by retrieving functions based on procedural similarity through an agentic workflow and refining outputs via static-analysis feedback, achieving a 41.14% Pass@1 on the REPOCOD benchmark.

Original authors: QiHong Chen, Aaron Imani, Iftekhar Ahmed

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: QiHong Chen, Aaron Imani, Iftekhar Ahmed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to write a new chapter for a massive, 10,000-page mystery novel. You know the plot point you need to hit, but you don't know the characters' names, the specific rules of the town, or how the detective usually solves crimes. If you just guess, you might invent a character who doesn't exist or break the town's laws, ruining the story. This is exactly what happens when Artificial Intelligence tries to write code for a big software project.

For a long time, AI assistants tried to find help by looking for "look-alikes." If you asked for a function to calculate temperature, the AI would search for other code that used the word "temperature" or looked similar on the surface. The paper argues that this is like trying to find a recipe for a cake just because the other recipe also mentions "sugar." It misses the real magic: how the cooking actually happens.

The authors, Qihong Chen, Aaron Imani, and Iftekhar Ahmed, built a new system called ProjAgent. They discovered that the most helpful code isn't always the code that looks the same; it's the code that thinks the same way.

The "Thinking" vs. The "Talking"

The paper suggests that code has two layers: the words it uses (like variable names and comments) and the actual steps it takes to solve a problem (the logic).

Imagine two chefs. Chef A is making a pizza, and Chef B is making a salad.

The Old Way (Lexical/Semantic Search): If you ask for help making a pizza, the old AI would ignore Chef B because they are making a salad. It would only look for other pizza chefs.
The New Way (ProjAgent): ProjAgent realizes that both chefs have to do the same thing: chop ingredients, check if they are fresh, and put them in a bowl. Even though one is making a salad and the other a pizza, their procedural steps are identical. ProjAgent finds Chef B and says, "Hey, look at how Chef B checks if the lettuce is fresh! You can use that same logic to check if your tomato is fresh!"

The paper explicitly rules out the idea that surface-level similarity (looking at the words) is enough. They argue that relying only on words or simple meaning often misses the critical "how-to" steps because different projects use different names for the same actions.

How ProjAgent Works: The Detective Agent

ProjAgent acts like a super-smart detective with a special tool. Here is how it solves the mystery:

Breaking it Down: It takes the new code you want to write and breaks it into tiny logical steps, like "check if the number is negative" or "convert the units."
The "Thinking" Search: Instead of just reading the words, ProjAgent uses a special trick to look at the "hidden thoughts" of the AI model. It strips away the fancy vocabulary and looks only at the pure logic. It finds other code in the project that does the exact same mental dance, even if it's written in a completely different part of the codebase.
The Agent's Hunt: If the first search doesn't find enough, a digital "agent" (a little AI robot) goes on a scavenger hunt through the project files. It reads files, searches for functions, and asks, "Is there anything here that thinks like this?"
Double-Checking: To make sure it didn't get tricked, the system uses a "static analysis" tool. Think of this as a spell-checker for logic. It checks if the code actually runs without crashing. If the AI makes a mistake, the system says, "Whoops, that won't work," and asks the AI to try again. This loop happens up to 10 times until the code is solid.

The Results: Did It Work?

The authors tested ProjAgent on a benchmark called REPOCOD, which contains 980 real-world coding problems from 11 different software projects.

The Score: ProjAgent got a 41.14% Pass@1 score. This means that on the first try, it wrote working code for about 41 out of every 100 problems.
The Comparison: This beat the previous best methods. The old "dense search" methods only got 28.83%, and the "sparse search" (looking for keywords) got 26.58%. Even a top-tier system called SpecAgent only reached 34.52%.
The Proof: The paper suggests that the biggest boost came from finding those "procedurally similar" steps. When they removed the procedural search, the score dropped to 25.76%. When they removed the semantic search (the word-matching part), it dropped to 32.29%. This proves that you need both the "how" and the "what" to succeed.

What the Paper Doesn't Claim

It's important to know what this paper doesn't say.

It doesn't claim to have solved all coding problems. The score is 41.14%, meaning it still fails more than half the time.
It doesn't say this works for every programming language. The tests were done only on Python projects.
It doesn't claim the "agent" is perfect. The authors admit that sometimes the agent runs out of time or memory and can't find the best code, suggesting that better exploration tools are still needed.

The Bottom Line

The paper suggests that to write good code for big projects, AI needs to stop just looking for words that match and start looking for patterns of thinking. By finding code that solves problems in the same way, even if the words are totally different, ProjAgent helps AI write better, more reliable software. It's a step forward, but the journey to perfect code generation is still ongoing.

Technical Summary: ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation

Problem Statement

Repository-level code generation requires Large Language Models (LLMs) to implement target functions while navigating complex cross-file dependencies, project-specific conventions, and distributed utilities. Existing retrieval-augmented generation (RAG) approaches for this task predominantly rely on lexical (e.g., BM25) or semantic (e.g., dense embeddings) similarity.

The authors identify a critical limitation in these methods: they often fail to retrieve repository functions that implement similar procedural logic but differ significantly in naming, data types, or application domains. For instance, a function validating input shapes in a cosmology module may be procedurally identical to a function validating temperature inputs in a physics module, yet surface-level similarity metrics (lexical or semantic) fail to link them. This "procedural gap" leads to LLMs hallucinating APIs or generating code that violates project conventions because the necessary logical patterns are missing from the retrieved context.

Methodology: ProjAgent

The paper proposes ProjAgent, a repository-level code generation system that introduces procedural similarity as an explicit retrieval signal, integrated within an agentic workflow and a static-analysis feedback loop.

1. Representing Procedural Similarity

To capture implementation behavior beyond surface text, ProjAgent utilizes reasoning-subspace projections derived from LLM hidden states.

Subspace Decomposition: Building on Hu et al. [14], the system decomposes the LLM's hidden state space into a semantic subspace (linguistic expression) and a reasoning subspace (internal logic). This is achieved via Singular Value Decomposition (SVD) on the unembedding layer.
Projection: The hidden states of code steps are projected onto the reasoning subspace to isolate procedural logic.
Debiasing: To address the anisotropy phenomenon (where vectors cluster in a common direction regardless of content), the authors introduce a Principal Component Analysis (PCA) based debiasing step. This removes the dominant shared direction and mean-centering, significantly improving the discriminative power of cosine similarity for procedural matching.

2. The Agentic Retrieval Workflow

ProjAgent employs a multi-stage retrieval process to construct a rich context for the target function:

Offline Preprocessing: The repository is preprocessed to decompose context functions into logical steps (e.g., input validation, state transformation). Each step is validated against the original code and projected into the reasoning subspace.
Procedural Retrieval (Agentic):
- Seed Finding: The target function is decomposed into steps. An initial set of procedurally similar context steps is identified via projection similarity.
- Agentic Exploration: An LLM agent navigates the repository using tools (e.g., read_func, search_func) to discover new functions that may contain procedurally similar steps, iteratively expanding the seed set.
- Seed Expansion & Verification: The system expands the search to unprocessed functions using projection similarity thresholds. A verification step (LLM-based) filters out false positives, particularly for generic operations, ensuring high-quality procedural matches.
Semantic Retrieval: To complement procedural logic, the system retrieves semantically related symbols (APIs, classes, constants) using a hybrid of lexical (BM25) and semantic (dense embedding) scores. This ensures the LLM has access to the specific repository entities required to implement the retrieved logic.
Code Generation & Feedback: The LLM generates code using the combined procedural and semantic context. A conservative static-analysis feedback loop then iteratively refines the code. It performs AST-based syntax checks and semantic consistency checks (method calls, field access, variable resolution), feeding errors back to the LLM for repair.

Key Contributions

Procedural Similarity as a Retrieval Dimension: The paper introduces procedural similarity as a distinct and effective retrieval signal, addressing the gap where functions share logical patterns but lack lexical or semantic overlap.
Reasoning-Subspace Projections: The authors leverage LLM hidden-state projections to represent procedural logic, enabling the retrieval of related functions across different naming conventions and domains.
ProjAgent System: A complete system integrating an agentic workflow for procedural context discovery, hybrid retrieval (procedural + semantic), and a static-analysis feedback loop for iterative code repair.
Comprehensive Evaluation: A rigorous evaluation on the REPOCOD benchmark, including ablation studies and analysis of retrieval component contributions.

Results

Evaluated on the REPOCOD benchmark (980 problems across 11 repositories) using the Qwen2.5-Coder-14B-Instruct model:

Performance: ProjAgent achieved a Pass@1 of 41.14%, outperforming existing baselines:
- SpecAgent (SOTA): 34.52%
- Dense Retrieval: 28.83%
- Sparse Retrieval: 26.58%
- Same-File Retrieval: 14.98%
Ablation Studies:
- Removing procedural retrieval caused the largest performance drop (41.14% $\to$ 25.76%), highlighting its critical role in providing implementation guidance.
- Removing semantic retrieval also significantly degraded performance (41.14% $\to$ 32.29%), confirming the need for repository-specific symbols.
- Removing the feedback loop resulted in a modest decrease (41.14% $\to$ 40.29%), attributed to the conservative nature of the static analysis.
Retrieval Effectiveness: The projection similarity method was found to be highly effective at filtering out dissimilar contexts (high precision in the "leftover" group) but required a two-stage design (projection + verification) to handle false positives in the "promoted" group.

Significance and Claims

The paper claims that procedural similarity is an effective and previously unexplored retrieval dimension for repository-level code generation. The authors argue that successful code generation depends not only on retrieving semantically relevant code but also on retrieving context that shares the same underlying computational procedures.

By integrating procedural similarity with conventional lexical and semantic signals, ProjAgent demonstrates that LLMs can be guided to implement complex logic even when the source code differs significantly in domain or vocabulary. The work suggests a shift toward behavior-aware retrieval methods in software engineering, moving beyond surface-level text matching to capture the functional intent of code. The authors remain modest regarding external validity, noting that results are specific to Python repositories and the specific backbone model used, and that search coverage remains a bottleneck for further improvements.