💬 NLP

Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation

This paper introduces OpenCoder, an uncertainty-aware framework that estimates and leverages source-specific uncertainty to filter and rank heterogeneous retrieval evidence, thereby improving repository-level code generation correctness while demonstrating that its benefits are dependent on the specific LLM backend and evidence interactions.

Original authors: Chandan Kumar Sah, Xiaoli Lian, Li Zhang

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chandan Kumar Sah, Xiaoli Lian, Li Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex LEGO castle, but you've been handed a single instruction manual that only covers the front door. You know the door, but the castle needs windows, a roof, and a secret underground tunnel. This is the daily struggle for Artificial Intelligence (AI) when it tries to write computer code for real-world projects. While AI has gotten incredibly good at writing small, isolated snippets of code, it often gets lost when asked to build something that fits into a massive, existing software "neighborhood." To solve this, researchers use a technique called Retrieval-Augmented Generation (RAG). Think of RAG as giving the AI a super-powered search engine: before it writes a single line of code, it looks up similar projects, checks the neighborhood rules (the project's specific conventions), and finds the right tools (APIs) to use.

However, there's a catch. Just because the search engine finds a lot of information doesn't mean that information is helpful. Sometimes the AI finds a piece of code that looks similar but actually breaks the project; other times, it finds a tool that doesn't fit the specific job. The information is there, but it's noisy, conflicting, or just plain wrong. The big question researchers have been asking is: How do we teach the AI to not just find the right information, but to know how much to trust it? If the AI can't tell the difference between a helpful clue and a misleading red herring, it will build a castle that collapses the moment you try to open the door.

This is where a new study from researchers at Beihang University comes in. They introduce a system called OpenCoder, which acts like a skeptical, ultra-organized project manager for the AI. Instead of blindly trusting every piece of information the search engine finds, OpenCoder assigns a "doubt score" to every single clue. It asks, "How uncertain are we that this API is the right one?" or "How likely is it that this similar code will clash with our project?" By treating uncertainty not as a bug, but as a useful signal, OpenCoder filters out the noise, ranks the clues based on how reliable they seem, and even knows when to stop and fix its own mistakes.

The researchers tested this system by asking the AI to write code for 32 different real-world tasks. They found that when using a powerful AI model called GPT, OpenCoder significantly boosted the success rate of the final code from 56.25% (with standard search methods) to 78.13%. However, the researchers discovered a crucial nuance: this improvement matched the performance of a control group that used standard search but added a "verify and repair" step. This suggests that while OpenCoder's uncertainty filtering helped, the massive jump in success was largely driven by the system's ability to verify and fix errors, rather than filtering alone. The secret sauce wasn't just finding more information; it was the system's ability to say, "This specific piece of evidence looks shaky, so let's ignore it," and "This other piece looks solid, so let's use it," all while having a safety net to catch mistakes.

However, the story isn't a simple "AI wins forever." The researchers were careful to note that this success depends heavily on which AI brain is doing the thinking. When they swapped GPT for a different model called Gemini, the results were much less clear. The improvements weren't statistically significant, suggesting that OpenCoder's "uncertainty radar" works differently depending on the AI's personality. Furthermore, the system hit a wall when the project was missing too much information. In these cases of incomplete evidence, a standard system with verification and repair actually outperformed OpenCoder. This indicates that when the search engine can't find the necessary tools to begin with, OpenCoder's filtering mechanism can sometimes suppress the few scraps of evidence that were available, rather than improving the final decision.

The study also discovered something surprising about how different types of information work together. You might think that having "similar code," "project context," and "API knowledge" is always better than having just one. But the researchers found that there is no universal rule. Sometimes, adding "similar code" actually confused the AI, unless it was paired with the right "project context." It's like having a map, a compass, and a GPS: if you only have the GPS, you might get lost; if you have the map and the compass but no GPS, you might be fine; but if you have all three and they contradict each other, you might end up going in circles. The value of each clue depends entirely on what other clues are present.

To make this work, OpenCoder uses a five-step dance. First, it builds a library of all the project's rules and tools. Second, it breaks the user's request down into small steps. Third, it goes hunting for clues, but this time it scores them based on how "uncertain" they feel. Fourth, it generates the code, but it keeps a close eye on the "uncertainty score" to avoid using shaky clues. Finally, and perhaps most importantly, it acts as its own quality control inspector. It runs the code through a series of tests. If the code fails, it doesn't just give up; it identifies the error and attempts to repair the code based on that validation feedback.

In the end, the paper suggests that the future of AI coding isn't just about making the AI smarter or giving it more data. It's about teaching the AI to be humble and critical. By treating uncertainty as a tool to guide decisions—filtering out bad data, verifying the output, and fixing errors on the fly—systems like OpenCoder can build more reliable software. But as the researchers warn, this isn't a magic wand that works in every situation. It works best when the AI has enough good information to work with and when the specific AI model is tuned to understand the "doubt scores" correctly. For now, OpenCoder is a powerful step forward, proving that sometimes, knowing what you don't know is the most important part of solving the puzzle.

Technical Summary: Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation

Problem Statement

Repository-level code generation requires Large Language Models (LLMs) to synthesize implementations that respect cross-file dependencies, project conventions, and private APIs. While Retrieval-Augmented Generation (RAG) addresses the limitation of parametric knowledge by supplying external repository evidence, existing approaches primarily focus on optimizing what information to retrieve (e.g., similar code, repository context, API knowledge). They largely neglect the inherent uncertainty associated with this evidence.

Retrieved evidence is often heterogeneous and unreliable: similar-code examples may be semantically related but functionally incompatible; repository context may contain redundant or conflicting dependencies; and API knowledge may be incomplete or incorrectly scoped. Current methods optimize retrieval relevance without explicitly modeling how uncertainty in these sources affects downstream generation, verification, and repair. Consequently, combining additional evidence can introduce noise and propagate retrieval errors, leading to executable code that still contains substantial API misuse or functional defects.

Methodology: OpenCoder

The authors introduce OpenCoder, an uncertainty-aware framework designed to treat uncertainty as an actionable control signal rather than a post-hoc confidence score. The framework operates through five phases:

Repository Knowledge & Uncertainty Profiling: Constructs an index containing three evidence sources: project-specific APIs ( $A$ ), contextual repository code ( $X$ ), and semantically similar code examples ( $S$ ).
Query Uncertainty Decomposition: Decomposes the user query into implementation steps and predicts the evidence required for each step.
Uncertainty-Aware Multi-Source Retrieval: Retrieves candidates from $A$ $A$ , $X$ $X$ , and $S$ $S$ . Crucially, OpenCoder assigns a source-specific uncertainty score $u_m(e|q) \in [0,1]$ $u_{m} (e ∣ q) \in [0, 1]$ to each retrieved item, where higher values indicate lower confidence in relevance, completeness, or compatibility.
- Evidence is scored using a combination of cosine similarity ( $r_m$ ), predicted source intent weight ( $w_m$ ), and uncertainty: $\gamma_m(e|s) = r_m \cdot w_m \cdot \max\{0, 1 - \alpha u_m(e)\}$ .
- Low-confidence evidence is filtered, and the remaining items are deduplicated and ranked under a shared context budget.
Uncertainty-Guided Code Generation: The LLM generates candidate code conditioned on the query, the filtered evidence set, and the source-wise uncertainty trace.
Verification & Uncertainty Mitigation: Generated candidates undergo static checks and executable tests. The first passing candidate is selected. If no raw candidate passes, the system identifies the "mode" candidate (from the largest self-consistency group) and performs up to two rounds of validation-guided repair.

The framework also employs Target-Aware API Refinement to remove self-retrieved, redundant, or intent-incompatible APIs before generation.

Key Contributions

Formulation: The paper formulates repository-level code generation as decision-making under uncertainty over heterogeneous evidence, moving beyond simple relevance ranking.
Framework: OpenCoder is a unified system that estimates source-specific uncertainty and operationalizes it across evidence filtering, multi-source integration, generation, verification, and repair.
Factorial Analysis: Through a full factorial analysis of API knowledge, repository context, and similar-code evidence, the authors demonstrate that retrieval utility is interaction-dependent. There is no universal additive ranking of sources; the value of a specific source depends on the accompanying evidence and the LLM backend.
Controlled Evaluation: The authors conduct a rigorous, matched evaluation across GPT and Gemini backends on executable repository-level tasks, isolating the impact of uncertainty-aware processing from downstream verification and repair.

Experimental Results

The evaluation utilizes an expanded set of 32 matched RepoExec-inline tasks and a stress test on 10 partial-context ExecRepoBench tasks.

GPT Performance: On the 32-task RepoExec-inline set, OpenCoder improved selected-output correctness from 56.25% (Baseline RAG) to 78.13% (a 21.88 percentage point gain, nominal $p = .039$ ). However, it tied with the RAG + Verify/Repair control (which uses the same evidence but lacks uncertainty-aware filtering), suggesting the gain over Baseline RAG is largely driven by the verification/repair budget rather than uncertainty filtering alone.
Gemini Performance: For Gemini, the differences between OpenCoder, Baseline RAG, and the control were not statistically supported. The control (RAG + Verify/Repair) performed strongest across most metrics.
Interaction Effects: The factorial analysis (RQ1) revealed that no single evidence source had a statistically significant marginal effect on performance. Instead, significant cross-source interactions were observed (e.g., repository context and similar-code evidence showed a positive interaction for GPT).
API Retrieval: Target-aware refinement substantially improved API-set retrieval, increasing macro F1 from 43.4% to 64.8% for GPT and to 59.1% for Gemini.
Boundary Conditions: On the context-limited ExecRepoBench stress test, the RAG + Verify/Repair control substantially outperformed OpenCoder. This indicates that when repository evidence is incomplete, uncertainty-aware filtering may suppress useful alternatives rather than improving the final decision.

Significance and Claims

The paper claims that uncertainty is most useful not as an isolated confidence estimate, but as an operational control signal coupled with evidence fusion, executable validation, and repair.

Backend Dependence: The benefits of OpenCoder are backend-dependent. While it significantly improved GPT output selection over plain RAG, it did not show statistically significant advantages over a verification-and-repair control, nor did it show clear benefits for Gemini.
Evidence Completeness: The framework's effectiveness relies on the completeness of repository evidence. In scenarios with incomplete evidence, the uncertainty mechanism may inadvertently filter out necessary information.
Actionable Control: The primary contribution is the shift from optimizing what to retrieve to modeling how reliable the retrieved evidence is, using that signal to guide filtering and generation.

The authors conclude that while OpenCoder demonstrates the potential of uncertainty-aware integration, its advantages are contingent on the specific LLM backend and the quality/completeness of the retrieved repository evidence. They emphasize that generated code still requires developer review and project-specific testing.