💻 computer science

Toward Linking Declined Proposals and Source Code: An Exploratory Study on the Go Repository

This paper presents an LLM-driven approach to establish traceability links between declined proposals and source code in the Go repository, demonstrating that while the pipeline achieves high accuracy in selecting the correct code granularity, challenges remain due to redundant discussions lacking concrete implementation details.

Original authors: Sota Nakashima, Masanari Kondo, Mahmoud Alfadel, Aly Ahmad, Toshihiro Nakae, Hidenori Matsuzaki, Yasutaka Kamei

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sota Nakashima, Masanari Kondo, Mahmoud Alfadel, Aly Ahmad, Toshihiro Nakae, Hidenori Matsuzaki, Yasutaka Kamei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a massive, bustling library called the "Go Repository." This library doesn't just store books (code); it also stores thousands of letters of suggestion (proposals) from people who want to add new sections, rewrite chapters, or build new wings to the library.

Most of the time, when someone writes a letter suggesting a change, the librarians (maintainers) say, "Great idea!" and they actually build it. Researchers have spent years studying how to connect those "Great idea!" letters to the actual blueprints of the new buildings.

But what about the letters that get rejected?

In the past, if a librarian said, "No, we can't build that," the letter was just filed away in a dusty box. The paper you're asking about is a groundbreaking study that says: "Wait a minute! Those rejected letters are full of gold."

Even though the feature wasn't built, the letter often explains why it was rejected, what the writer was thinking, and what the constraints were. If a future writer wants to build something similar, they need to know why the last attempt failed. The problem? Those rejected letters are floating in the air, disconnected from the library's actual shelves.

The Mission: Building a Bridge to the "No" Pile

The authors of this paper wanted to build a magic bridge (a computer program) that automatically connects a "Rejected Letter" to the specific part of the library (the source code) it was talking about.

They faced a tricky puzzle:

The Scale Problem: A letter might be talking about the whole "History Section" (Directory), a specific "Book" (File), or just a single "Paragraph" (Function). The computer needs to know which level to look at.
The Ghost Problem: Since the feature was rejected, the new code doesn't exist yet. The computer can't just look for the new code; it has to guess where the old code is that the letter was trying to change.

The Solution: The "Super-Intelligent Librarian" (LLM)

To solve this, the researchers built a pipeline driven by a Large Language Model (LLM). Think of this LLM as a super-intelligent, hyper-observant librarian who has read every book in the library and every letter ever written.

Here is how their "Magic Bridge" works, step-by-step:

Step 1: The "Zoom Lens" Decision (Granularity)

First, the Super-Librarian reads the rejected letter and asks: "Is this person talking about the whole building, a specific room, or just a single sentence?"

Directory Level: "We need a new wing!" (The whole package).
File Level: "We need to rewrite this specific chapter." (The file).
Function Level: "We need to fix this one typo in this paragraph." (The function).
Result: The computer gets the "Zoom Lens" set to the right level.

Step 2: The "Treasure Hunt" (Localization)

Once the lens is set, the Super-Librarian goes hunting.

If the letter is about a Directory, the librarian scans the library map to find the right wing.
If it's about a File, they walk into that wing and find the specific book.
If it's about a Function, they open the book and find the exact paragraph.
The Trick: Unlike older methods that just looked for similar words (like a basic search engine), this Super-Librarian understands the context. They know that if someone says "fix the login button," they are talking about the login.go file, even if the words "login" and "button" aren't in the file name.

Step 3: The "Final Verdict" (Link Decision)

Finally, the librarian holds the letter in one hand and the code in the other and asks: "Do these two belong together?" If yes, they staple them together with a digital paperclip (a traceability link).

What Did They Find?

The researchers tested this on the official Go language library. Here is the "report card":

It's Pretty Good at the Big Picture: The system was very good at figuring out if a letter was about a whole wing or a specific room (83.6% accuracy).
It's Okay at the Details: When it came to finding the exact paragraph (function), it was a bit shaky (only 40% accuracy).
The "Precision" Score: When the system did make a link, it was right about 64% of the time. That's a solid start for a task no one has ever tried to automate before!

Why Does It Sometimes Fail? (The "Why" Analysis)

The researchers dug into the failures and found two main reasons the Super-Librarian got confused:

The "Vague Request" Problem: Sometimes the letter said, "We need to make the system faster," but didn't say how or where. It's like telling a contractor, "Fix the house," without pointing to the broken window. The computer didn't know where to look.
The "Noise" Problem: Sometimes the letter was 50 pages long, full of arguments, jokes, and side conversations. The important clue (e.g., "change the auth function") was buried in the middle of a rant about coffee breaks. The computer got distracted by the noise.

The Big Takeaway

This paper is like discovering a new way to organize a library. Before, if you rejected an idea, you threw the paper away. Now, we have a tool that says: "Don't throw it away! Let's pin it to the wall right next to the code it was talking about."

This helps future developers see the "ghosts" of past decisions. They can walk up to a piece of code and see a note that says, "Hey, we thought about adding this feature back in 2024, but we decided against it because it would break security. Here's why."

It turns the "No" pile into a valuable history lesson, ensuring that the library learns from its past rejections, not just its successes.

1. Problem Statement

Software traceability links connect artifacts like issue reports, design documents, and source code, facilitating tasks such as bug localization and code comprehension. While existing research focuses heavily on accepted proposals (those implemented in the codebase), declined proposals (those rejected after discussion) have been largely overlooked.

The Gap: Declined proposals contain valuable design rationales, failed trade-offs, and implicit decision criteria. However, because the code was never implemented, these discussions are disconnected from the source code, making it difficult for developers to access this knowledge or understand why certain features were rejected.
The Challenge: Establishing traceability links for declined proposals is significantly harder than for accepted ones because the "ground truth" code does not exist in the repository. Automated methods relying on code changes (commits) cannot be used.
Goal: To automatically generate traceability links between declined proposals and the relevant source code artifacts (directories, files, or functions) that would have been affected, thereby making the knowledge within rejected discussions accessible.

2. Methodology

The authors propose an LLM-driven pipeline designed specifically for the unique characteristics of declined proposals. The pipeline consists of three main phases:

A. Phase 0: Granularity Decision

Unlike standard traceability recovery, the scope of a declined proposal varies widely (e.g., a high-level architectural change vs. a specific function modification).

Task: The LLM analyzes the proposal text to determine the appropriate granularity level for the link:
- Directory Level: Changes to package structure (adding/removing files/directories).
- File Level: Changes confined to existing files (adding functions/types).
- Function Level: Changes to internal logic or documentation of specific functions/methods.
Output: A selected granularity level that dictates the subsequent steps.

B. Phase 1: Localization (Hierarchical Search)

Inspired by the Agentless fault-localization technique, the pipeline uses a hierarchical, context-aware approach to narrow down the code scope, avoiding the limitations of pure embedding similarity.

Directory Localization: The LLM receives the repository structure (as a compact tree) and the proposal text to identify relevant directories.
File Localization: Using the identified directories, the LLM identifies specific files.
Function Localization: For files, the LLM processes compressed function signatures (not full file contents) to identify specific methods/functions.

Execution Logic: The pipeline only executes localization steps necessary for the granularity determined in Phase 0 (e.g., if Phase 0 selects "Directory," only Phase 1.1 runs).

C. Phase 2: Link Decision

Task: The LLM performs a binary classification for each candidate code element (directory, file, or function) paired with the proposal.
Prompt: "Is this source code element relevant to the given proposal? (Yes/No)"
Output: A set of final traceability links.

D. Dataset and Evaluation

Dataset: Proposals from the official Go repository (GitHub issues). The study used 341 declined proposals and 262 accepted proposals (for baseline comparison).
Model: DeepSeek-V3.1 (non-thinking mode) with temperature set to 0.0.
Ground Truth: Since automated verification is impossible for declined code, ground truth was established via manual labeling by authors using strict criteria for granularity and link correctness. Inter-rater agreement (Cohen's $\kappa$ ) was substantial (>0.6).

3. Key Contributions

Novel Task Definition: The first study to establish traceability links between declined proposals and source code, unlocking the value of unimplemented design discussions.
LLM-Driven Pipeline: A novel architecture that combines Granularity Decision (Phase 0) with Hierarchical Localization (Phase 1) and Link Decision (Phase 2), moving beyond simple embedding-based retrieval.
Empirical Analysis & Failure Taxonomy: A detailed analysis of why the pipeline fails, categorizing issues into "missing implementation guidance" and "obscured key information."

4. Results

Performance on Declined Proposals (RQ1)

Granularity Accuracy: The pipeline achieved an overall accuracy of 0.836 in selecting the correct code granularity.
- File Level: 0.887 (Highest)
- Directory Level: 0.696
- Function Level: 0.407 (Lowest, indicating difficulty in pinpointing specific functions without implementation code).
Link Precision: The pipeline generated correct links with a mean precision of 0.643.
- File Level: 0.688
- Directory Level: 0.529
- Function Level: 0.253 (Significantly lower, largely due to incorrect granularity selection).
Note: Recall could not be calculated as it would require manually verifying every code element in the repository against every proposal.

Comparison with Baseline (RQ1 - Preliminary)

When tested on accepted proposals (where code exists), the proposed LLM pipeline outperformed the state-of-the-art RAG-based (embedding) baseline (LiSSA) in F1 score and precision across all top- $k$ settings, validating the LLM-based localization approach.

Failure Analysis (RQ2)

The study identified two primary reasons for failure:

Granularity Misclassification:
- Insufficient Implementation Guidance (42.9%): The discussion described what to do but not how or where.
- Conflicting/Hidden Guidance: Multiple approaches were suggested, or key signals were buried in long, redundant text.
- Confusing References: Lexical references pointed to the wrong granularity level.
Link Generation Errors (Correct Granularity, Wrong Link):
- Performance heavily depended on Reference Explicitness.
- Level-1 (Direct): Explicitly naming the file/function resulted in high precision (0.841).
- Level-3 (Implicit): Requiring inference resulted in low precision (0.408).

Discussion Length: Surprisingly, the length of the discussion did not correlate with performance ( $\rho = 0.035$ ).

5. Significance and Future Directions

Knowledge Preservation: This work enables developers to trace design rationales and constraints even for features that were never built, preventing the repetition of failed ideas.
Methodological Shift: It demonstrates that LLMs, when guided by hierarchical context and specific granularity decisions, outperform traditional embedding-based retrieval for complex, non-implemented software artifacts.
Future Work:
- Input Augmentation: Enriching proposals with external context (related proposals, code comments) when implementation details are missing.
- Summarization: Using summarization techniques to filter out irrelevant content that obscures key linking signals.
- Generalization: Extending the approach to other ecosystems (e.g., Python PEPs) to validate generalizability.

In conclusion, the paper establishes a viable, though imperfect, automated method for recovering traceability in the "shadow" of software development—where ideas are discussed but not implemented—providing a foundation for better design understanding and maintenance.