💻 computer science

Think Harder and Don't Overlook Your Options: Revisiting Issue-Commit Linking with LLM-Assisted Retrieval

This paper evaluates various issue-commit linking techniques by combining efficient retrieval methods with reranking models, finding that dense retrieval improves recall and traditional machine learning-based reranking outperforms computationally expensive large language models, suggesting that simpler, retrieval-based pipelines remain a practical solution for large-scale software traceability.

Original authors: Cole Morgan, Muhammad Asaduzzaman, Shaiful Chowdhurry, Shaowei Wang

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cole Morgan, Muhammad Asaduzzaman, Shaiful Chowdhurry, Shaowei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive library where every book (a piece of software code) has a note attached to it explaining why it was written. Sometimes, the librarian (the developer) writes the note clearly, saying, "This fixes the broken window in the kitchen." But often, they forget to write the note, or they write it in a way that doesn't match the "ticket" the kitchen staff filed earlier.

This paper is about building a better system to find those missing notes and match them to the right tickets. The authors call this "Issue-Commit Linking." They wanted to see if the newest, most expensive technology (Large Language Models or LLMs) is actually necessary, or if older, simpler tools can do the job just as well.

Here is a breakdown of their findings using simple analogies:

1. The Search Window: When to Look

Imagine you are looking for a specific letter that was written to fix a leak.

The Old Idea: Some researchers said, "Just look at the letters written in the 7 days before and after the leak was reported."
The New Idea (EasyLink): Others said, "Look at everything written in the whole year after the report."
The Paper's Discovery: The authors tested this and found that the "whole year" rule works great for some libraries but misses the mark for others. Sometimes, the fix happens right when the leak is closed, not just when it's reported.
The Solution: They created a "Hybrid Window." Think of it as looking at the whole year after the report PLUS a 30-day buffer around the day the issue was officially closed. This catches 97% of the correct links without sifting through too much junk.

2. The Search Engine: How to Find the Needle

Once you know when to look, you need to find the right document among thousands. The authors tested two types of search engines:

The Keyword Search (Sparse): This is like searching a library catalog by exact words. If the ticket says "broken window" and the note says "window repair," it finds it. But if the note says "glass replacement," it might miss it because the words don't match exactly.
The "Vibe" Search (Dense): This uses AI to understand the meaning. It knows that "broken window" and "glass replacement" are about the same thing, even if the words are different.
The Result: The "Vibe" search (Dense) was much better at finding the right documents. However, the authors found a clever trick: Mix them. By combining the Keyword search and the "Vibe" search, they got the best of both worlds. It's like having a librarian who knows the exact location of books and understands the story behind them.

3. The Sorting Hat: Who Gets to Rank First?

After the search engine finds a shortlist of 20 possible candidates, a "Reranker" has to decide which one is the real fix. The authors tested three types of "Sorters":

The Old School Sorter (Traditional Machine Learning): These are like experienced detectives who look at clues like "Who wrote this?", "When was it written?", and "Does the text sound similar?" They are fast, cheap, and very smart.
The Deep Learning Sorter: These are like detectives who have read every book in the library and can spot subtle patterns.
The Super-Intelligent AI (LLMs): These are the "Genius" sorters (like ChatGPT or GPT-5.1). They are incredibly powerful but require expensive supercomputers to run.

The Big Surprise:
The authors found that the Old School Sorter (Traditional Machine Learning) actually performed just as well, and sometimes even better, than the Genius AI.

Why? The Old School Sorter was very good at using "context clues" (like who the developer is and when they worked). The Genius AI was great at understanding language, but it didn't always use the context clues as effectively as the simpler model.
The Cost: Running the Genius AI on all their data cost about $128. The simpler models cost almost nothing and run on a standard laptop.

The Main Takeaway

The paper argues that before you spend a fortune on expensive, complex AI to solve a problem, you should try the simpler, cheaper tools first.

In this specific case of linking software bugs to code fixes:

Time matters: Look at a specific time window (1 year + 30 days around closure).
Mix your search: Combine keyword searches with "meaning" searches.
Don't overcomplicate: A simple, well-trained detective (Traditional Machine Learning) often solves the case just as well as a super-genius AI, but much faster and cheaper.

The authors conclude that for large-scale software projects, these simpler, retrieval-based pipelines are the most practical and effective solution.

1. Problem Statement

Software traceability relies on linking issue reports (e.g., bug reports, feature requests) to the specific code commits that resolve them. While developers often manually include issue identifiers in commit messages, this practice is inconsistent, leading to a significant gap in traceability. Automated Issue-Commit Link Recovery is essential for maintenance, debugging, and evolution analysis.

Existing automated techniques range from simple heuristics to complex Deep Learning (DL) and Large Language Model (LLM) approaches. However, recent studies (e.g., EasyLink) have introduced two-stage pipelines (retrieval + reranking) using LLMs and specific time windows (e.g., a one-year window post-issue creation) without rigorously validating:

The universality of the selected time windows across different datasets.
The optimal retrieval methods (sparse vs. dense) for narrowing the search space.
Whether the computational cost of LLMs is justified compared to simpler, traditional machine learning (ML) models for the reranking stage.

2. Methodology

The authors conducted a comprehensive empirical study involving three distinct datasets and a two-stage pipeline (Retrieval $\rightarrow$ Reranking).

A. Datasets

The study utilized and constructed three datasets to ensure generalizability:

BTLink Dataset: 13 Apache projects + 1 GitHub project (39k true links).
EasyLink Dataset: 20 open-source projects (159k true links). Note: Apache Airflow was excluded due to Jira-to-GitHub migration issues.
Apache Dataset (New): 10 popular Apache projects (Spark, Hadoop, Solr, etc.) using Jira, collected from 2006–2025 (208k true links).
Data Cleaning: Commits were filtered to remove merges and non-code changes. A manual validation of 400 links confirmed a high inter-rater agreement (Cohen's kappa = 0.93).

B. Experimental Pipeline

The study evaluated the pipeline in three phases corresponding to three Research Questions (RQs):

Phase 1: Time Window Optimization (RQ1)
- Tested various temporal boundaries: 1-year post-creation, 7-day/30-day buffers around issue closure, and hybrid combinations.
- Goal: Determine the window that captures the maximum number of true links without introducing excessive noise.
Phase 2: Retrieval Method Evaluation (RQ2)
- Sparse Retrievers: BM25, BM25L.
- Dense Retrievers: SBERT-Semantic Search (SBERT-SS), HNSW, ANNOY, LSH.
- Hybrid: Reciprocal Rank Fusion (RRF) combining sparse and dense results.
- Metric: Recall@K (to ensure the true commit is in the candidate set) and Precision@K.
Phase 3: Reranking Technique Evaluation (RQ3)
- Input: Top 20 candidates retrieved by the best-performing retrieval method (RRF).
- Models Evaluated:
  - Traditional ML: FRLink (TF-IDF), RCLinker (Random Forest with metadata/text), Hybrid-Linker.
  - Deep Learning: BTLink (Bi-encoders), Cross-Encoder (ms-marco-MiniLM).
  - LLMs: GPT-5.1 (API), Llama-3.1-8B, Gemma-7B, Qwen3-32B (Local).
- Constraint: Issue keys were removed from commit messages to prevent data leakage and force semantic understanding.

3. Key Contributions & Findings

RQ1: Time Window Selection

Finding: A rigid "one-year post-creation" window is not universally optimal. While it captured 97% of links in the EasyLink dataset, it only captured ~91-94% in BTLink and Apache datasets.
Insight: There is a strong correlation between true links and the issue-closure date.
Solution: A hybrid temporal heuristic (1 year post-creation + 30 days before/after issue closure) was proposed. This captured >97% of true links across all three datasets, balancing high recall with a manageable search space.

RQ2: Retrieval Method Selection

Finding: Dense retrieval methods (semantic embeddings) significantly outperform sparse methods (keyword matching) in Issue-Commit linking.
- SBERT-SS achieved the highest Recall@10 (~82-88%).
- ANN methods (HNSW, ANNOY) offered a strong trade-off, achieving performance close to SBERT-SS with better efficiency.
- Sparse methods (BM25) struggled due to vocabulary mismatch between issue descriptions and commit messages.
Hybrid Success: Combining dense and sparse methods via Reciprocal Rank Fusion (RRF) yielded the highest Recall, leveraging the complementary strengths of both approaches (semantic understanding + exact keyword matching).

RQ3: Reranking Technique Selection

Finding: Simpler models often outperform or match complex LLMs.
- RCLinker (a Random Forest model using metadata and textual features) achieved the highest performance across all datasets (e.g., P@1 of 94.9% on Apache), surpassing both Deep Learning and LLM approaches.
- LLMs: Performance was mixed. Large models like GPT-5.1 and Qwen3-32B performed well (comparable to Cross-Encoders), but smaller models (Llama, Gemma) were less effective.
- Cost-Benefit: RCLinker and Cross-Encoders are computationally cheap (minutes to train/run, no API costs). In contrast, running GPT-5.1 on the full test set cost ~$128 USD, scaling linearly with data size.
Conclusion: Increased model complexity does not guarantee better results. Traditional ML models leveraging repository metadata (developer overlap, temporal distance) remain highly competitive.

4. Significance and Implications

Revisiting Assumptions: The paper challenges the assumption that "bigger is better" in Issue-Commit linking. It demonstrates that state-of-the-art LLMs are not strictly necessary for high-precision reranking if a robust retrieval pipeline and metadata-rich traditional models are used.
Practical Pipeline: The authors propose a practical, cost-effective pipeline:
1. Filter: Use the hybrid time window (1yr + 30d closure).
2. Retrieve: Use RRF (Dense + Sparse) to get top candidates.
3. Rerank: Use a lightweight model like RCLinker or a Cross-Encoder.
Resource Efficiency: For large-scale industrial applications, the proposed pipeline avoids the high financial and hardware costs associated with LLM inference while maintaining superior or comparable accuracy.
Future Work: The authors suggest exploring task-specific prompts for smaller open-source LLMs and investigating learned fusion strategies for retrieval.

Summary Table of Key Results

Component	Best Performer	Key Insight
Time Window	Hybrid (1yr + 30d closure)	Universal 1-year window fails; closure date is critical.
Retrieval	RRF (Dense + Sparse)	Dense > Sparse; Fusion maximizes Recall.
Reranking	RCLinker (Random Forest)	Simpler ML > Complex LLMs; Metadata is crucial.
LLM Performance	GPT-5.1 / Qwen3-32B	Good, but expensive; smaller LLMs underperform.

This study serves as a critical "reality check" for the software engineering community, advocating for a balanced approach that prioritizes data quality, retrieval strategy, and cost-effectiveness over the blind adoption of expensive LLMs.