💻 computer science

CoRaCommit: A VS Code Extension for Commit Message Generation with Exemplar Retrieval

CoRaCommit is a VS Code extension that improves commit message generation by retrieving similar exemplars for context, comparing parallel LLM outputs, and dynamically adapting recommendations based on user feedback, thereby outperforming existing tools in quality metrics.

Original authors: Chaoran Cai, Bo Xiong, Chong Wang, Lulu He, Peng Liang

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaoran Cai, Bo Xiong, Chong Wang, Lulu He, Peng Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a software developer. Every time you finish a small piece of work, you have to save it to your project's history. To do this, you write a "commit message"—a short note explaining what you changed. Think of this like writing a receipt for a purchase; it tells you (and your team) exactly what you bought and why.

The problem is, writing these receipts is boring. Developers are busy fixing bugs and building features, so they often skip the notes, write vague ones like "fixed stuff," or make them inconsistent. This makes it hard to understand the project's history later.

Enter CoRaCommit, a tool designed to fix this. It's like a smart assistant living inside your coding software (VS Code) that writes these notes for you, but with a few special tricks to make them actually good.

Here is how it works, using simple analogies:

1. The "Smart Librarian" (Retrieval-Augmented Generation)

Most tools that write commit messages just look at your code changes and guess what you did. It's like asking a chef to describe a new dish just by looking at the raw ingredients on the counter.

CoRaCommit is different. Before it writes your note, it acts like a smart librarian. It looks at your code changes and immediately runs to a massive library of past project notes (called the ApacheCM dataset). It finds notes from the past that are very similar to what you are doing right now.

The Analogy: If you are fixing a leak in a pipe, CoRaCommit doesn't just guess; it finds a note from last year where someone fixed a similar pipe leak. It uses that old note as a "cheat sheet" or a reference to help write your new note. This ensures the language and style match what real humans have used before.

2. The "Taste Test" (Multiple LLMs)

Instead of asking just one AI to write the note, CoRaCommit asks several different AIs to write it at the same time.

The Analogy: Imagine you are ordering dinner. Instead of asking one waiter for a recommendation, you ask three different chefs to cook a small sample of the dish. You then get three different versions of the commit message. You can read them all, pick the one you like best, and maybe tweak it slightly. This gives you choices instead of just one guess.

3. The "Learning Coach" (Feedback-Driven Recommendation)

This is the tool's superpower. CoRaCommit doesn't just stop after you pick a message. It watches what you do.

The Analogy: Think of it like a sports coach watching a game.
- If you pick the message written by "AI Chef A" without changing it, the coach gives Chef A a high score.
- If you pick "AI Chef B" but have to rewrite half of it, the coach gives Chef B a medium score.
- If you ignore "AI Chef C" completely, the coach gives them a low score.

Over time, the tool learns which AI is the best at writing notes for specific types of tasks. If you are fixing a database error, it might learn that "AI Chef A" is the best for that specific job, even if "AI Chef B" is better at writing about user interfaces. It creates a personalized ranking to help you pick the right AI for the job next time.

4. The "Privacy Guard" (Preprocessing)

Before sending your code to the cloud to be processed, the tool does a quick cleanup on your computer. It strips out sensitive things like passwords or secret keys and removes binary files (like images) that don't need to be read. It ensures that only the necessary text is sent out, keeping your code safe.

The Results

The authors tested this tool against other popular tools that do similar things. They used 945 real-world code changes to see which tool wrote the best notes.

The Verdict: CoRaCommit won. It produced notes that were much closer to what human experts would write.
Why? Because it didn't just guess; it used the "Smart Librarian" to find similar past examples, giving the AI a much better context to work with.

In Summary

CoRaCommit is a VS Code extension that helps developers write better project history notes by:

Looking up similar past notes to use as a reference.
Asking multiple AIs to write drafts so you can choose the best one.
Learning from your choices to recommend the best AI for future tasks.
Cleaning your code before sending it out for privacy.

It turns the boring task of writing commit messages into a smarter, faster, and more accurate process.

Technical Summary: CoRaCommit

Problem Statement

Commit messages are critical textual artifacts for version control, code review, and historical tracking, yet they are frequently authored manually, leading to inconsistent quality, missing details, and non-uniform expression. While existing VS Code extensions (e.g., Auto Commit Message, AI Commit, Commit Sage) attempt to automate this process, they suffer from two primary limitations:

Lack of Contextual Reference: Most tools rely solely on the current code diff as input, failing to leverage similar historical commit exemplars that could provide relevant expression patterns and terminology.
Absence of Adaptive Recommendation: Existing tools generally do not support user feedback-driven evaluation or dynamic recommendation when multiple Large Language Models (LLMs) are available, leaving developers without guidance on which model might perform best for a specific task.

Methodology

The paper proposes CoRaCommit, a VS Code extension designed to address these limitations through a retrieval-augmented generation (RAG) framework and an adaptive evaluation mechanism. The system architecture consists of a lightweight VS Code frontend, a Node.js orchestration layer, and a Python-based backend for retrieval and evaluation.

Core Workflow

Staged Code Diff Extraction: The extension extracts the staged code diff from the Git repository on the client side. It performs preprocessing to normalize line endings, filter binary patches, mask sensitive information (API tokens), and control length to ensure privacy and efficiency.
Hybrid Retrieval-Augmented Generation:
- The system targets the ApacheCM commit dataset.
- It employs a hybrid retrieval strategy: First, the code diff is encoded into dense vectors (using CodeBERT or jina-embeddings-v2-base-code) for semantic recall. Second, candidates are reranked using BM25 scores to enhance sensitivity to local features like function and path names.
- The top- $k$ similar commit exemplars (historical diff-message pairs) are retrieved and injected into the prompt as context for the LLM.
Concurrent LLM Invocation: The system supports the parallel invocation of multiple configured LLMs (e.g., ChatGPT, DeepSeek, Qwen). It aggregates results from all selected models to present diverse candidate commit messages to the user.
Dynamic Evaluation and Recommendation:
- Feedback Loop: When a developer selects or edits a candidate message, the system records this interaction as feedback.
- Scoring Mechanism: An asynchronous background process updates LLM scores based on:
  - Similarity Score: Weighted combination of semantic and lexical similarity between the generated candidate and the adopted message.
  - Preference Score: Based on user behavior (direct adoption vs. edited adoption vs. rejection).
- Ranking: Scores are normalized and updated using an Exponential Moving Average (EMA) to maintain both global LLM rankings and task-specific (exemplar-level) recommendations.

Key Contributions

The paper outlines three main contributions:

Implementation of CoRaCommit: A VS Code extension integrating the CoRaCMG framework. It features staged diff extraction, hybrid retrieval from ApacheCM, and retrieval-augmented generation to deliver candidate messages within standard development workflows.
LLM Evaluation and Recommendation Mechanism: A novel system that updates both global and exemplar-level LLM scores based on generation results and user feedback. This supports both a global leaderboard and task-specific LLM suggestions.
Comparative Evaluation: A rigorous experimental comparison against existing VS Code CMG extensions to validate the quality of the generated commit messages.

Experimental Results

The authors evaluated CoRaCommit on 945 commits randomly sampled from the ApacheCM dataset, comparing it against Auto Commit Message, AI Commit, and Commit Sage. The evaluation used four standard Natural Language Generation (NLG) metrics: BLEU, CIDEr, METEOR, and ROUGE-L.

Performance: CoRaCommit outperformed all baselines across all four metrics.
- BLEU: 5.28 (vs. 2.13, 1.66, 2.39)
- CIDEr: 0.432 (vs. 0.036, 0.002, 0.021)
- METEOR: 21.40 (vs. 1.31, 17.12, 19.06)
- ROUGE-L: 20.92 (vs. 6.37, 11.18, 14.18)
Analysis: The authors attribute the significant improvement, particularly in CIDEr and ROUGE-L, to the retrieval of similar commit exemplars. These external references provide the LLM with additional modification expressions and task clues, leading to better summarization of core content and semantic consistency compared to models relying solely on the current code diff.

Significance and Claims

The paper claims that CoRaCommit demonstrates the effectiveness of retrieval-augmented context for commit message generation. By integrating similar historical commits, the tool helps LLMs generate messages that are more accurate and complete regarding code changes. Furthermore, the feedback-driven recommendation component is presented as a practical solution for helping developers select the most appropriate LLM for specific tasks, moving beyond static model selection.

The authors acknowledge limitations, noting that the system's effectiveness is currently dependent on the coverage and quality of the ApacheCM dataset and that retrieval efficiency may require optimization for larger datasets or high-concurrency scenarios. Future work is proposed to expand the exemplar corpus and refine recommendation strategies.