CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation

1. Problem Statement

In software engineering, commit messages are critical for documenting code changes, facilitating collaboration, and aiding maintenance. However, manually writing them is time-consuming, often resulting in low-quality, vague, or incomplete messages. While Commit Message Generation (CMG) aims to automate this process using Large Language Models (LLMs), existing approaches face significant limitations:

• Contextual Gap: Standard LLMs often lack project-specific context, leading to generic outputs that miss specific terminologies and writing styles.
• Static Training: Learning-based methods require expensive retraining to adapt to new projects or changing coding conventions.
• Dataset Quality: Existing public datasets (e.g., CommitGen, NNGen) suffer from poor quality control, lack of metadata (e.g., repository names, timestamps), and limited language diversity (mostly Java-only).
• Retrieval Limitations: Traditional retrieval-based methods simply copy existing messages, which may not perfectly match the query, while hybrid methods often rely on complex, rigid architectures.

2. Methodology: CoRaCMG Framework

The authors propose CoRaCMG (Contextual Retrieval-augmented framework for Commit Message Generation), a training-free framework that leverages a hybrid retrieval mechanism to guide LLMs. The framework operates in three distinct phases:

Phase I: Retrieve (Hybrid Retriever)

Instead of relying on a single similarity metric, CoRaCMG employs a hybrid retriever to find the most relevant historical <diff, message> pairs from a high-quality database (ApacheCM-10K).

• Lexical Similarity: Uses BM25 to capture exact keyword matches (e.g., specific variable names, API calls).
• Semantic Similarity: Uses a pre-trained code embedding model (Jina AI jina-embeddings-v2-base-code) to capture the structural and logical intent of code changes.
• Fusion: The scores from both methods are normalized and combined with a 5:5 weighting ratio to produce a final hybrid score. The top-ranked pair is selected as the "example pair."
• Data Leakage Prevention: A mechanism ensures the retrieved example is not identical to the query diff.

Phase II: Augment (Structured Prompting)

The system constructs a structured prompt that concatenates:

1. The Query Diff (the code change to be described).
2. The Retrieved Diff (the similar historical code change).
3. The Retrieved Message (the human-written commit message for the historical change).
   This structure allows the LLM to perform In-Context Learning, explicitly observing the project's specific terminology and writing style before generating the new message.

Phase III: Generate

The augmented prompt is fed into a Large Language Model (LLM) (e.g., GPT-4o, DeepSeek-R1, Qwen-2.5). The LLM generates a commit message that aligns with the retrieved example's style while accurately describing the specific query diff. No fine-tuning of the LLM is required.

3. Key Contributions

1. Novel Framework: Introduction of CoRaCMG, a training-free, dynamic retrieval-augmented framework that effectively bridges the gap between generic LLMs and project-specific requirements.
2. High-Quality Dataset (ApacheCM): The authors constructed and utilized ApacheCM, a comprehensive dataset derived from 50 top Apache projects. It covers nine programming languages, includes rich metadata (SHA, author, files), and undergoes rigorous filtering (removing bots, merge commits, and low-quality messages), addressing the limitations of previous datasets.
3. Hybrid Retrieval Strategy: Demonstrated that combining lexical (BM25) and semantic (Dense) retrieval yields superior performance compared to single-mode retrieval, balancing exact terminology matching with semantic understanding.
4. Comprehensive Evaluation: Extensive experiments across multiple LLMs, languages, and metrics (both objective and human evaluation).

4. Experimental Results

The framework was evaluated on the ApacheCM-10K subset using five LLMs (GPT-4o, GPT-4.1, DeepSeek-V3, DeepSeek-R1, Qwen-2.5) and compared against State-of-the-Art (SOTA) baselines (NNGen, CoRec, CCT5, etc.).

• Performance Gains: CoRaCMG significantly outperformed direct LLM usage and SOTA baselines across all metrics (BLEU, ROUGE-L, METEOR, CIDEr).
  • BLEU Improvement: Average increase of 65% across models.
  • Specific Highlights:
    • DeepSeek-R1 with CoRaCMG achieved a 76% relative improvement in BLEU and 71% in CIDEr with just one retrieved example.
    • GPT-4o showed the highest relative improvement in BLEU (89%) when augmented.
• Human Evaluation: In blind evaluations by three software engineers, CoRaCMG-generated messages scored higher than both direct LLM generation and the original developer-written "ground truth" messages in Clarity, Completeness, and Correctness. This suggests that LLMs, when guided by context, can produce more informative messages than the often-vague human originals.
• Ablation & Configuration:
  • Hybrid vs. Single: Hybrid retrieval (Lexical + Semantic) consistently outperformed pure lexical or pure semantic retrieval.
  • Embedding Model: Jina embeddings were found superior to CodeBERT due to better handling of long contexts (up to 8k tokens vs. 512).
  • Number of Examples: Performance gains plateaued after three example pairs, indicating diminishing returns and potential context interference with more examples.
• Generalizability: The framework demonstrated robust performance across Java, C++, and Python, proving its language-agnostic nature.

5. Significance and Impact

• Practical Utility: CoRaCMG offers a training-free solution that can be immediately deployed in any software project without the computational cost of fine-tuning models. It adapts dynamically to new projects by simply accessing their historical commit history.
• Quality Improvement: It addresses the "vague commit message" problem by forcing LLMs to adhere to project-specific conventions, thereby improving the maintainability and readability of software repositories.
• Benchmarking: The introduction of ApacheCM sets a new standard for CMG research, providing a reproducible, multi-language, and metadata-rich benchmark that overcomes the flaws of previous datasets.
• Insight on LLMs: The study reveals that for CMG, reasoning capabilities (e.g., in DeepSeek-R1) are less critical than contextual adaptation and terminology alignment, suggesting that retrieval augmentation is a more effective lever for this specific task than model architecture complexity.

In conclusion, CoRaCMG demonstrates that integrating Retrieval-Augmented Generation (RAG) with LLMs is a highly effective strategy for automating commit message generation, significantly outperforming existing methods by leveraging the "search-and-imitate" workflow inherent in human developer practices.