Conventional Commit Classification using Large Language Models and Prompt Engineering

This paper demonstrates that training-free large language models, particularly DeepSeek-R1-32B using few-shot prompting, can effectively classify conventional commits from code diffs, offering a practical alternative to traditional supervised machine learning approaches.

The Gist

Imagine you are the manager of a massive, chaotic library where thousands of books are being added every day. To keep things organized, the library has a strict rule: every new book must have a specific label on its spine (like "New Feature," "Bug Fix," or "Documentation") so that robots can automatically sort them, update the catalog, and tell visitors what's new.

However, in reality, the people adding the books often ignore the rules. They scribble messy notes like "fixed the thing" or "changed some code," making it impossible for the robots to know what category the book belongs to.

This paper is about teaching a super-smart robot (an AI) to read those messy notes and figure out the correct label, without having to spend years studying thousands of examples first.

The Problem: Messy Notes vs. Strict Rules

In software development, programmers write "commit messages" (notes) every time they save changes to their code. The industry has a standard format called Conventional Commits that acts like a strict filing system. It requires notes to start with a specific tag (e.g., feat:, fix:).

But humans are messy. They often forget the tags. Traditionally, to fix this, researchers would build a custom robot by feeding it thousands of labeled examples (like a student memorizing a textbook). This takes a lot of time and data.

The New Approach: The "Prompt" Strategy

Instead of training a new robot from scratch, the authors asked: Can we just give a very smart, pre-existing AI a set of instructions (a "prompt") to do the job?

They treated the AI like a brilliant intern who already knows a lot about language but needs to know exactly what task to do. They tested three different ways of giving instructions:

1. Zero-Shot (The "Just Tell Me" Approach):

   • The Analogy: You walk up to the intern and say, "Here is a messy note. Please tell me what category it belongs to based on the rules." You give no examples.
   • Result: The intern guesses, but often gets it wrong because they don't know exactly what you want.

2. Few-Shot (The "Show Me Examples" Approach):

   • The Analogy: You say, "Here is a messy note that means 'New Feature.' Here is another that means 'Bug Fix.' Now, look at this new messy note and tell me what it is." You show the intern a few clear examples first.
   • Result: This worked the best. The intern understood the pattern quickly and sorted the books accurately.

3. Chain-of-Thought (The "Think Out Loud" Approach):

   • The Analogy: You say, "Before you give me the answer, please write down your step-by-step reasoning: 'I see the word 'fix', so I think it's a bug...'"
   • Result: Surprisingly, this didn't help. For this specific task of sorting labels, making the intern "think out loud" just added extra steps without making the final answer better. It was like asking a librarian to write an essay before shelving a book; it slowed them down without improving the result.

The Contenders: How Big Does the Brain Need to Be?

The researchers tested three different "interns" (AI models) of varying sizes:

• Mistral-7B: A medium-sized brain (7 billion parameters).
• LLaMA-3-8B: A slightly larger brain (8 billion parameters).
• DeepSeek-R1-32B: A giant brain (32 billion parameters).

The Finding: The bigger brain won. The DeepSeek-R1-32B was the most accurate at reading the messy notes and finding the right label. This suggests that for this kind of task, having a larger, more powerful AI model makes a real difference.

The Bottom Line

The paper concludes that you don't need to build a custom machine learning model from scratch to organize messy software notes. Instead, you can use a powerful, pre-existing AI and simply give it a few good examples (Few-Shot prompting) to get the job done.

• Best Strategy: Show the AI a few examples first.
• Best AI: The biggest, most powerful model available.
• Waste of Time: Making the AI write a long reasoning process before answering.

This approach saves time and effort because it skips the need to collect and label thousands of training examples, letting developers automate their file organization immediately.

Technical Summary

Technical Summary: Conventional Commit Classification using Large Language Models and Prompt Engineering

Problem Statement

Conventional Commits provide a structured format for commit messages (e.g., type: description), enabling automation tools like changelog generators and semantic versioning systems. However, developers frequently write unstructured messages, necessitating automatic classification to restore this structure. Traditional approaches rely on machine learning (ML) and deep learning (DL) models trained on large, labeled datasets. These methods require significant data preprocessing, feature engineering, and frequent retraining when project terminology or classification taxonomies evolve. This paper investigates a training-free alternative: leveraging Large Language Models (LLMs) via prompt engineering to classify commit messages without the overhead of curating labeled training data or fine-tuning models.

Methodology

The study employs a training-free framework where LLMs classify code changes directly based on prompts, without model fine-tuning.

• Dataset: A balanced dataset of 3,200 commits was extracted from the open-source InfluxDB repository. The dataset includes commit messages and their corresponding code diffs (changes). The data is approximately balanced across commit types (e.g., feat, fix, docs, refactor, test, chore).
• Models Evaluated: Three open-source LLMs of varying parameter scales were tested:
  • Mistral-7B-Instruct (7B parameters)
  • LLaMA-3-8B (8B parameters)
  • DeepSeek-R1-32B (32B parameters)
• Prompting Strategies: Three distinct prompting approaches were evaluated:
  1. Zero-Shot: The model is provided with class definitions and instructions but no examples.
  2. Few-Shot: The model is provided with representative labeled examples of commit messages and code diffs alongside instructions.
  3. Chain-of-Thought (CoT): The model is instructed to generate intermediate reasoning steps (identifying cues and mapping them to types) before outputting the final classification.
• Evaluation: The models were evaluated using standard classification metrics: Accuracy, Precision, Recall, and F1-score. The ground truth was derived from the existing labels in the InfluxDB repository.

Key Results

The empirical evaluation yielded the following findings:

1. Prompting Strategy Performance:

   • Few-shot prompting consistently achieved the highest accuracy across all models. For instance, DeepSeek-R1-32B achieved 0.5749 accuracy with few-shot prompting, compared to 0.5652 (Zero-shot) and 0.5497 (CoT).
   • Chain-of-Thought (CoT) did not yield additional gains for this specific classification task. In several cases, CoT performance was intermediate or lower than few-shot, suggesting that structured classification relies more on pattern recognition than multi-step reasoning.
   • Zero-shot generally resulted in the lowest or near-lowest performance, particularly for smaller models like Mistral-7B-Instruct (0.3356 accuracy).

2. Model Scale Impact:

   • DeepSeek-R1-32B (32B parameters) demonstrated the strongest overall performance across all metrics and prompting strategies, achieving an average accuracy of 0.563.
   • LLaMA-3-8B recorded the lowest performance (average accuracy of 0.372).
   • Mistral-7B-Instruct showed intermediate performance.
   • The results suggest that model scale plays a meaningful role in the ability to comprehend context and classify code changes accurately.

3. Aggregated Metrics:

   • Across all models, Few-shot prompting achieved the highest average accuracy (0.531) and F1-score (0.468).
   • Zero-shot prompting achieved the highest average precision (0.508) but lower recall.

Key Contributions

The paper outlines three primary contributions:

1. Novel Approach: It proposes a training-free approach to conventional commit classification using LLMs and prompt engineering, contrasting with traditional ML/DL pipelines that require labeled datasets.
2. Empirical Evaluation: It provides a comprehensive empirical evaluation of this approach using a broad dataset (3,200 commits) and multiple state-of-the-art open-source models.
3. Practical Guidance: It discusses the effectiveness of prompt-based classification, identifying that few-shot prompting is superior to zero-shot and CoT for this task, and highlighting the trade-offs within the software engineering framework.

Significance and Claims

The authors position this work as a practical guide for researchers and practitioners aiming to automate commit classification without the resource-intensive process of data curation and model training.

• Prompt Design is Critical: The study emphasizes that prompt design significantly impacts performance, often exceeding the influence of the model architecture itself. Specifically, including clear, diverse examples (few-shot) acts as an implicit training signal.
• Limitations of Reasoning: The paper modestly claims that while CoT is effective for complex reasoning, it does not necessarily improve performance for structured classification tasks like commit typing, where pattern recognition is paramount.
• Scope of Comparison: The authors explicitly state that they do not aim to outperform traditional ML/DL baselines directly. Instead, the study focuses on analyzing how different prompt designs influence LLM-based classification. The claim that "larger models perform better" is strictly relative to the evaluated LLMs and not a comparison against external supervised baselines.
• Future Directions: The paper suggests future work could involve testing generalization across more diverse repositories, applying automated prompt optimization, and exploring hybrid approaches combining LLMs with traditional ML techniques. It also notes the potential for applying these findings to automated release note generation and bug prediction.