🤖 AI

CommitLLM: A Fine-Tuned Pipeline for Git Commit Message Generation

CommitLLM is a three-stage pipeline that fine-tunes a small language model and applies constrained decoding with deterministic post-processing to generate concise, format-compliant git commit messages from code diffs, demonstrating that structured output tasks benefit more from pipeline engineering than model optimization alone.

Original authors: Md Rafid Haque, Poojan Narendrabhai Patel, Meetkumar Vijaybhai Raychura

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Rafid Haque, Poojan Narendrabhai Patel, Meetkumar Vijaybhai Raychura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but every time you find a clue, you scribble it down on a napkin with a messy note like "fixed stuff" or "update." Years later, when you try to look back at your case file, those napkins are useless. You can't tell what actually happened, who did what, or why. This is exactly the problem software developers face with "git commit messages." When they save changes to their code, they often write lazy, unhelpful notes that turn the history of their project into a confusing mess. To fix this, researchers are building smart computer programs—specifically, a type of Artificial Intelligence called a Large Language Model (LLM)—that can read the messy changes in code and write a clear, perfect summary for them. Think of an LLM as a super-smart, well-read librarian who has read millions of books and can instantly summarize a new chapter. The big question this paper asks is: Can we teach this librarian to write short, perfect summaries using just a standard computer, without needing a massive, expensive supercomputer?

The researchers behind this study, from the University of Illinois at Chicago, built a system they call CommitLLM. Their goal was to create a tool that takes a "diff" (which is just a list of code changes) and turns it into a clean, professional note that follows strict rules, known as "Conventional Commits." These rules are like a dress code for code notes: they must start with a specific tag like "fix:" or "feat:" and be short enough to fit on a single line.

To do this, they didn't just rely on the AI to "learn" better on its own. Instead, they built a three-stage pipeline, which is like a factory assembly line for writing notes.

Stage 1: The Training (Teaching the Librarian)
First, they took a powerful but manageable AI model called Mistral-7B and gave it a crash course using a dataset of thousands of real code changes and their good summaries. They used a clever trick called QLoRA, which is like teaching the librarian new skills without making them forget everything they already know. This allowed them to train the model on a standard graphics card (an NVIDIA T4) that a regular developer could afford, rather than needing a million-dollar server farm.

Stage 2: The Constraints (The Strict Editor)
Even after training, the AI had a bad habit: it was too chatty. It would write long paragraphs, say "Here is the message you asked for," and ramble on. To stop this, the researchers added a strict rule at the output stage: "You can only write 20 words, and you must be serious." This is like putting a speed limit on the librarian's pen. If they try to write too much, the pen stops. This forced the AI to be concise.

Stage 3: The Cleanup (The Final Polish)
Finally, they added a simple, automatic cleanup step. Even with the rules, the AI sometimes slipped up and added conversational filler. The system simply chopped off any extra words before the actual message started and made sure the message followed the "Conventional Commits" format perfectly. It's like a spellchecker that not only fixes typos but also deletes the "Dear Sir" and "Sincerely" if the note is supposed to be a quick headline.

The Results: A Surprising Discovery
When they tested their system on 50 examples, the results were impressive. The "vanilla" (untouched) AI model was terrible at following the format, getting it right only 22% of the time. The trained model did a bit better, but it was still too wordy. However, the full pipeline—the trained model plus the strict rules and cleanup—achieved a 98% success rate in following the format.

Here is the most interesting part of their discovery: The training wasn't the most important part.
The researchers found that simply training the AI improved its quality score by a tiny bit (from 1.97 to 2.20 out of 5). But adding the strict rules and the cleanup step (the pipeline) boosted the score all the way to 3.68 out of 5. This suggests that for tasks where you need a specific, short format, building a smart system around the AI is actually more powerful than just trying to make the AI smarter. The AI does the thinking, but the pipeline does the formatting, and that combination is what makes the magic happen.

Why This Matters
The best part is that this entire system runs on a single, affordable graphics card. You don't need a cloud supercomputer or a monthly subscription to a big tech company. This means any developer, even a student or a hobbyist, could run this tool on their own laptop to keep their code history clean, organized, and actually useful for debugging and reviewing in the future. The paper suggests that for these kinds of structured tasks, engineering a good system is often more effective than just obsessing over the model itself.

Technical Summary: CommitLLM

Problem Statement

Developers frequently generate uninformative Git commit messages (e.g., "fix" or "update stuff"), which degrades the utility of version-control history for code review, debugging, and onboarding. While Large Language Models (LLMs) offer a potential solution, existing approaches often struggle to produce concise, format-compliant outputs without excessive verbosity or conversational filler. Furthermore, fine-tuning large models typically requires hardware beyond the reach of individual developers. This paper addresses the practical question of whether a small, open-source LLM can be fine-tuned to generate high-quality, Conventional Commits-compliant messages from code diffs using only consumer-grade hardware (a single NVIDIA T4 GPU).

Methodology

The authors propose CommitLLM, a three-stage pipeline designed to treat commit generation as a system engineering problem rather than a pure model optimization task. The approach consists of the following layers:

Fine-Tuning (Layer 1):
- Base Model: Mistral-7B-Instruct-v0.2.
- Technique: QLoRA (Quantized Low-Rank Adaptation) using 4-bit NF4 quantization to fit the model on a 16 GB VRAM GPU.
- Configuration: LoRA adapters (rank 16, $\alpha=32$ ) applied to attention and MLP projection layers.
- Dataset: CommitPackFT, trained for one epoch using the HuggingFace SFTTrainer with group_by_length enabled to reduce padding overhead.
- Scale: Approximately 42 million trainable parameters (1.1% of the total model).
Constrained Decoding (Layer 2):
- To suppress the "chatty" habits of instruction-tuned models, the inference process employs aggressive constraints:
  - max_new_tokens=20: Hard cap on output length.
  - temperature=0.1: Near-greedy decoding to prevent creative drift.
  - repetition_penalty=1.1: Prevents looping on common tokens.
Deterministic Post-Processing (Layer 3):
- A string-level cleaning pipeline applied after generation to enforce strict formatting:
  - Tag Slicing: Removes content before the final [/INST] token.
  - Guillotine Truncation: Retains only the first line of the output.
  - Prefix Stripping: If the line contains a colon, the system checks if the preceding word is a valid Conventional Commits type (e.g., feat, fix). If not, it strips the conversational prefix up to and including the colon.

Key Contributions

System-Centric Approach: The paper argues that for structured-output tasks, engineering a deterministic pipeline around a model is more effective than optimizing the model weights alone. The post-processing layers contributed significantly more to quality improvements than the fine-tuning itself.
Hardware Accessibility: The entire pipeline, including training and inference, runs on a single consumer NVIDIA T4 GPU (16 GB VRAM), making it accessible without cloud API dependencies.
Robust Evaluation Framework: The authors developed a multi-metric evaluation suite including:
- Format Compliance: A corrected regex-based metric that accounts for multi-line failures and ID-prefixed commits.
- Length Analysis: Measuring character count and adherence to the 72-character de-facto limit.
- BERTScore: Contextual embedding similarity to human ground truth.
- LLM-as-a-Judge: Using Llama-3 models (via Groq) to grade outputs on a 1–5 rubric.

Results

Evaluated on a 50-sample test slice, CommitLLM (Full Pipeline) significantly outperformed both the vanilla Mistral baseline and the raw fine-tuned model:

Metric	Vanilla Mistral	Raw Fine-Tuned	CommitLLM (Pipeline)
Format Compliance	22%	36%	98%
Avg. Length	154.8 chars	113.4 chars	37.9 chars
% Under 72 chars	~0%	38%	100%
BERTScore (F1)	0.8129	0.8259	0.8688
LLM-Judge Score	1.97	2.20	3.68

Key Findings:

Impact of Post-Processing: The jump from the raw fine-tuned model to the full pipeline yielded a +1.48 increase in LLM-Judge score, compared to only a +0.23 increase from fine-tuning alone.
Length Control: The pipeline successfully collapsed the output length distribution, eliminating the "long tail" of verbose paragraphs seen in the raw model.
Metric Sensitivity: The authors noted that an initial, flawed compliance metric artificially inflated the baseline's performance and penalized the pipeline. Correcting this metric revealed the true 98% compliance rate of the final system.

Significance and Claims

The paper claims that for tasks requiring strict structural adherence (like commit messages), the "system" matters more than the "model." The deterministic post-processing layers provided the majority of the quality gains, suggesting that treating the LLM as a component within a constrained pipeline is a superior strategy for structured generation tasks compared to solely optimizing model weights.

The authors explicitly state that the system is designed to be a practical tool for individual developers, running entirely on local consumer hardware. They acknowledge limitations, such as the small evaluation set (50 samples) due to API token limits and the potential for hallucinated issue IDs, framing these as areas for future work rather than current failures. The paper does not claim the system is perfect for all codebases but demonstrates that a high-quality, compliant commit generator is achievable with modest resources and a layered engineering approach.