🤖 AI

Automated Classification of Source Code Changes Based on Metrics Clustering in the Software Development Process

This paper presents an automated method for classifying source code changes by clustering eleven change metrics using k-means with cosine similarity, followed by expert mapping, which was validated on five software systems to achieve high classification purity and reduced review time.

Original authors: Evgenii Kniazev

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Evgenii Kniazev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head chef of a massive, busy kitchen. Every day, hundreds of new ingredients arrive, old ones are thrown away, and recipes are tweaked. Your job is to keep the kitchen running smoothly, but you can't possibly taste every single change to see if it's a new dish, a fix for a burnt sauce, or just someone rearranging the spice rack.

This is exactly the problem software developers face. They write millions of lines of code, and every day, thousands of tiny "changes" happen. Some changes add new features (like a new dish), some fix bugs (like fixing a burnt sauce), some clean up old code (like throwing away expired ingredients), and some just reorganize things (like moving the spice rack).

Checking every single change manually takes forever. This is where Evgenii Knyazev's research comes in. He created a smart, automated "sous-chef" that helps organize these changes so human experts don't have to do all the heavy lifting.

Here is how his method works, broken down into simple steps:

1. The "Fingerprint" of a Change

Instead of reading the code like a book, Knyazev's system looks at the statistics of the change. Think of every code change as a person walking into a room. Instead of asking them their name, the system measures their "fingerprint" using 11 different rulers:

How many lines of text did they add?
How many did they delete?
Did they make the logic more complicated?
Did they add new "doors" (interfaces) or remove old ones?

These measurements create a unique metric vector—a digital ID card for that specific change.

2. The "Party Game" of Grouping (Clustering)

Once the system has the ID cards for thousands of changes, it plays a game of "Group the Similar."

Imagine a giant dance floor.
The system uses a mathematical rule called k-means (think of it as a very strict dance instructor) to group people who look similar.
Crucially, it uses Cosine Similarity. Imagine two people are walking in the same direction but one is walking fast and the other slow. A normal ruler might say they are different because of the speed. But Knyazev's system only cares about the direction. If two changes are "walking in the same direction" (e.g., both are adding complexity), they get grouped together, even if one is a huge change and the other is tiny.

3. The Human "Captain"

Here is the clever part: The computer is great at grouping, but it doesn't know why the groups exist. It just sees that "Group A" looks like "Group B."

The computer automatically sorts the thousands of changes into, say, 12 different piles (clusters).
Then, a human expert (the "Captain") only needs to look at a tiny sample from each pile.
The Captain says, "Ah, this pile is mostly 'Bug Fixes'," and "That pile is mostly 'New Features'."
Once the Captain labels the piles, the computer automatically labels every single change in those piles.

Why This is a Big Deal

In the past, if a developer wanted to review 2,000 changes, they might have to read all 2,000.
With this method:

The computer does the heavy lifting of sorting.
The human expert only needs to review a tiny fraction (in the paper's example, they only looked at 73 changes out of 2,069 to teach the computer how to sort the rest).
The result is that 75% of the changes are correctly sorted automatically.

The Real-World Test

Knyazev tested this on real software projects, including Subversion (a tool for managing code) and NHibernate (a tool for connecting databases).

The Result: The system worked like a charm for standard changes. It successfully reduced the time experts spent reviewing code by a huge margin.
The Limitation: If a single change is a "monster" that does everything at once (adds a feature, fixes a bug, and cleans up code all in one go), the system gets confused. It's like a person who is wearing a chef's hat, a firefighter's jacket, and a construction vest all at once—the system doesn't know which group to put them in.

The Bottom Line

This paper proposes a hybrid approach: Let the computer do the boring, repetitive sorting based on math, and let the human expert do the high-level decision-making. It's not about replacing the human; it's about giving the human a superpower to see the big picture without getting lost in the weeds.

In short: It turns a chaotic pile of code changes into organized, labeled folders, saving developers hours of work and letting them focus on building better software.

1. Problem Statement

Modern software development involves managing vast volumes of source code, making manual code review and quality assurance time-consuming and prone to bottlenecks. While version control systems provide a history of changes, reviewing every modification is impractical due to time constraints.

The Challenge: Developers need to distinguish between different types of changes (e.g., bug fixes, new features, refactoring, formatting) to prioritize reviews, control development processes, and assess quality.
Current Limitations: Existing automated classification methods often rely on:
- Heuristic keyword searches (unreliable for diverse data).
- Syntactic analysis (high complexity, language-dependent, and inflexible).
- Data mining requiring large, pre-labeled training sets (difficult to construct).
- Most methods produce fixed classifications that cannot be easily reconfigured for different project needs.
Goal: Develop a method to partially automate the classification of source code changes to reduce the manual effort required by experts while maintaining high accuracy.

2. Methodology

The proposed method operates on a two-step hybrid approach: automatic clustering of changes based on metrics, followed by expert mapping of clusters to semantic classes.

A. Input Data & Metrics

For every source code change ( $\delta$ ), a metric vector is computed. The paper utilizes 11 specific source code metrics covering size, complexity, and structural changes:

LOC: Lines of code added ( $LOC^+$ ), deleted ( $LOC^-$ ), and modified ( $LOC^*$ ).
CC: Cyclomatic complexity of added ( $CC^+$ ), deleted ( $CC^-$ ), and modified ( $CC^*$ ) code.
Structural Counts: Files modified ( $F^*$ ), Interfaces added/deleted ( $I^+, I^-$ ), and Classes/Structures added/deleted ( $CS^+, CS^-$ ).

The change is represented as an edit script (insertions, deletions, replacements) to calculate these metrics.

B. Clustering Algorithm

Algorithm: k-means clustering.
Distance Metric: Cosine Similarity ( $\rho$ $ρ$ ) is used instead of Euclidean distance.
- Reasoning: Euclidean distance groups changes based on the scale (magnitude) of the change. Cosine similarity groups changes based on the direction (proportional composition) of the metric vector. This allows the system to distinguish between a "large bug fix" and a "small bug fix" as belonging to the same class, provided their metric ratios are similar.
Process:
1. Compute metric vectors for all changes.
2. Initialize $k$ clusters (initially set to the number of expert classes $n$ ).
3. Iteratively assign changes to the nearest cluster center based on cosine similarity.
4. Quality Check: A density functional ( $I$ ) measures the "tightness" of clusters. If $I$ is below a threshold, $k$ is incremented, and clustering is re-run.

C. Expert Mapping & Automation

Clustering: The system automatically groups thousands of changes into $k$ clusters.
Mapping: An expert reviews a small, representative sample of changes from each cluster (specifically those closest to the cluster center).
Assignment: The expert assigns the dominant class of the sample to the entire cluster (e.g., if 80% of the sample in Cluster A is "Bug Fixing," the whole cluster is labeled "Bug Fixing").
Classification: All changes in that cluster are automatically classified.

D. Quality Evaluation

The method is evaluated using Purity ( $P_C$ ) and Entropy ( $E_C$ ) on a verification set, validated via bootstrap resampling to ensure statistical significance given limited expert-labeled data.

Purity: Ratio of correctly classified changes in a cluster.
Entropy: Measure of disorder/mixing within clusters (lower is better).

3. Key Contributions

Feasibility Justification: Proved that partial automation of change classification is possible without a pre-labeled training set, relying instead on unsupervised clustering of metrics.
Metric-Based Clustering Strategy: Justified the use of k-means with cosine similarity for code changes, demonstrating that it effectively separates change types regardless of the magnitude of the code modification.
Hybrid Workflow: Developed a workflow that minimizes expert time by requiring manual classification only for a small subset of changes (cluster centers) rather than the entire dataset.
Software Implementation: Created a tool integrating metric computation (via edit scripts), clustering (using the CLUTO toolkit), and classification logic.

4. Experimental Results

The method was validated on five software systems, including two open-source projects (Subversion and NHibernate) and three industrial systems developed by Transas Technologies.

Case Study (NHibernate):
- Dataset: 2,069 total changes.
- Expert Effort: Experts manually classified only 73 changes (to map clusters) and a verification set of 68 changes.
- Performance:
  - Classification Purity ( $P_C$ ): $0.75 \pm 0.05$ (75% of changes correctly classified).
  - Entropy ( $E_C$ ): $0.37 \pm 0.06$ .
- Efficiency: The method reduced the manual classification workload significantly (from 2,069 to ~73 changes for mapping).
Observations:
- The method performs best on uniform changes (e.g., pure refactoring or pure bug fixes).
- Performance degrades when changes are heterogeneous (combining multiple types of modifications in a single commit), particularly for the "Refactoring" class, which showed lower separation in clusters.

5. Significance and Impact

Process Control: Enables development managers to filter changes (e.g., "Show me only bug fixes" or "Flag all refactoring") without reading every commit message.
Quality Assurance: Reduces the time required for code review by prioritizing high-risk or complex changes, allowing early detection of architectural or algorithmic errors.
Scalability: The approach is language-agnostic (based on structural metrics rather than syntax trees) and adaptable to different classification schemas by simply changing the expert mapping step.
Practical Application: The method has been successfully deployed in industrial settings (Transas Technologies) for systems like Navi-Manager and LRIT, proving its viability in real-world software engineering environments.

In conclusion, Knyazev's work demonstrates that combining software metrics with unsupervised clustering provides a robust, scalable, and efficient solution for automating the initial stages of source code change classification, significantly alleviating the burden on human reviewers.