Technical Summary: Assessing Language Models for Salient Class Identification

Problem Statement

Code review is a critical practice for maintaining software quality, yet it becomes significantly challenging when a single commit modifies multiple classes. In such scenarios, reviewers must reconstruct the overall intent of the change from scattered modification fragments. To assist reviewers, the concept of Salient Classes was introduced: one or more primarily modified classes that drive the core intent of a commit and induce modifications in other classes (often referred to as "ripple effects").

Accurate automated identification of these salient classes is difficult. Existing state-of-the-art (SOTA) approaches, such as ISC and GBSCI, rely on complex program-analysis procedures. These methods require Abstract Syntax Tree (AST) parsing, extraction of inter-class relations, construction of dependency graphs, and handcrafted feature engineering. These dependencies create significant barriers: they fail in repositories with incomplete builds or mixed languages, incur high costs for adaptation to new languages, and hinder integration into lightweight review workflows.

This paper investigates whether Language Models (LMs), specifically Large Language Models (LLMs) and Small Language Models (SLMs), can identify salient classes directly from raw code diffs without feature engineering, graph construction, or supervised training.

Methodology

Dataset Construction: ApacheJavaCM

To evaluate LMs, the authors constructed a new dataset, ApacheJavaCM, derived from the ApacheCM dataset.

• Scope: Limited to Java projects to ensure compatibility with existing baselines (which rely on Java AST parsing) and to maintain consistency with prior research settings.
• Labeling Rules: A class is labeled as salient (positive) if its name is explicitly mentioned in the commit message. Classes modified in the commit but not mentioned in the message are labeled non-salient (negative).
• Filtering Pipeline: The dataset underwent an 8-step filtering process to exclude commits with no Java classes, commits where all classes were mentioned (degenerate cases), and commits lacking a clear contrast between positive and negative samples.
• Final Statistics: The dataset contains 7,911 commits and 25,914 labeled classes (8,382 positive, 17,532 negative), with an average of 3.276 classes per commit.

Experimental Design

The study systematically evaluated the performance of LMs against two baselines:

1. Most-Modified-Class: A heuristic baseline selecting the class with the most modified lines.
2. ISC: The strongest reproducible SOTA baseline based on static analysis and random forest classification.

Models Evaluated:

• LLMs: GPT-5.4 (Closed-source) and DeepSeek-V3.2 (Open-source).
• SLM: Qwen3.5-9B (Open-source, 9B parameters).

Prompting Strategies:

• Zero-shot: Direct task definition and salient class definition.
• Few-shot: Inclusion of two labeled examples to demonstrate the distinction between salient classes and ripple effects.
• Chain-of-Thought (CoT): Explicit reasoning steps requiring the model to list changes, infer intent, and separate causes from consequences before outputting the answer.

Evaluation Metrics:
Given the class imbalance (approx. 1:2.1), the study prioritized Commit Accuracy (CmtAccuracy) (a commit is correct only if all its modified classes are correctly classified) and Matthews Correlation Coefficient (MCC). Other metrics included Precision, Recall, Accuracy, and F1 score.

Key Contributions

1. First Systematic Empirical Study: This work provides the first comprehensive evaluation of LMs (both LLMs and SLMs) for salient class identification directly from code diffs, bypassing static analysis.
2. New Benchmark (ApacheJavaCM): The construction of a curated dataset of 7,911 commits with 25,914 labeled classes, specifically designed for this task and compatible with existing Java-based baselines.
3. Comprehensive Evaluation: A thorough analysis covering nine configurations (3 models × 3 prompting strategies) across three dimensions of commit characteristics (total classes, salient class count, and diff token length).
4. Qualitative Failure Analysis: A detailed investigation into the root causes of LM errors, categorizing them into implicit dependency issues, attention diffusion in long diffs, and confusion between root causes and reactive implementations.

Results

Performance vs. Baselines (RQ1)

• Superiority of LMs: All three LMs substantially outperformed both the heuristic baseline and the ISC SOTA baseline across all metrics.
• Magnitude of Improvement: Even the weakest LM (DeepSeek-V3.2) improved upon the ISC baseline by 18.52% in CmtAccuracy and 26.19% in MCC.
• Efficiency: Unlike ISC, which requires AST parsing and feature extraction, LMs achieved these results using only raw code diffs.

Impact of Model Selection and Prompting (RQ2)

• LLM vs. SLM: The 9B-parameter open-source SLM (Qwen3.5-9B) achieved performance comparable to the closed-source LLM (GPT-5.4). Under few-shot prompting, Qwen3.5-9B reached a CmtAccuracy of 0.7232, nearly matching GPT-5.4's 0.7204.
• Prompting Strategies:
  • LLMs (GPT-5.4, DeepSeek-V3.2): Performance was relatively stable across zero-shot, few-shot, and CoT strategies. CoT did not consistently improve results.
  • SLM (Qwen3.5-9B): Benefited significantly from few-shot prompting, achieving its best results with this strategy. CoT actually yielded the lowest performance for the SLM.
• Implication: Lightweight, locally deployable SLMs are sufficient for this task, offering a practical alternative to closed-source LLMs to reduce costs and privacy risks.

Impact of Commit Characteristics (RQ3)

• Total Classes: Performance did not decline monotonically with the number of classes. Instead, it followed a U-shaped pattern, with medium-sized commits (5–10 classes) being the most challenging.
• Salient Class Count: Performance dropped significantly as the number of salient classes per commit increased. Commits with multiple salient classes remain a primary challenge for all methods.
• Diff Token Length: As diff length increased, CmtAccuracy declined (monotonically), indicating that longer diffs make it harder to identify the complete set of salient classes, likely due to attention diffusion.

Failure Analysis (RQ4)

Qualitative analysis identified three main failure modes:

1. Implicit Dependency: LMs missed root classes when dependencies were not explicit in the diff, often selecting downstream classes with larger modification volumes instead.
2. Attention Diffusion: In long diffs, key semantic signals were diluted by formatting or peripheral changes, causing the model to miss critical classes.
3. Reactive Implementation Confusion: LMs confused the "repair carrier" (classes modified to implement a fix) with the "root cause" (the class motivating the change).

Significance and Claims

The paper claims that Language Models represent a viable, lightweight alternative to traditional static analysis methods for salient class identification. Key takeaways include:

• Feasibility: LMs can identify salient classes directly from code diffs without the overhead of repository-level static analysis, graph construction, or supervised training.
• Practicality of SLMs: Open-source SLMs (e.g., Qwen3.5-9B) can achieve performance comparable to large closed-source LLMs, making them suitable for industrial deployment where cost and data privacy are concerns.
• Limitations: While effective, LMs still struggle with commits containing multiple salient classes or very long diffs, and they lack the structural context (e.g., call graphs) that static analysis provides for resolving implicit dependencies.

The authors conclude that while LMs offer a significant step forward in reducing setup costs and improving accessibility for code review tools, future work should explore integrating structured program context (like call graphs) to address the remaining failure modes.