💻 computer science

ConcernBERT: Learning Responsibilities Using Class Membership

This paper introduces ConcernBERT, a BERT-based embedding model trained with triplet loss on a large-scale dataset of over two million Java files to effectively learn software responsibilities and class memberships, thereby outperforming existing models in tasks like architecture recovery and extract class refactoring.

Original authors: J. Lefever, J. Xu, Y. Cai, R. Kazman, E. Pisch

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: J. Lefever, J. Xu, Y. Cai, R. Kazman, E. Pisch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a massive, chaotic library where books have been thrown onto shelves without any order. Some books are about cooking, others about space travel, and some are just random pages torn from different manuals. Your job is to figure out which pages belong together to rebuild the original books.

In the world of computer programming, this is a common problem. Programmers write code in "classes" (think of these as the books). A well-designed class should only do one specific job (like a cookbook only having recipes). But over time, classes can get messy, mixing unrelated tasks together. This is called a "God Class," and it's hard to fix because it's difficult to tell which lines of code belong to which job.

For decades, software engineers have tried to use computers to sort this mess out. The old way was to look at the words (tokens) used in the code. If two pieces of code both used the words "log," "error," and "file," the computer assumed they were related.

The Problem with the Old Way
The paper argues that looking just at words is like trying to sort a library by looking only at the first letter of the title.

Scenario A: Two books both have "Space" in the title. They are likely about space. (Good match).
Scenario B: One book is about "Space Travel" and another is about "The Space Between Stars." They share the word "Space," but they are totally different topics. (Bad match).
Scenario C: One book is about "Cooking" and another is about "Chemistry." They don't share any words, but they both involve mixing ingredients. (Missed connection).

The old computer models were getting fooled by Scenario B and missing Scenario C. They were too focused on the surface-level vocabulary.

The New Solution: ConcernBERT
The authors created a new AI model called ConcernBERT. Instead of just reading the words, this model learns by looking at who the code is hanging out with.

Think of it like a high school cafeteria.

The Old Model: If you see two students wearing the same "Math Club" t-shirt, the model assumes they are best friends.
ConcernBERT: It looks at who actually sits at the same lunch table. Even if two students are wearing different shirts, if they are consistently sitting together, eating the same food, and talking about the same things, the model knows they belong to the same group.

In software terms, the model was trained on millions of existing codebases. It learned a simple rule: "If a method (a line of code) is inside the same class file as another method, they probably share a responsibility."

By learning from these real-world groupings, the model learned to understand the intent or "concern" of the code, not just the words.

How They Tested It
To see if this new model worked, the researchers played a game of "Mix and Match":

They took code from two (or even 100) different classes and smashed them all into one giant, messy pile.
They asked the AI to sort the pile back into the original groups.
They compared ConcernBERT against seven other famous AI models (like CodeBERT, LSI, and LDA).

The Results
ConcernBERT was a massive winner.

In simple tests (sorting 2 classes), it was 31% better than the next best model.
In hard tests (sorting 100 classes mixed together), it was 55% better.
Even compared to its "cousin" model (CodeBERT), which has the exact same brain structure but was trained differently, ConcernBERT was nearly double as good.

The paper shows that by teaching the AI to pay attention to context (who is in the same class) rather than just vocabulary (what words are used), it can much more accurately figure out what a piece of software is actually responsible for.

What This Means (According to the Paper)
The paper concludes that ConcernBERT isn't a magic wand that automatically fixes all software. Instead, it's a powerful foundation. It provides a much smarter way to measure how "cohesive" (how well-organized) a piece of code is. This makes it a better tool for future tasks like:

Breaking down messy "God Classes" into smaller, manageable pieces.
Measuring software quality more accurately.
Helping developers recover the original architecture of old software.

In short, ConcernBERT teaches computers to understand the job of the code, not just the vocabulary, by learning from the way developers naturally group their work together.

Technical Summary: ConcernBERT

Problem Statement

The principles of separation of concerns, high cohesion, and single responsibility are foundational to software design, yet their practical application often relies on developer intuition rather than actionable metrics. A critical challenge in software engineering is the reliable identification of cohesive groups of program entities (methods and attributes) that collectively fulfill a common responsibility. This capability is essential for tasks such as architectural recovery, extract class refactoring, and cohesion measurement.

Existing approaches for identifying these groups largely rely on token-based semantic similarity (e.g., shared vocabulary). However, the paper demonstrates that token similarity is insufficient:

False Negatives: Semantically related methods may use entirely different tokens (e.g., different access control implementations).
False Positives: Methods with similar tokens may serve unrelated concerns (e.g., logging vs. payment processing).
Limitation: Purely lexical models fail to capture the underlying "concern-level" semantics that developers implicitly encode when grouping entities into classes.

Methodology

The authors propose ConcernBERT, a BERT-based embedding model designed to map program entities into a high-dimensional "concern space" where entities sharing responsibilities are positioned closely.

Core Innovations

Entity-Level Supervision via Triplet Loss: Unlike standard models trained with token-level masked language modeling (which optimize for filling in blanks based on context), ConcernBERT is trained at the entity level. It utilizes triplet loss to directly optimize the relative positioning of methods and attributes.
- Anchor: A program entity.
- Positive: Another entity from the same Java structure (class, interface, or enum).
- Negative: An entity from a different structure.
- Objective: Minimize the Euclidean distance between the anchor and positive, while maximizing the distance to the negative.
Class Membership as Context: The model treats developer-assigned class membership as an implicit label for shared responsibility. The underlying assumption is that developers group methods into classes based on shared concerns. By learning from millions of these grouping decisions across thousands of repositories, the model captures patterns of concern-level similarity that generalize even when entities do not share overlapping tokens.

Architecture and Training

Base Model: Built upon CodeBERTa-small-v1, adding a mean pooling layer to aggregate token embeddings into a single vector per entity (following the rationale of SentenceBERT).
Dataset: A large-scale dataset comprising 2,430,138 source files across 6,766 repositories. The training set includes over 1.9 million essential Java files from 5,414 repositories.
Training Procedure: Trained for three epochs on a single NVIDIA A40 GPU using mini-batch stochastic gradient descent. The model was trained for three days, processing approximately 15 million entities.

Evaluation

The authors evaluate ConcernBERT using a Class-Membership Recovery Test.

Setup: Methods from $k$ distinct classes (where $k \in \{2, 10, 50, 100\}$ ) are merged into a single unlabeled group. The model embeds these methods, and a clustering algorithm attempts to recover the original class memberships.
Baselines: The model is compared against seven existing models:
- Traditional: Latent Semantic Indexing (LSI), Latent Dirichlet Allocation (LDA), Doc2Vec.
- Transformer-based: CodeBERT, GraphCodeBERT, CodeT5+, ModernBERT.
Metrics: Performance is measured using Adjusted Mutual Information (AMI) across 12 configurations (4 clustering algorithms $\times$ 3 distance metrics).

Key Results

ConcernBERT significantly outperforms all reference models across all group sizes and clustering configurations.

Small Groups ( $k=2$ ): ConcernBERT achieves a mean AMI of 0.728, outperforming the next best model (LSI at 0.558) by 31%. Compared to CodeBERT (its architectural base), the improvement is 115%.
Large Groups ( $k=100$ ): The performance gap widens with task complexity. ConcernBERT achieves a mean AMI of 0.565, outperforming LSI by 55% and CodeBERT by 94%.
Statistical Significance: All pairwise comparisons are statistically significant ( $p < 0.01$ ). The effect sizes (Cliff's delta) are large, indicating that the improvement is not merely due to architectural differences but stems from the specialized training process.
Consistency: ConcernBERT maintains the highest and most compact distribution of AMI scores across all test sizes, whereas baseline models show significant degradation and variance as task complexity increases.

A case study on the TsFileTool class demonstrated that ConcernBERT successfully identified cohesive clusters (e.g., directory management, schema processing) that aligned with human intuition, whereas CodeBERT and LSI produced large, semantically incoherent clusters.

Significance and Claims

The paper claims that ConcernBERT provides a stronger foundation for downstream software engineering tasks by effectively encoding concern-level semantics.

Operationalizing Design Principles: The work takes a step toward transforming philosophical principles like "separation of concerns" and "single responsibility" into actionable, automated metrics and tooling.
Foundation, Not End-to-End Tool: The authors explicitly state that ConcernBERT is not intended as a complete, end-to-end refactoring or architecture recovery tool. Instead, it serves as a foundational embedding component to support these tasks.
Superiority of Training Strategy: The results confirm that architectural advances alone (e.g., using newer Transformer models) do not capture concern-level semantics as effectively as a training process specifically designed to leverage class membership context.

The paper concludes that by learning similarity directly from class membership rather than surface-level token overlap, ConcernBERT moves beyond traditional approaches to deliver more reliable insights into program responsibilities.