💻 computer science

A Concern-Centric Empirical Evaluation of Multi-Language Code Smells: An LLM-Assisted Study of JNI Software Evolution

This paper presents a concern-centric empirical evaluation of JNI code smells using LLM-assisted analysis of 8,207 commits across 15 open-source projects, revealing that existing smell definitions cover only 36.5% of developer maintenance concerns and proposing three new smell definitions to address the identified gaps.

Original authors: Md Shahrukh Ansari, Salman Abdul Moiz

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Shahrukh Ansari, Salman Abdul Moiz

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern software often feels like a well-oiled machine, but underneath the sleek interface, it is frequently built from many different parts speaking different languages. To make a program fast, efficient, or capable of talking to specialized hardware, developers often combine code written in one language with code written in another. A common way to do this is through a bridge called the Java Native Interface, which allows a program written in Java to reach out and use powerful tools written in C or C++. While this mix of languages gives software great power, it also creates a unique kind of mess. Just as a translator might struggle to keep two different dialects consistent, the software can develop hidden flaws where the two languages fail to agree on how to share data, manage memory, or handle errors. These flaws, known in the industry as "code smells," are not bugs that crash the program immediately, but rather design choices that make the software difficult to fix, update, or understand over time. For years, experts have tried to catalog these smells, creating lists of what bad cross-language design looks like. But a critical question remained unanswered: do these lists actually match the real problems that developers face every day when they try to keep these complex systems running?

A team of researchers set out to answer this question by looking directly at the history of how software changes over time, rather than just staring at the code itself. They focused on fifteen popular open-source projects that rely heavily on this Java-to-C bridge. Instead of guessing what might be wrong, they examined thousands of updates, or "commits," that developers had made to these projects over the years. They were looking for moments where a developer had to stop and fix a specific maintenance issue related to the connection between the two languages. To handle the sheer volume of data, they used an advanced artificial intelligence tool to read the changes in the code and the notes the developers wrote about them. The AI acted as a highly skilled assistant, scanning through the modifications to identify the specific problem a developer was trying to solve, such as fixing a memory leak, securing a data transfer, or reorganizing how two parts of the system talk to each other. The researchers then manually checked a sample of these findings to ensure the AI was correct, confirming that the method was reliable.

The study uncovered a clear picture of what developers actually struggle with. They identified eleven distinct families of problems that keep coming up. The most common issues involved keeping the boundary between the two languages safe and making sure that resources, like memory or file handles, were properly cleaned up after use. These two categories alone accounted for nearly sixty percent of all the maintenance work the researchers found. This suggests that the most urgent daily work for these developers is simply keeping the connection between the languages from breaking or leaking. However, when the researchers compared these real-world problems against the existing lists of known "code smells," they found a significant gap. The current catalogs, which were largely created by experts based on theoretical design principles, only covered about thirty-six percent of the actual problems developers were fixing. In other words, the existing lists missed the majority of the work developers were doing to keep their software healthy.

The researchers realized that the missing problems were not random errors but recurring patterns that deserved their own names. They found that developers frequently had to make coordinated changes to both the Java code and the C code whenever a single detail changed, a situation that made updates slow and error-prone. They also saw cases where the software was exposing hidden internal details across the language barrier, weakening the security and organization of the system. Finally, they noticed that responsibilities were often placed in the wrong language, forcing one side of the system to constantly ask the other side to do its work, which created unnecessary complexity. Based on these repeated observations, the team proposed three new definitions for code smells that specifically address these cross-language issues. They named them Cross-Language Shotgun Surgery, describing the need to change many files at once; Cross-Language Abstraction Leakage, where hidden details are accidentally exposed; and Wrong Responsibility Allocation, where tasks are assigned to the wrong language layer.

This work shifts the focus from what experts think should be a problem to what developers are actually fixing. By listening to the history of the software itself, the researchers showed that the current understanding of multi-language design flaws is incomplete. The existing lists are good at catching simple, local errors, but they miss the deeper architectural challenges that arise when two different worlds of code try to work together. The new definitions provide a vocabulary for these hidden struggles, offering a way for developers to spot and fix these specific types of messes before they become unmanageable. The study concludes that to truly understand software quality, we must look not just at the static code, but at the long, messy history of how that code is kept alive and evolving.

Technical Summary: A Concern-Centric Empirical Evaluation of Multi-Language Code Smells

Problem Statement
Multi-language software systems, particularly those utilizing the Java Native Interface (JNI), offer significant engineering benefits but introduce complex maintainability challenges due to cross-language dependencies. While existing research has proposed catalogs of JNI-specific code and design smells based on software design principles and static analysis, a critical gap remains: it is unknown whether these predefined smell definitions correspond to the actual maintenance concerns developers encounter and address during software evolution. Developers rarely describe maintenance activities using "smell" terminology; instead, they address concrete engineering problems such as resource leaks, boundary safety, and cross-language communication. Consequently, the practical relevance and completeness of current JNI smell catalogs remain largely unexplored.

Methodology
This study adopts a concern-centric rather than a smell-centric perspective. Instead of starting with predefined smell definitions, the authors mined the software evolution history of 15 open-source JNI projects to identify recurring maintenance concerns. The methodology proceeded in three phases:

Data Collection and Extraction: The complete Git history of 15 projects (comprising 74,656 commits) was analyzed. Using specific JNI constructs (e.g., native methods, JNIEnv, JNIEXPORT), 8,207 JNI-related commit diffs were extracted.
LLM-Assisted Concern Extraction: A Large Language Model (ChatGPT) was employed to analyze commit messages and code diffs to identify the underlying maintenance concerns addressed. The LLM was instructed to focus on maintenance activities resolving existing issues, excluding feature additions or formatting changes. To ensure reliability, a random sample of 10% (65 commits) was manually validated, yielding a precision of 96.92%.
Mapping and Gap Analysis: The 652 manually validated concern-addressing commits were categorized into recurring concern families. These families were then systematically mapped to the existing catalog of JNI code and design smells by two independent researchers (Cohen's $\kappa$ = 0.913). Concerns that could not be adequately represented by existing smells were classified as "unmapped."
Candidate Smell Derivation: Recurring unmapped concerns that exhibited conceptual consistency and cross-language nature were analyzed to propose new, candidate JNI-specific smell definitions.

Key Contributions
The paper makes three primary contributions:

A Concern-Centric Methodology: A systematic framework for empirically evaluating multi-language smell catalogs by mining software evolution history, demonstrated through a large-scale study of JNI systems.
Empirical Taxonomy of JNI Maintenance: A taxonomy of 11 recurring JNI maintenance concern families extracted from 652 validated commits, providing an empirical characterization of cross-language maintenance problems.
Three Candidate JNI-Specific Smell Definitions: The proposal of three new smell definitions derived from recurring unmapped concerns:
- Cross-Language Shotgun Surgery (CLSS): Motivated by the need for coordinated modifications across Java and native code (e.g., hardcoded class names/signatures in native code).
- Cross-Language Abstraction Leakage (CLAL): Motivated by the exposure of implementation details across language boundaries (e.g., native code accessing private Java fields).
- Wrong Responsibility Allocation (WRA): Motivated by responsibilities implemented in a language where the majority of their behavioral dependencies reside (e.g., Java methods that merely orchestrate native logic).

Results

Maintenance Concerns: The study identified 11 recurring concern families. The most frequent were JNI Boundary Safety (33.80% of concern-addressing commits) and JNI Resource Leakage (25.39%), followed by categories such as Cross-Language Communication, Thread Management, and Responsibility Relocation.
Coverage Gap: The existing JNI smell catalog provides only partial coverage of developer maintenance concerns. Only 36.5% (238 out of 652) of the identified concern-addressing commits mapped to existing JNI smells.
Mapping Specifics: Existing smells like Local Reference Abuse and Memory Management Mismatch strongly supported resource management concerns. However, architectural concerns such as cross-language change propagation, abstraction boundaries, and responsibility allocation were largely uncovered by current definitions.
New Smells: The analysis of unmapped concerns justified the three candidate smells (CLSS, CLAL, WRA), which address architectural deficiencies not captured by the current catalog.

Significance and Claims
The paper claims that software evolution history serves as a valuable, complementary source of empirical evidence for evaluating and extending multi-language smell catalogs.

Validation of Existing Work: The study confirms that existing smell catalogs effectively capture implementation-level issues (resource management, exception handling) but often miss recurring architectural concerns.
Bottom-Up Discovery: The proposed methodology offers a systematic way to discover new smells based on actual developer effort rather than solely on theoretical design principles.
Generalizability: While focused on JNI, the authors assert that the concern-centric methodology is applicable to any multi-language ecosystem where evolution history can be mined.
Modest Scope: The authors explicitly state that the proposed smell operationalizations are initial detection strategies derived from representative manifestations observed in the study, not exhaustive definitions. They acknowledge that additional manifestations may be identified in future work and that the findings may not directly generalize to proprietary systems or other interoperability technologies without further investigation.

Technical Summary: A Concern-Centric Empirical Evaluation of Multi-Language Code Smells

More like this