Can Code Evaluation Metrics Detect Code Plagiarism?
This paper empirically demonstrates that Code Evaluation Metrics, particularly CrystalBLEU, can effectively detect source code plagiarism across various modification levels and often outperform or rival dedicated plagiarism detection tools like Dolos and JPlag, especially when preprocessing is applied.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a teacher grading hundreds of programming assignments. You suspect some students have copied each other's work, but they've tried to hide it by changing the font, renaming variables, or rearranging the order of their code. This is the problem of Source Code Plagiarism.
For years, teachers have used special "detective tools" (like JPlag and Dolos) to catch these cheaters. These tools are like specialized forensic scanners designed specifically to find copied code.
But recently, a new type of tool has become popular in the tech world: Code Evaluation Metrics (CEMs). Think of these as "quality checkers" originally built for a different job: comparing a computer's generated code against a perfect reference code to see how well the computer did its homework. They weren't built to catch cheaters; they were built to grade AI.
This paper asks a simple question: Can these "quality checkers" (CEMs) also act as "plagiarism detectives"?
Here is the story of how the authors tested this, using simple analogies.
The Experiment: The "Copy-Paste" Levels
The researchers didn't just look at obvious copying. They tested the tools against six different levels of "hiding" the theft, ranging from easy to nearly impossible:
- Level 1 (The Cosmetic Change): Like changing the font color or adding extra spaces. (Easy to spot).
- Level 2-3 (The Renaming Game): Changing "myVariable" to "x" or swapping the order of sentences. (Medium difficulty).
- Level 4-5 (The Structural Shuffle): Taking a loop and turning it into a different type of loop, or splitting one big function into three small ones. (Hard).
- Level 6 (The Logic Rewrite): Rewriting the entire logic so it looks completely different but does the exact same thing. (Very Hard).
They tested five different "Quality Checkers" (CEMs) against the two "Forensic Scanners" (JPlag and Dolos) using two large sets of real student code.
The Tools: How They "Think"
To understand the results, it helps to know how these tools "see" code:
- The Lexical Tools (like CrystalBLEU): These look at the words (tokens). Imagine reading a book and counting how many words match. CrystalBLEU is smart; it ignores common phrases like "import java" or "public class" (the "boilerplate") so it doesn't get fooled by everyone using the same template.
- The Structural Tools (like TSED): These look at the shape of the code, like comparing the skeleton of two buildings.
- The Semantic Tools (like CodeBERTScore): These try to understand the meaning of the code, like a human reader.
The Findings: Who Won the Race?
1. The "Raw" Test (No Prep Work)
When the tools looked at the code exactly as the students submitted it (with comments, extra spaces, and templates):
- Dolos (the dedicated detective) was the overall winner. It was the most consistent at spotting cheaters.
- However, CrystalBLEU (the quality checker) came in a very close second, beating the older detective, JPlag.
- The "Team Up" Strategy: The authors tried combining the top three quality checkers into one super-tool called FusionTop3. This team-up performed almost as well as the best detective, Dolos.
2. The "Preprocessed" Test (Cleaning the Code First)
Before testing, the researchers "cleaned" the code: they removed comments, extra spaces, and standard template lines. This is like cleaning a crime scene of dust before looking for fingerprints.
- The Shift: After cleaning, the quality checkers got much better. CrystalBLEU and the FusionTop3 team actually surpassed Dolos in the overall rankings.
- Why? The cleaning process removed the "noise" that confused the quality checkers, allowing them to focus on the actual code structure.
3. The Difficulty Curve (The "L4" Wall)
This was the most critical finding.
- Levels 1, 2, and 3: All tools (both the dedicated detectives and the quality checkers) were excellent. They caught almost everyone.
- Level 4 and beyond: Performance dropped sharply for everyone.
- The Analogy: Imagine a student who copies a recipe but changes the ingredients, the cooking method, and the order of steps, yet the dish tastes the same. Even the best tools struggled to say, "This is the same recipe!"
- The Exception: CrystalBLEU remained surprisingly strong even at the hardest levels (Level 6), while others struggled more.
The Limitations: Where They Failed
- CodeBERTScore (The "Meaning" Reader): This tool failed miserably. It gave high similarity scores to everything, even code that wasn't copied. It was like a security guard who thinks everyone in the building is a thief because they all look human.
- TSED (The "Skeleton" Reader): It struggled when students changed the syntax (the words) but kept the logic. It couldn't handle the "renaming game."
- The "L4" Wall: No tool, whether a dedicated detective or a quality checker, could reliably catch the most complex forms of plagiarism (Levels 4, 5, and 6) without a high rate of false alarms.
The Conclusion: What Does This Mean?
The paper concludes that Code Evaluation Metrics can indeed detect plagiarism, and in some cases (especially after cleaning the code), they perform just as well as, or even better than, the specialized tools built for the job.
However, they are not a magic bullet.
- Best Use: They are great for screening. They can quickly rank submissions to show a teacher, "Hey, look at these 10 pairs first; they look suspicious."
- Not for Final Judgment: Because they struggle with complex logic changes (Level 4+), a human teacher should always make the final call.
The Takeaway: You don't need to throw away your specialized plagiarism detectors. Instead, you can use these new "quality checkers" as a powerful, complementary tool—especially if you clean the code first—to help catch cheaters who try to hide their tracks.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.