💻 computer science

Detection of LLM-assisted Code Plagiarism Using k-gram Software Birthmarks

This paper demonstrates that k-gram software birthmarks based on Java opcodes remain effective in detecting LLM-assisted code plagiarism across various models and similarity measures, despite the challenges posed by code paraphrasing.

Original authors: Nikolay Fedorov, Akito Monden, Hiroki Inayoshi, Haruaki Tamada, Masateru Tsunoda

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nikolay Fedorov, Akito Monden, Hiroki Inayoshi, Haruaki Tamada, Masateru Tsunoda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a secret recipe for a delicious cake. In the old days, if someone wanted to steal your recipe, they would have to copy it word-for-word. It was easy to catch them because the handwriting or the typing would look exactly the same.

But now, imagine a super-smart robot chef (an AI) that can look at your recipe, rewrite it entirely, and give you a new version that tastes exactly the same but looks completely different. It changes the font, swaps "sugar" for "sweetener," rearranges the steps, and uses different measuring cups. To a human eye, it looks like a brand-new recipe. This is what the paper calls LLM-assisted code plagiarism.

Here is how the researchers in this paper tackled the problem of catching these "robot-chef" thieves, explained simply:

The Problem: The "Magic" Rewrite

Software developers write code (the instructions for computers). Large Language Models (LLMs) are AI tools that can take existing code and "paraphrase" it. They change the look and structure of the code so much that traditional plagiarism detectors (which look for identical words or lines) get fooled. The program still does the exact same thing, but it looks like it was written by a different person.

The Solution: The "Software Fingerprint"

The researchers didn't try to read the rewritten code like a human. Instead, they used something called Software Birthmarks.

Think of a birthmark like a unique fingerprint. Even if you change your clothes, haircut, or glasses, your fingerprint stays the same.

How they made it: They took the computer's "recipe" (the code), translated it into a list of basic machine instructions (called opcodes), and chopped them up into small chunks of 2 to 6 instructions at a time.
The Analogy: Imagine you have a sentence: "The cat sat on the mat."
- A 1-gram (chunk of 1) is just the words: "The", "cat", "sat"... (Too common; many sentences have "the").
- A 2-gram (chunk of 2) is pairs: "The cat", "cat sat", "sat on"... (More unique).
- A 3-gram is triples: "The cat sat", "cat sat on"... (Even more unique).

They turned the whole program into a bag of these chunks. Even if the AI rearranged the sentences, the specific types of chunks (the fingerprint) would likely still be there.

The Experiment: Who is the Best Thief?

The researchers tested three different "robot chefs" (AI models):

ChatGPT-5.1-Codex-Mini
DeepSeek-V4-Flash
Claude-Haiku-4.5

They took real, open-source Java programs, asked these AIs to rewrite them, and then tried to catch the AIs using their "fingerprint" method. They also tested different ways to compare the fingerprints (like counting how many chunks match vs. how close the order is).

The Findings: What Worked?

1. The Size of the Chunk Matters (The "Goldilocks" Zone)

Too small (1 chunk): It was like trying to identify a person just by the word "the." Too many people use it, so it wasn't a good fingerprint.
Too big (6 chunks): It was like trying to match a whole paragraph. If the AI moved just one sentence, the whole paragraph didn't match, and the fingerprint broke.
Just right (2 or 3 chunks): This was the sweet spot. These small groups of instructions were unique enough to identify the original code but sturdy enough to survive the AI's rewriting.

2. How You Compare Matters

Order doesn't matter: The researchers found that looking at the order of the instructions (like checking if the steps were in the exact same sequence) was a bad idea. The AI scrambled the order so much that this method failed.
Counting matters: The best method was simply counting how many unique chunks appeared in both the original and the rewritten code, regardless of order. It's like saying, "Do both recipes contain 'flour' and 'eggs'?" rather than "Did they add the eggs before the flour?"

3. Which AI was the Hardest to Catch?

ChatGPT-5.1-Codex-Mini was the sneakiest. It managed to rewrite the code so thoroughly that it was the hardest to detect, yet it still produced code that actually worked (compiled without errors).
The other two AIs (DeepSeek and Claude) were easier to catch because their rewrites left more obvious "fingerprints" behind.

The Bottom Line

Even though AI can rewrite code to look completely different, it can't change the fundamental "DNA" of how the computer thinks. By looking at small, unique patterns of machine instructions (birthmarks) and counting how many match, we can still catch the thief, even if they are using a super-smart robot to disguise the crime.

The paper concludes that this method works well against modern AI plagiarism, especially when using small chunks of code (2 or 3 instructions) and focusing on what is in the code rather than where it is.

Technical Summary: Detection of LLM-assisted Code Plagiarism Using k-gram Software Birthmarks

Problem Statement
The rapid advancement of Large Language Models (LLMs) has lowered the technical barrier to software plagiarism. Unlike traditional plagiarism involving direct code copying, modern threats involve "code paraphrasing," where an LLM rewrites existing source code to preserve functionality while significantly altering syntax, structure, identifiers, and formatting. This capability allows attackers to generate semantically identical programs that evade traditional source-code similarity detection techniques. Furthermore, real-world plagiarism often involves the redistribution of executable artifacts rather than source code, rendering source-code-based detection methods ineffective. The paper addresses the critical question of whether software birthmarks—characteristic features extracted from executable files—remain effective against LLM-assisted code paraphrasing.

Methodology
The study employs k-gram software birthmarks, a technique that represents a program as a set of unique contiguous sequences of $k$ Java bytecode instructions (opcodes). The experimental design involves the following components:

Dataset: The researchers curated a dataset from actively maintained, BSD-2-Clause licensed Java projects on GitHub. They filtered for individually compilable source files with a Logical Line Count (LLC) between 30 and 1,000 to ensure module-wise plagiarism detection. The final dataset consisted of 352 source files.
LLM Models: Three contemporary models were utilized to generate paraphrased versions of the source files:
- ChatGPT-5.1-Codex-Mini (OpenAI)
- DeepSeek-V4-Flash (DeepSeek)
- Claude-Haiku-4.5 (Anthropic)
- Prompting Strategy: The models were explicitly instructed to modify syntax, formatting, and structure while preserving functionality and ensuring the output remained compilable without new dependencies.
Birthmark Extraction: For each file, the Java compiler and javap tool were used to extract bytecode, which was then converted into opcode sequences. Unique k-grams were generated for $k$ values ranging from 1 to 6.
Similarity Measures: Five similarity functions were evaluated to compare birthmarks:
- Vector-based: Cosine similarity (Count and TF-IDF).
- Set-based: Dice index, Jaccard coefficient, and Simpson index.
- Sequence-based: Edit-distance (Levenshtein distance).
Evaluation Metrics: Performance was measured using the harmonic mean (Hmean) of Resilience (ability to detect plagiarized code) and Credibility (ability to distinguish independent code).

Key Contributions

Empirical Investigation: The paper provides the first empirical evaluation of k-gram software birthmarks specifically against LLM-generated code clones produced via paraphrasing.
Comparative Analysis: It systematically compares the effectiveness of different $k$ -gram configurations ( $k=1$ to $6$) and multiple similarity functions in the context of modern AI-assisted plagiarism.
Model-Specific Insights: The study analyzes the relative difficulty of detecting plagiarism generated by different LLMs, highlighting variations in their code transformation capabilities.

Experimental Results

Impact of $k$ (RQ1): Detection performance is highly dependent on the choice of $k$ $k$ .
- 1-gram: Performed poorly due to insufficient structural information, leading to low credibility.
- 2-gram: Achieved the highest average performance (Hmean = 0.830), offering the best balance between resilience and descriptiveness.
- 3-gram: Performed closely behind 2-gram (Hmean = 0.822).
- 4-6-gram: Performance gradually declined as larger $k$ values became too sensitive to the structural modifications introduced by LLMs.
Impact of Similarity Functions (RQ2):
- Set-based methods (Dice, Simpson, Jaccard) significantly outperformed other approaches. The Dice index achieved the highest average Hmean (0.946).
- Edit-distance (Sequence-based) performed poorly (Hmean = 0.276) because code paraphrasing frequently alters the ordering of instructions, rendering sequence-based comparisons ineffective.
- Cosine similarity (both Count and TF-IDF) yielded strong results but were generally slightly lower than the Dice index.
Impact of LLM Selection (RQ3):
- ChatGPT-5.1-Codex-Mini generated the most difficult-to-detect clones (lowest average Hmean of 0.8603), suggesting it performs more extensive transformations while maintaining compilability.
- Claude-Haiku-4.5 (Hmean = 0.9322) and DeepSeek-V4-Flash (Hmean = 0.9258) produced paraphrased code that was comparatively easier to detect.
- Notably, the model that generated the most difficult-to-detect code (ChatGPT) also had the fewest compilation errors, indicating high-quality transformation capabilities.

Significance and Claims
The paper claims that k-gram software birthmarks remain effective for detecting LLM-assisted plagiarism, provided that appropriate configurations are selected. Specifically, the authors conclude that:

Short opcode sequences (specifically $k=2$ ) provide an optimal trade-off between resilience against transformations and the ability to distinguish independent software.
Set-based similarity measures (particularly the Dice index) are superior to sequence-based measures for this task, as they focus on the presence of shared elements rather than their order, which is easily disrupted by paraphrasing.
The choice of LLM significantly impacts detectability, implying that future detection systems must account for the varying transformation capabilities of different models.

The authors maintain a modest stance, noting that while their findings are robust within the context of BSD-2-Clause Java projects, generalization to other licenses or more complex plagiarism strategies requires further investigation. They emphasize that their work provides a foundation for executable-based plagiarism detection in an era where source-code similarity is no longer a reliable indicator of theft.