🤖 machine learning

mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code

The mcdok system for SemEval-2026 Task 13 adapts the mdok text detection approach by fine-tuning code-specific large language models to competitively address machine-generated code detection, attribution, and hybrid code identification, though significant performance gaps remain compared to top systems.

Original authors: Adam Skurla, Dominik Macko, Jakub Simko

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adam Skurla, Dominik Macko, Jakub Simko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Who wrote this piece of code? Was it a human programmer, or was it generated by an Artificial Intelligence (AI)?

In the past, it was easy to tell the difference. But today, AI models are so good at writing code that they can sound almost exactly like a human. To tackle this, a global competition called SemEval-2026 created a challenge (Task 13) for researchers to build "AI detectors" specifically for computer code.

A team from Slovakia and the Czech Republic, calling their system "mcdok," entered the competition. Here is how they did it, explained simply.

The Three Challenges (Subtasks)

The competition wasn't just one big test; it was broken down into three different levels of difficulty, like video game levels:

Level 1 (Subtask A): The "Is it AI?" Test.
- The Goal: Look at a snippet of code and say, "Yes, this is AI" or "No, this is human."
- The Analogy: Imagine a teacher grading essays. They just need to spot if the essay was written by a student or copied from a robot.
- The Languages: The code came in three languages: C++, Python, and Java.
Level 2 (Subtask B): The "Who is the Robot?" Test.
- The Goal: If the code is AI, which specific family of AI made it? There were 10 different AI families (like "OpenAI," "Google," "Meta") plus the "Human" category.
- The Analogy: This is like a forensic expert looking at a fingerprint and saying, "This wasn't just a robot; it was a specific robot named 'Qwen' or 'Gemma'."
Level 3 (Subtask C): The "Tricky Mix" Test.
- The Goal: Distinguish between four types: Human, Pure AI, Hybrid (a human started it, AI finished it), and Adversarial (AI trying to trick the detector by pretending to be human).
- The Analogy: This is the hardest level. It's like a chef trying to taste a soup and figure out: "Did a human cook this? Did a robot cook this? Did they cook it together? Or did the robot try to disguise its flavor to look like a human dish?"

How mcdok Solved It

The team didn't invent a brand new super-computer from scratch. Instead, they took an existing tool they had built for detecting AI text (called mdok) and gave it a "code makeover."

The Makeover: They realized that a model trained to read novels might not understand Python or C++. So, they swapped the "brain" of their system for models specifically designed to understand code (like Gemma, CodeGemma, and Qwen).
The Training: They taught these new brains using a technique called Fine-tuning. Think of this as taking a general knowledge expert and giving them a crash course specifically on "How to spot AI code."
The Strategy: They didn't use a complicated team of different detectors (an "ensemble"). They kept it simple: one strong brain per task.

The Results

The team did very well, but they weren't perfect.

Level 1: They were quite good at spotting AI code, especially when they adjusted their "confidence settings." Interestingly, a general-purpose model (Gemma-3) actually worked better than the specialized code models!
Level 2: This was tough. Identifying which specific AI family wrote the code was hard. Their system was better than random guessing, but it sometimes confused one AI family for another.
Level 3: They did surprisingly well here, ranking in the top 20% of all participants. They could tell the difference between pure AI and "Hybrid" code, though they sometimes got confused between "Machine" and "Hybrid" (which makes sense, since they are very similar).

The Big Takeaway

The paper concludes that AI detectors built for text can be successfully adapted for code, but you have to be careful. You can't just use any model; you need one that understands programming logic.

However, the "margins" (the gap between the winners and the rest) were still significant. The team admits that while their system is competitive, there is still room for improvement. The "cat and mouse" game between AI writers and AI detectors is far from over.

In short: The mcdok team built a specialized "code detective" that is getting better at catching AI imposters, but the imposters are getting smarter, too.

1. Problem Definition

The paper addresses the increasing difficulty of distinguishing between human-written and machine-generated programming code due to advancements in Large Language Models (LLMs). The authors participated in SemEval-2026 Task 13, a shared task designed to evaluate detection systems in a complex, multi-domain, and multi-generator setting. The task is divided into three specific subtasks:

Subtask A (Binary Detection): Determine if a code snippet is human-written or machine-generated. The dataset includes C++, Python, and Java in the training phase, with evaluation covering seen and unseen domains/languages.
Subtask B (Attribution/Multi-class): Identify the specific LLM family (out of 10 families) or the human-written class, resulting in an 11-class classification problem.
Subtask C (Hybrid/Adversarial Detection): Distinguish between four classes: Human-written, Machine-generated, Hybrid (co-written by humans and LLMs), and Adversarial (machine-generated code specifically modified to mimic human style).

2. Methodology

The authors' system, mcdok (machine-code detector of KInIT), is an adaptation of their previous mdok system, which was designed for machine-generated text detection. The core methodology involves Parameter-Efficient Fine-Tuning (PEFT) of Large Language Models.

Key Technical Components:

Base Model Selection: Instead of using a single model for all tasks, the team experimented with various base models specialized for code understanding (e.g., CodeGemma, Qwen2.5-Coder, CodeLlama) and general-purpose models (Gemma).
- Subtask A: Best performance achieved with Gemma-3-27B-PT (a general-purpose model).
- Subtask B: Best performance achieved with CodeGemma-7B.
- Subtask C: Best performance achieved with Qwen2.5-Coder-14B.
Fine-Tuning Approach:
- Technique: QLoRA (4-bit quantization) using the bitsandbytes and transformers libraries.
- Configuration: LoRA rank ( $r$ ) of 64, alpha of 16, dropout of 0.1, and no bias.
- Training Strategy: Single sample per step (batch size 1), paged AdamW optimizer, learning rate of $2 \times 10^{-5}$ with a cosine scheduler, and a warmup ratio of 0.03.
- Loss Function: Weighted cross-entropy loss to handle class imbalance, where weights are the inverse of class distribution.
Data Preprocessing:
- Unlike text detection, no text preprocessing (e.g., anonymization, lowercasing) was applied to the code to avoid invalidating syntax.
- Data Balancing: The training data was subsampled to balance classes (e.g., limiting to 10k samples per generator/language, then further balancing to 20k per class for Subtask A).
- Validation: The test_sample provided by organizers was used for Out-of-Distribution (OOD) validation and checkpoint selection.
Threshold Adjustment: For Subtask A, the authors observed that default thresholds (0.5) yielded poor results. They significantly improved performance by fixing the classification threshold to 1.0 (requiring 100% confidence for the "machine" class), a technique derived from their previous text detection work.

3. Key Contributions

Adaptation of Text Detectors to Code: Successfully demonstrated that architectures and fine-tuning strategies effective for machine-generated text detection can be transferred to code detection, provided the base model is suitable for code understanding.
Task-Specific Model Selection: Identified that different subtasks benefit from different model architectures. Notably, a general-purpose model (Gemma-3-27B) outperformed code-specialized models in binary detection (Subtask A), while code-specialized models (CodeGemma, Qwen) performed better in complex attribution and hybrid detection tasks.
Robust Evaluation Framework: Conducted extensive analysis using the Droid test set (an external dataset heavily related to the shared task data) to evaluate performance on unseen languages and generators, providing insights beyond the official leaderboard.
Open Source Reproducibility: The authors released the full source code for training and inference, allowing the community to replicate the mcdok detectors.

4. Results

The systems were evaluated using Macro F1 scores.

Subtask A (Binary Detection):
- The Gemma-3-27B-PT model achieved the highest Macro F1 of 0.6975 on the official test set (with threshold adjustment).
- Code-specialized models (CodeGemma-7B, Qwen2.5-Coder-14B) scored between 0.63 and 0.64.
- The system ranked 10th out of 81 submissions (88th percentile).
- Performance was stable across languages, with notable success on C and C# (unseen in training).
Subtask B (Attribution):
- The CodeGemma-7B model achieved a Macro F1 of 0.3955.
- The system outperformed both the CodeBERT baseline and a random baseline significantly.
- It ranked 13th out of 34 submissions (62nd percentile).
- Analysis showed strong performance in identifying "Human," "OpenAI," and "BigCode" classes, but confusion existed among other families.
Subtask C (Hybrid/Adversarial):
- The Qwen2.5-Coder-14B model achieved the best Macro F1 of 0.6864.
- The system ranked 5th out of 32 submissions (84th percentile).
- The system struggled most with the Hybrid class, often confusing it with the Machine-generated class.

5. Significance and Limitations

Significance:
The paper highlights that the detection of machine-generated code is a viable extension of machine-generated text detection. It emphasizes that model selection is critical and task-dependent; a "one-size-fits-all" approach is less effective than tailoring the base model to the specific nuance of the subtask (e.g., general models for binary detection vs. code-specialized models for attribution). The work also underscores the challenge of detecting hybrid and adversarial code, which remains a significant gap in current detection capabilities.

Limitations:

Data Constraints: Experiments were limited to the official dataset and specific sampling techniques; results may vary with different data distributions.
Generalization: While some general-purpose models performed well, the results are specific to the tested models and cannot be universally generalized to all LLMs.
Performance Gaps: Despite competitive rankings, there remains a significant margin between the submitted systems and the top-performing systems, particularly in Subtask B, indicating room for improvement in fine-tuning strategies and hyperparameter tuning.