💬 NLP

Fine-Tuning Pre-Trained Code Models for AI-Generated Code Detection

The paper presents the "Archaeology" team's system for SemEval-2026 Task 13, which leverages fine-tuned pre-trained code models and specialized strategies like leave-one-language-out cross-validation and sandwich token packing to achieve top-tier performance in detecting AI-generated code and attributing its source.

Original authors: Jany-Gabriel Ispas, Sergiu Nisioi

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jany-Gabriel Ispas, Sergiu Nisioi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a digital detective agency called Archaeology. Their job isn't to dig up ancient pottery, but to sift through piles of computer code to answer two big questions:

Who wrote this? Was it a human programmer or a robot (AI)?
Which robot wrote it? If it was a robot, which specific AI model created it?

This team entered a high-stakes competition called SemEval-2026 Task 13 to prove their skills. Here is how they did it, explained simply.

The Challenge: A Two-Part Mystery

The competition had two distinct puzzles, each requiring a different detective strategy.

Puzzle A: The "Human vs. Robot" Test (Binary Classification)

The Goal: Look at a piece of code and say, "Human" or "AI."
The Trap: The team was trained mostly on Python code (91% of their practice data), but the final test included languages they had never seen before, like JavaScript, Go, and C#. It's like teaching a chef to cook only Italian food, then asking them to identify if a dish is homemade or factory-made when it's actually a Thai curry.
The Clue: AI code tends to be longer and more uniform, while human code is shorter and more chaotic. However, this rule flips depending on the language (e.g., in C++, humans actually write longer code than AI).

Puzzle B: The "Which Robot?" Test (Multi-Class Attribution)

The Goal: If the code is AI-generated, identify exactly which of the 11 different AI models wrote it.
The Trap: The data was wildly unbalanced. One category (Human-written code) made up 88% of the data, while some specific AI models had fewer than 2,000 examples. It's like trying to find a needle in a haystack, where the haystack is 99% hay and the needles are different colors, but you only have one red needle and a million blue ones.
The Clue: Different AI models have subtle "handwriting" styles, but these styles are hard to spot in long, complex code.

The Detective's Toolkit

The team didn't just guess; they used four different "super-eyes" (pre-trained AI models) and gave them special training techniques for each puzzle.

For Puzzle A (Human vs. Robot)

The "Language Swap" Drill: To prepare for the surprise languages in the test, they trained the models by hiding one language at a time. They taught the model to recognize patterns without relying on specific language tricks (like how Python uses # for comments).
The "Scrub and Mask" Workout: They stripped away comments and replaced numbers with placeholders during training. This forced the models to look at the logic of the code, not just the surface decorations.
The "Chunking" Strategy: Since the test code was often too long to read in one go, they sliced it into overlapping pieces (like reading a long book by reading a few pages at a time). They then took the average of these slices, ignoring the most extreme "noisy" guesses, to make a final decision.
The "Difficult Case" Calibration: They created a special "hard test" using examples that a simple computer program got wrong. They tuned their decision threshold (the line between "Human" and "AI") based on these tricky cases to ensure they didn't get fooled by the test data.

For Puzzle B (Which Robot?)

The "Sandwich" Technique: Since the code was too long to fit in the model's memory, they didn't just chop off the middle. Instead, they kept the Head (start) and the Tail (end) of the code and stuffed them together like a sandwich, with a special marker in the middle. This preserved the style at the beginning and end, which is often where the "robot signature" hides.
The "Fairness" Weight: Because some AI models were rare in the data, the team told the model, "Don't just guess the most common one! Pay extra attention to the rare ones." They adjusted the scoring system so the model learned to spot the rare robots, not just the popular ones.
The "Crowd Wisdom" Vote: They ran the same code through the model multiple times with slight variations and let the models vote on the answer. This reduced errors and increased confidence.

The Results

The "Archaeology" team did very well:

Puzzle A: They got a score of 0.737, placing them 6th out of 81 teams. Their best tool was CodeBERT, which seemed to understand the "logic" of code better than the others.
Puzzle B: They got a score of 0.422, placing them 7th out of 34 teams. Their best tool here was UniXcoder, which benefited from looking at longer chunks of code and using the "voting" strategy.

The Big Takeaway

The team discovered that how a model was trained matters more than how big it is.

The smaller models (125 million parameters) that were trained specifically to understand code logic performed better at spotting AI.
The giant model (220 million parameters) didn't win, suggesting that for this specific job, being a "specialist" is better than being a "generalist."

In short, the team proved that with the right training tricks—like slicing long code, ignoring surface noise, and voting on answers—you can build a system that is surprisingly good at catching AI-generated code, even when it tries to disguise itself in new languages.

Technical Summary: Archaeology at SemEval-2026 Task 13

Problem Statement

The paper addresses the challenge of detecting AI-generated source code, a critical issue for academic integrity, software supply-chain security, and intellectual property attribution. The authors participated in SemEval-2026 Task 13, which is divided into three subtasks; their system, "Archaeology," focused on the first two:

Subtask-A: Binary classification to distinguish between human-written and AI-generated code.
Subtask-B: Multi-class attribution to identify the specific Large Language Model (LLM) family that generated a given code snippet among 11 classes (including human-written code and 10 distinct AI models).

Both subtasks utilize the macro-averaged F1 score as the primary evaluation metric. The tasks present distinct challenges: Subtask-A requires generalization to out-of-domain (OOD) programming languages not seen during training, while Subtask-B must handle extremely imbalanced class distributions (a 225:1 ratio between the largest and smallest classes) and long code sequences.

Methodology

Baseline

The team established a baseline using TF-IDF features (character n-grams from bigrams to 5-grams) combined with Logistic Regression. Code was truncated to the first 10,000 characters. For Subtask-B, balanced class weights were applied to mitigate label imbalance. This baseline achieved macro-F1 scores of 0.266 (Subtask-A) and 0.255 (Subtask-B), demonstrating that surface-level lexical patterns alone are insufficient.

Pre-Trained Models

The core system fine-tunes four pre-trained code models:

CodeBERT: Encoder-only (RoBERTa-base, 125M params), pre-trained with Masked Language Modeling (MLM) and Replaced Token Detection.
GraphCodeBERT: Encoder-only (RoBERTa-base), extending CodeBERT with data-flow graph structures.
UniXcoder: Encoder-only (RoBERTa-base), a unified model supporting understanding and generation via cross-modal objectives.
CodeT5+: Encoder-decoder (220M params), pre-trained with denoising and causal language modeling on code and code-text pairs.

Strategies for Subtask-A (Binary Classification)

To address the OOD language challenge (training on Python, C++, Java; testing on additional languages like JavaScript, Go, PHP), the authors employed:

Leave-One-Language-Out (LOLO) Cross-Validation: Models were trained on two languages and validated on the third to simulate OOD conditions. Training data was balanced by sampling equal numbers of examples per language and label.
Code Augmentation: Stochastic augmentations were applied to reduce reliance on language-specific surface features, including comment stripping, literal masking, and header dropping. Deterministic normalization removed markdown fences and standardized whitespace.
Chunked Inference: Since test code is longer than training code, snippets exceeding the model's context window were split into overlapping chunks. Probabilities were aggregated using a trimmed-mean (dropping min/max values) blended with a top-2 mean (averaging the two highest probabilities) to balance robustness against noise with sensitivity to strong signals.
Threshold Calibration: A two-stage calibration was used. First, an optimal threshold was found on OOF predictions. Second, a "difficult dataset" of 1,000 misclassified examples (from a Logistic Regression cross-validation) was constructed to derive a second threshold. The final decision threshold was an equal-weight blend of these two, with additional per-language thresholds applied where language detection was possible.

Strategies for Subtask-B (Multi-Class Attribution)

To handle extreme class imbalance and long sequences:

Sandwich Token Packing: Instead of truncating long code, the input sequence was constructed using the first $h$ tokens (head), a delimiter, and the last $t$ tokens (tail). During training, the head fraction was randomly sampled to act as data augmentation; during inference, it was fixed at 0.60.
Class-Balanced Loss: To counter the 225:1 imbalance, the authors used class-balanced loss weights based on the effective number of samples ( $\beta = 0.9995$ ), giving higher weight to underrepresented classes without the extreme upweighting of inverse-frequency methods.
Ensembling and Test-Time Augmentation (TTA):
- UniXcoder: Trained with 3 random seeds and run with 3 different head fractions during inference (9 forward passes total).
- CodeBERT/GraphCodeBERT: Single seed with 3 head fractions (3 passes).
- CodeT5+: Single seed, fixed head fraction, 1 epoch of training due to model size.

Results

The system achieved the following macro-F1 scores on the test set:

Subtask-A: 0.737 (6th place out of 81 teams) using CodeBERT.
Subtask-B: 0.422 (7th place out of 34 teams) using UniXcoder.

Key Observations:

All fine-tuned models significantly outperformed the TF-IDF baseline.
For Subtask-A, encoder-only models focused on code understanding (CodeBERT, GraphCodeBERT) outperformed the encoder-decoder CodeT5+ and the generation-oriented UniXcoder.
For Subtask-B, UniXcoder outperformed the other models, attributed to its larger context window (1,024 tokens vs. 512), the 3-seed ensemble, and the use of the <encoder-only> mode token.
Error analysis revealed that models struggled more with OOD languages in Subtask-A and with distinguishing stylistically similar AI generators in Subtask-B (e.g., Class 1 had an F1 $\le$ 0.11 despite having sufficient training data).

Significance and Contributions

The paper claims three primary technical contributions:

Chunked Inference Pipeline: A method for binary code classification using trimmed-mean aggregation to handle long code snippets exceeding model context limits.
Threshold Calibration Strategy: A hybrid approach blending OOF thresholds with thresholds derived from a "difficult dataset" of misclassified examples, further refined by per-language calibration.
Sandwich Token Packing: A strategy for multi-class attribution that preserves information from the beginning and end of long code sequences, combined with test-time augmentation and class-balanced loss.

The authors conclude that the choice of pre-training objective is more critical than model size for these tasks. Models pre-trained for code understanding (CodeBERT, GraphCodeBERT) excelled at binary detection, while the unified UniXcoder, enhanced by longer context windows and ensembling, performed best on multi-class attribution. The paper notes that CodeT5+, despite being the largest model (220M parameters), did not achieve the top scores, suggesting that size alone does not guarantee performance in this domain.

Limitations

The authors acknowledge several limitations:

The use of relatively small pre-trained models (125M–220M parameters) and limited context windows (up to 1,024 tokens).
The trade-off in chunked inference between coverage and signal dilution.
The reliance on a "difficult dataset" for threshold calibration, which may not always be available in real-world deployment.
A lack of extensive ablation studies due to computational constraints, leaving the isolated contribution of specific components (e.g., specific augmentations or aggregation methods) unquantified.