💬 NLP

Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection

The Dream team proposes a SALSA-style single-pass autoregressive classification framework combined with balanced sampling and conservative fine-tuning to achieve robust out-of-distribution detection of machine-generated code, significantly outperforming the CodeBERT baseline on the SemEval-2026 Task 13 leaderboard.

Original authors: Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to figure out if a student's homework was written by the student or copied from a super-smart AI. This is the challenge the authors of this paper tackled for a competition called SemEval-2026 Task 13.

Here is the story of how they solved it, using simple analogies:

The Problem: The "Chameleon" Code

In the past, detecting AI-written code was like trying to spot a chameleon in a forest. The AI could write code in many different languages (like Python, Java, or C++) and for many different jobs. If you trained a detector to spot AI code in Python, it would often fail when the AI switched to Java. The detector was too "specialized" and couldn't handle new, unseen situations.

The Solution: The "One-Word Answer" Game

The team from Dream Security Ltd. proposed a new way to play the game, which they call SALSA (Single-pass Autoregressive LLM Structured Classification).

Think of a normal AI chatbot like a chatty parrot. If you ask it, "Is this code AI-written?", it might say, "Well, looking at the syntax and the variable names, I think it's probably AI because..." and then write a whole paragraph. This is messy and hard to grade.

SALSA changes the rules:

The Prompt: They give the AI a code snippet and ask a very specific question: "Is this machine-generated?"
The Constraint: They tell the AI, "You are only allowed to answer with one single word: either '0' (No) or '1' (Yes)."
The Result: Instead of a long essay, the AI is forced to make a quick, decisive choice. It's like asking a judge to drop a single gavel instead of writing a 10-page legal opinion.

The Training: "Less is More"

Usually, when you teach a computer model a new task, you might drill it for a long time until it memorizes the specific examples you gave it. But the authors realized that if you drill a model too hard on specific examples, it forgets how to handle new situations (this is called "overfitting").

They used a "Conservative Training" approach:

The Learning Rate: Imagine the learning rate is the volume knob on a radio. They turned the volume down very low. This allowed the model to learn the new task without shouting over its own existing knowledge.
The Duration: They only let the model study the examples one single time (one epoch). They found that studying too long made the model forget how to be a generalist and become too specialized in just the training data.
Balanced Diet: The training data had way more Python code than Java or C++. To stop the model from becoming a "Python-only" expert, they forced it to eat an equal amount of every language, even if it meant throwing away some of the extra Python data.

The Results: Beating the Baseline

The competition had a "baseline" detector (CodeBERT) that was like an old, rusty metal detector. It only got a score of 0.305 (very poor) when tested on new, unseen languages.

The Dream Security team's SALSA system was like a high-tech, adaptive scanner.

Zero-Shot (No training): Even without special training, their AI was decent (around 0.5).
With Training: After their careful, "one-epoch" training, their best system achieved a score of 0.789.

This is a massive jump. It means their system is much better at spotting AI code even when the code is written in a language or style it has never seen before.

The Catch

The paper notes a funny quirk: Before training, the AI was very shy. It was so afraid of making a mistake that it almost always guessed "Human-written" (0), even when it was AI. This made it look like it had high "precision" (when it did guess AI, it was right), but terrible "recall" (it missed almost all the AI code). The training helped fix this shyness, teaching the model to be brave enough to say "Yes" when it sees AI code.

In short: They built a detector that doesn't write essays, doesn't memorize the textbook, and doesn't get overwhelmed by new languages. It just looks at the code, thinks for a split second, and drops a single, accurate "Yes" or "No."

Technical Summary: SALSA for Single-Pass Machine-Generated Code Detection

Problem Definition
The paper addresses the critical challenge of detecting code generated by Large Language Models (LLMs), a task essential for maintaining assessment integrity and software trust. Specifically, the authors tackle SemEval-2026 Task 13 Subtask A, which frames the problem as a binary classification task (human-written vs. machine-generated) over code snippets. The primary difficulty lies in Out-of-Distribution (OOD) generalization: the system must perform robustly on unseen programming languages and application domains (e.g., moving from algorithmic training data to research/production evaluation scenarios). Traditional approaches relying on hand-crafted features or decision rules often lack the necessary domain coverage and robustness.

Methodology: SALSA
The authors propose SALSA (Single-pass Autoregressive LLM Structured Classification), a formulation that reframes classification as a constrained token generation task rather than a standard sequence-to-sequence or feature-based problem.

Structured Prompting & Token Mapping:
- Instead of allowing the model to generate verbose explanations or inconsistent formats, SALSA maps each class to a single, dedicated output token (e.g., "0" for human, "1" for machine).
- The prompt is structured with clear delimiters and a direct instruction (e.g., "is the code machine generated?"), requiring the answer in a specific format: <ANSWER>#</ANSWER>.
- To prevent chain-of-thought reasoning which introduces latency and format variance, the authors utilize the /no_think flag and an empty <thought> block (specifically for Qwen3 models) to force a single-token output.
Training Strategy for OOD Robustness:
- Objective: The model is trained using a causal language modeling objective, but the loss is evaluated exclusively at the position of the label token. This focuses gradient updates on the next-token distribution for the target classes.
- Parameter-Efficient Fine-Tuning (PEFT): The authors employ LoRA (Low-Rank Adaptation) to reduce overfitting risks and accelerate training.
- Conservative Training Regime: To prevent the model from over-specializing to the training domain (which degrades OOD performance), the authors adopt a conservative approach:
  - Single Epoch: Training is limited to one epoch.
  - Low Learning Rate: A rate of $5 \times 10^{-6}$ is used.
  - Balanced Sampling: Given the training dataset's heavy skew toward Python (91.5%), the authors implement balanced sampling across (language, label) groups. This prevents the model from learning Python-specific patterns at the expense of general task alignment.
Inference:
- Inference involves a single forward pass. The system extracts logits only for the target class tokens ("0" and "1") and applies a softmax to determine the predicted label, minimizing decoding overhead.

Key Contributions

Formulation: The introduction of SALSA, which converts classification into a reliable, single-token prediction task, eliminating the brittleness of open-ended LLM generation.
OOD Optimization: A demonstration that conservative fine-tuning (low learning rate, single epoch, balanced sampling) preserves the base model's general capabilities, significantly outperforming aggressive training strategies in OOD settings.
Performance: The system achieves a substantial improvement over the official baseline, addressing the gap in multi-language and cross-domain code detection.

Results
The authors evaluated their system on the SemEval-2026 Task 13 leaderboard:

Baseline Comparison: The official CodeBERT baseline achieved an OOD F1 score of 0.305.
SALSA Performance: The best system, utilizing Qwen2.5-72B-Instruct fine-tuned with SALSA, achieved an OOD F1 score of 0.789.
Zero-Shot vs. Fine-Tuned: Zero-shot performance varied significantly by model size and architecture, with some models showing strong bias toward predicting "human-written" code (high precision, very low recall). Fine-tuning with the proposed conservative strategy resolved these biases, achieving near-perfect validation F1 scores (0.996) while maintaining strong OOD generalization.
Model Scaling: Performance generally scaled with model size, with the 72B parameter model outperforming the 32B and 8B variants.

Significance
The paper claims that SALSA provides a robust, efficient, and scalable solution for machine-generated code detection. By delegating the authorship decision to the model via structured prompting and carefully managing the fine-tuning process to avoid overfitting, the approach successfully generalizes to unseen languages and domains. The results suggest that preserving the base model's general-purpose features through conservative training is more effective for OOD tasks than maximizing in-domain accuracy. The authors note that their code is publicly available, facilitating further research in this area.