UCSC-NLP at SemEval-2026 Task 13: Multi-View Generalization and Diagnostic Analysis of Machine-Generated Code Detection

1. Problem Statement

The paper addresses the critical challenge of distinguishing between human-written and AI-generated code, a task essential for maintaining academic integrity, ensuring hiring fairness, and securing software supply chains. The authors participate in SemEval-2026 Task 13, which presents two distinct subtasks with unique challenges:

• Subtask A (Binary Detection): Distinguishing human code from machine-generated code. The primary challenge here is distribution shift: the training data is restricted to Python, C++, and Java within an algorithmic domain, while the test set includes unseen languages (JavaScript, C#, Go, etc.) and unseen domains (Research, Production), including 33.4% "mixed-content" snippets.
• Subtask B (Multi-Class Attribution): Identifying the specific Large Language Model (LLM) family that generated the code (Human vs. 10 different LLM families). The primary challenge is extreme class imbalance: the dataset is dominated by human code (88.4%), creating a 221:1 ratio between the majority class and the smallest minority classes. This leads to "catastrophic minority-class failure" where standard models simply predict the majority class.

2. Methodology

Subtask A: Multi-View Generalization Framework

To address the language and domain mismatch, the authors fine-tune UniXcoder-base using a multi-view training framework designed to learn generator-invariant representations. The system processes three views of the input code:

1. Original View: Standard tokenization.
2. Delexicalized View: Identifiers, string literals, and numbers are replaced with generic tokens (ID, STR, NUM) to remove language-specific surface cues.
3. Mixed-Content View: Random text fragments are injected into code snippets to simulate the mixed-content distribution found in the test set.

Key Techniques:

• 3-Domain Structural Prefixes: Snippets are prepended with a prefix indicating their type (clean, mixed, or fragment) to help the model align content types.
• Symmetric KL Consistency Loss: The model is trained to minimize the symmetric Kullback-Leibler divergence between the predictions of the original and delexicalized/mixed views, enforcing consistency across different representations.
• Token Dropout: 15% of non-special tokens are randomly masked during training to prevent reliance on spurious language-specific keywords.
• Inference Strategy: The final prediction is an average of the logits from all three views, acting as a zero-cost ensemble.

Subtask B: Diagnostic Baseline and Class-Weighted Extension

The authors use CodeBERT-base with a linear softmax classification head.

1. Diagnostic Baseline: A standard fine-tuned model without class reweighting. This serves as a control to demonstrate that standard fine-tuning fails under extreme imbalance.
2. Class-Weighted System: A competitive system that applies inverse-frequency class weighting to the cross-entropy loss. This upweights minority LLM classes proportionally to their underrepresentation to force the model to learn discriminative features for rare generators.

3. Key Contributions

1. Robust Binary Detection Framework: A novel multi-view training approach that achieves strong generalization across unseen languages and domains by enforcing consistency between delexicalized, mixed, and original code views.
2. Diagnostic Analysis of Imbalance: A rigorous demonstration that standard fine-tuning fails catastrophically in multi-class attribution tasks with extreme class imbalance. The authors show that high accuracy (88.4%) can mask a complete failure to detect minority classes (Macro F1 ~0.08).
3. Empirical Evidence of Embedding Overlap: Visualization via t-SNE reveals that while human and machine code are separable in embedding space (Subtask A), different LLM families overlap significantly with each other and human code (Subtask B), explaining the difficulty of fine-grained attribution.

4. Results

Subtask A (Binary Detection)

• Validation: Achieved a near-perfect 0.993 Macro F1.
• Test Set: Achieved 0.845 Macro F1 on the out-of-distribution test set.
• Analysis: The 14.8-point gap between validation and test scores reflects the difficulty of cross-lingual generalization rather than overfitting. The model showed balanced performance between Human and Machine classes, with no systematic bias.

Subtask B (Multi-Class Attribution)

• Diagnostic Baseline: Achieved 88.4% Accuracy but a catastrophic 0.086 Macro F1. The model effectively predicted "Human" for almost all inputs, failing to identify any specific LLM family.
• Class-Weighted System: By applying inverse-frequency weighting and training for 3 epochs, the system recovered the Macro F1 to 0.345 (a 301% relative improvement).
• Trade-off: This improvement came at the cost of overall accuracy dropping to 78.7%, indicating the model successfully shifted from a majority-class bias to genuine multi-class discrimination.

5. Significance and Future Directions

• Practical Impact: The work highlights that while binary detection of AI code is becoming robust, fine-grained attribution (identifying which AI wrote the code) remains an open challenge due to architectural homogeneity among LLMs and data imbalance.
• Methodological Insight: The paper proves that standard metrics like accuracy are misleading in imbalanced code detection tasks; Macro F1 is the critical metric.
• Future Work: The authors suggest that future improvements in attribution may require:
  • Contrastive learning to maximize inter-class separation.
  • Incorporating structural features (ASTs, control flow graphs) rather than relying solely on token-level signals.
  • Advanced loss functions like Focal Loss or class-balanced sampling.
  • Exploring decoder-based or encoder-decoder architectures (e.g., CodeT5).

The paper concludes with an ethics statement emphasizing that due to false positive rates (Human Recall of 0.760), human review remains essential before making consequential decisions regarding authorship or hiring.