FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals

Technical Summary: Lightweight Detection of LLM-Generated Code via Stylometric Signals

Problem Statement
The paper addresses the challenge of detecting machine-generated code (LLM-generated) versus human-written code, specifically within the context of SemEval-2026 Task 13 Subtask A. The task requires binary classification that generalizes across unseen programming languages (e.g., Go, PHP, C#, C, JavaScript) and application domains (research and production code). The authors highlight that while large language models (LLMs) have transformed software development, their use raises concerns regarding educational integrity, technical hiring, and software security (e.g., insecure logic or hidden backdoors). Unlike natural language, source code is constrained by strict syntax and semantics, which can obscure the stylistic differences typically used for machine-generated text detection. Furthermore, existing approaches relying on large pretrained models often degrade under distribution shifts, such as when encountering unseen languages or domains.

Methodology
The authors propose a lightweight, interpretable detection pipeline that prioritizes ratio-based stylometric features and heuristic rules over high-capacity neural representations. The system architecture consists of the following components:

1. Feature Extraction and Engineering:

   • Ratio-Based Descriptors: The core of the approach involves length-normalized, ratio-based features designed to be less sensitive to snippet length. Key features include the Comment Ratio (comment lines to total lines), Verb Comment Ratio (verb tokens to word tokens in comments), Text-like Ratio (lines classified as natural language to total code lines), and Identifier Verb Ratio.
   • Syntactic and Structural Features: Using the Tree-Sitter incremental parser, the system extracts features such as Completeness Score (syntactic completeness), Error Nodes Near EOF (parsing errors near the end of the file), Cyclomatic Complexity, and Statements-to-LOC ratios.
   • Language Identification: A dedicated Multinomial Naïve Bayes classifier, trained on the Rosetta Code dataset using character n-gram TF-IDF features, identifies the programming language to enable language-specific parsing.

2. Auxiliary Classifiers:

   • Code-vs-Text Line Classifier: To handle samples containing raw natural language mixed with code, a binary classifier is trained on a custom dataset combining Stack Overflow code snippets and Twitch chat data. It uses a TF-IDF vectorizer with character n-grams to distinguish code lines from text lines at the granularity of individual lines.

3. Classification and Decision Logic:

   • Shallow Classifiers: The system employs logistic regression and shallow decision trees (depth 2) trained on the extracted features.
   • Heuristic Rules: The final predictions integrate classifier outputs with specific heuristics derived from data analysis:
     • Language Markers: Instances where the programming language name appears as a standalone line preceding the code (a common LLM artifact) are flagged.
     • Text-like Ratio Threshold: A threshold of 0.3 is applied to the Text-like Ratio. Samples exceeding this are directly classified as LLM-generated, prioritizing specificity over sensitivity.
   • Bucketed Scaling: To account for length variations, code samples are partitioned into small, medium, and large buckets based on line count, with separate feature scaling and decision thresholds applied to each.

Key Contributions
The paper outlines four primary contributions:

1. Analysis of Pretrained Encoders: An investigation into the limitations of pretrained code encoders (e.g., GraphCodeBERT, CodeT5+) under distribution shift, demonstrating their poor generalization to unseen languages compared to feature-based methods.
2. Interpretable Ratio-Based Features: The introduction of specific ratio-based descriptors that capture descriptiveness and stylistic leakage (e.g., verbose comments, explanatory text) common in LLM-generated code.
3. Code-vs-Text Dataset: The construction of a dataset for line-level classification between natural language and source code, enabling the detection of natural language artifacts embedded within code samples.
4. Lightweight Pipeline: A computationally efficient detection system that achieves competitive performance using only CPU resources and offers near-instant inference, contrasting with the high computational costs of large neural models.

Results
The system was evaluated on the SemEval-2026 Task 13 official test set, which includes seven programming languages and diverse domains.

• Pretrained Encoders: Models like GraphCodeBERT and CodeT5+ achieved high validation scores (e.g., 98.21% and 97.46% macro-F1) but failed to generalize to the test set, dropping to 25.11% and 49.39% respectively. The authors attribute this to overfitting on language-specific cues (e.g., Python indentation) that do not transfer to other languages.
• Feature-Based Approach: The final system, combining a decision tree (depth 2) with heuristics, achieved a macro-F1 score of 67.35.
  • The decision tree alone scored 65.62.
  • Logistic regression with threshold tuning scored 63.16.
  • The addition of heuristics provided an absolute improvement of approximately 2%.
• Auxiliary Components: The code-vs-text classifier achieved 96.07% macro-F1 on the validation set, while the language identification classifier reached 95.10% accuracy.
• Ranking: The approach ranked within the top 15% of all submissions on the official leaderboard.

Significance and Claims
The authors position their work as a robust, interpretable alternative to complex neural models for code authorship detection. They claim that their approach effectively addresses the challenge of generalization across languages and domains without requiring GPU resources or large-scale training. By focusing on stylometric signals—specifically the presence of verbose comments, explanatory text, and structural artifacts like standalone language markers—the system captures the "descriptiveness" often inherent in LLM outputs. The paper emphasizes that while deep learning models offer strong semantic representations, they suffer from distribution shifts and lack interpretability. In contrast, the proposed lightweight pipeline offers a practical solution for scenarios where computational efficiency, interpretability, and cross-lingual robustness are critical, achieving near-instant inference times on standard CPU hardware.