Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection

Technical Summary: Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection

Problem Statement
Software vulnerabilities pose significant risks to security, financial stability, and system maintenance, particularly as modern applications increasingly integrate multiple programming languages. While traditional static and dynamic analysis techniques suffer from high false-positive rates and limited coverage, and conventional machine learning methods rely on handcrafted features or coarse-grained analysis, transformer-based models have emerged as promising alternatives due to their ability to capture contextual information and long-range dependencies in source code. However, existing research predominantly focuses on single languages (e.g., C/C++) or limited language sets, creating a gap in understanding how these models generalize across diverse syntaxes and semantics in multilingual environments.

Methodology
This study frames software vulnerability detection as a binary classification problem (vulnerable vs. safe) and conducts an early comparative evaluation of three transformer architectures: BERT, RoBERTa, and CodeBERT.

• Dataset: The researchers utilized the CVEFixes dataset, specifically selecting a subset of 9,000 samples across four languages: HTML (285 samples), Python (1,564 samples), JavaScript (1,562 samples), and PHP (5,589 samples). The dataset was preprocessed to remove incomplete records, resulting in a near-balanced distribution of vulnerable (encoded as 1) and safe (encoded as 0) samples for each language.
• Experimental Design: The models were evaluated using language-wise three-fold stratified cross-validation to preserve class distribution. Source code was tokenized with a maximum sequence length of 512 tokens using the respective tokenizers for each model.
• Implementation: Fine-tuning was performed using the Hugging Face Trainer framework. Performance was measured using accuracy, precision, recall, and F1-score.

Key Contributions
The paper outlines three primary contributions:

1. The development of an early multilingual transformer-based pipeline for software vulnerability detection.
2. A direct, language-wise comparison of BERT, CodeBERT, and RoBERTa across HTML, Python, JavaScript, and PHP.
3. The provision of empirical evidence demonstrating that transformer model performance varies substantially across different programming languages.

Results
The experimental results indicate that no single model consistently outperforms the others across all languages, and performance is highly language-dependent:

• HTML: This language yielded the strongest results. CodeBERT achieved the highest F1-score of 0.7171, followed closely by RoBERTa (0.7078) and BERT (0.7064). All models achieved F1-scores above 0.70.
• Python, JavaScript, and PHP: These languages presented significantly greater challenges, with all models achieving F1-scores below 0.45.
  • Python: CodeBERT performed best with an F1-score of 0.4367.
  • JavaScript: RoBERTa achieved the highest F1-score of 0.4289.
  • PHP: BERT achieved the strongest F1-score of 0.4036, though all models struggled with this language (lowest overall performance).

Significance and Claims
The paper modestly positions these findings as an "early language-wise comparison" rather than a fully generalizable benchmark. The authors claim that the substantial variation in performance suggests that vulnerability patterns are language-dependent. Consequently, effective multilingual vulnerability detection likely requires language-specific tuning, improved handling of class imbalance, and richer source-code representations rather than a "one-size-fits-all" model approach.

The authors acknowledge limitations, noting that the selected languages do not represent the full spectrum of programming languages and that the findings are specific to the current experimental setting. They conclude that future work must expand language coverage, investigate mixed-language training strategies, and employ more robust evaluation metrics (such as macro F1-score, Matthews correlation coefficient, and PR-AUC) to better assess model generalization.