An Empirical Study of the Imbalance Issue in Software Vulnerability Detection

1. Problem Statement

Software vulnerability detection is critical for cybersecurity, yet it faces a significant class imbalance problem. In real-world source code, vulnerable functions are extremely rare compared to secure functions (e.g., 1 vulnerability per 500 functions).

• The Core Issue: Deep Learning (DL) models trained on imbalanced datasets tend to bias toward the majority class (secure code). This results in a high False Negative Rate (FNR), meaning the model fails to detect actual vulnerabilities, even if the overall accuracy appears high.
• The Gap: While imbalance solutions are well-studied in Computer Vision (CV) and Natural Language Processing (NLP), their effectiveness in software vulnerability detection remains an open question. Furthermore, existing literature often uses Accuracy as the primary metric, which is misleading in imbalanced scenarios.

2. Methodology

The authors conducted a comprehensive empirical study to validate the impact of imbalance and evaluate existing mitigation strategies.

Datasets

• Scope: 9 open-source datasets derived from 3 repositories (Devign, Lin2018, CodeXGLUE).
• Content: C-language code at the function level.
• Imbalance Ratios (IR): Varied significantly, ranging from nearly balanced (Devign FFmpeg, IR $\approx$ 0.96) to highly imbalanced (Asterisk, IR $\approx$ 304.82).

Models

Two State-of-the-Art (SOTA) foundation models were used:

1. CodeBERT: Pre-trained on bimodal data (code + text) and unimodal code.
2. GraphCodeBERT: Incorporates data flow graphs of source code.
   Both models were fine-tuned for binary classification (Secure vs. Vulnerable).

Imbalance Solutions Evaluated

Seven methods were tested, categorized into Data-level and Model-level:

• Data-level:
  • Random Down-sampling: Removing secure code samples.
  • Random Over-sampling: Duplicating vulnerable code samples.
  • Adversarial Attack-based Augmentation: Generating new vulnerable code via variable renaming (using CodeBERT's masked language prediction).
• Model-level (Loss Function/Threshold adjustments):
  • Threshold-moving: Adjusting the decision threshold post-training.
  • Mean False Error (MFE) Loss: Separating loss calculation for secure and vulnerable sets.
  • Class-Balanced (CB) Loss: Reweighting based on the effective number of samples.
  • Focal Loss (FL): Focusing on hard-to-classify samples.

Evaluation Metrics

The study argues against using Accuracy alone. Instead, it evaluates performance using:

• Precision: Reliability of positive predictions.
• Recall: Ability to find all vulnerabilities (inverse of FNR).
• F1-Measure: Harmonic mean of Precision and Recall.
• False Positive Rate (FPR): Reliability of negative predictions.

3. Key Contributions & Findings

RQ1: Impact of Imbalance

• Finding: Imbalance causes models to learn significantly better from secure code than vulnerable code.
• Evidence: In highly imbalanced datasets (e.g., Asterisk), models achieved ~100% accuracy on secure code but only ~44% on vulnerable code. The total loss on secure code was negligible, while the loss on vulnerable code remained high, indicating the model ignores the minority class during training.

RQ2: Appropriate Evaluation Metrics

• Finding: Accuracy is misleading in this domain. A model can achieve 99% accuracy by simply classifying everything as "secure," yet it is useless for security.
• Recommendation: Precision, Recall, and F1-measure are the most informative metrics.
  • Recall is critical for identifying vulnerabilities.
  • Precision is critical for reducing the manual effort of filtering false positives.

RQ3: Effectiveness of Existing Solutions

No single solution excels across all metrics and datasets. The performance depends on the specific goal:

• To Improve Precision: Focal Loss is the most effective. It forces the model to focus on hard/misclassified samples, improving the correctness of positive predictions.
• To Improve Recall: Mean False Error (MFE) and Class-Balanced (CB) Loss are superior. They explicitly weight the minority class higher, forcing the model to identify more vulnerabilities (though this may increase false positives).
• To Improve F1-Measure: Random Over-sampling generally yields the best balanced performance.
• Ineffective Methods: Random Down-sampling often performs the worst, as it discards valuable information from the majority class, leading to underfitting.

RQ4: External Factors Hindering Solutions

The study identified that external factors significantly influence solution effectiveness:

1. Vulnerability Type Absence: If a specific vulnerability type (e.g., "Overflow") exists in the test set but not in the training/validation sets, no solution can detect it.
2. Inherent Difficulty: Some vulnerability types are inherently harder to detect; solutions cannot overcome this if the model has reached its learning limit for that specific pattern.
3. Distribution Shift: Differences between validation and test data distributions can cause solutions (like Threshold-moving) to fail or degrade performance.

4. Significance and Implications

• Paradigm Shift: The paper challenges the reliance on "Accuracy" in vulnerability detection literature, advocating for a multi-metric approach (Precision/Recall/F1).
• Practical Guidance: It provides a "menu" for practitioners:
  • If the goal is to minimize false alarms (high precision), use Focal Loss.
  • If the goal is to catch every possible bug (high recall), use MFE or CB Loss.
  • If a balanced score is needed, use Over-sampling.
• Future Directions: The authors conclude that generic solutions from CV/NLP are insufficient. There is a critical need for task-specific solutions for software vulnerability detection that account for code semantics, specific vulnerability types, and data distribution shifts.

5. Conclusion

The study empirically validates that class imbalance is the root cause of poor vulnerability detection performance in DL models. While existing imbalance mitigation techniques offer partial improvements, they are not a "silver bullet." The field requires new, specialized approaches that consider the unique characteristics of source code and the specific trade-offs between precision and recall in security contexts. The authors have made their datasets and artifacts publicly available to facilitate reproducibility.