A Comprehensive Evaluation of Code Language Models for Security Patch Detection

Technical Summary: A Comprehensive Evaluation of Code Language Models for Security Patch Detection

Problem Statement
Automated detection of vulnerability-fixing commits (VFCs) is essential for timely security patch deployment, particularly given that advisory databases often lag patch releases by a median of 25 days and many fixes never receive public advisories. While Code Language Models (Code LMs) are increasingly adopted for Security Patch Detection (SPD), their ability to recognize security fixes based solely on code changes remains unclear. Existing performance reports are often confounded by three critical factors: (1) models relying heavily on commit messages rather than code semantics; (2) data leakage caused by evaluating models on projects present in both training and testing sets; and (3) uncertain label quality, particularly for commits lacking explicit CVE associations. The joint effect of these factors on reported SPD performance has not been rigorously quantified.

Methodology
To address these issues, the authors constructed a unified framework consolidating 20 fragmented VFC datasets (spanning over 180,000 commits) into a coherent evaluation suite. This framework includes tools for systematic parsing, normalization, deduplication, and context enrichment.

The evaluation protocol involved training 270 distinct models ranging from 125M to 80B parameters (including encoder, encoder-decoder, and decoder architectures) across four dataset compositions (D1–D4) with varying scopes of language support and label sources (CVE-mapped, advisory-based, and automated tooling). The study employed three specific research strategies:

1. Strict Evaluation Protocols: Utilizing group-stratified splits to prevent project-level data leakage and isolating code-only inputs to measure genuine code reasoning capabilities.
2. Context Enrichment: Implementing lightweight intra-procedural context enrichment (using control-flow and data-flow analysis via Tree-sitter and GumTree) and evaluating inter-procedural repository context (RepoSPD) to determine if additional semantic context improves detection.
3. Label Audit: Conducting a manual expert audit of 200 commits to quantify label accuracy and analyzing error distributions to distinguish between model failures and labeling noise.

The evaluation metrics included F1 scores and the Patch Detection Score (PD-S), defined as the False Negative Rate at a strict False Positive Rate (FPR) of 0.5% (0.005), to mitigate the base rate fallacy.

Key Contributions

1. Unified Framework: The release of a framework integrating 20 VFC datasets with automated pipelines for normalization, deduplication, and context enrichment, enabling systematic comparative evaluation across labeling strategies and languages.
2. Rigorous Re-evaluation: A comprehensive study of code LM-based SPD that isolates the impact of training protocols, model capacity, and context signals, providing concrete recommendations for future evaluations on aggregated corpora.
3. Label Quality Quantification: A direct quantification of commit-level label accuracy through expert auditing, revealing that label errors concentrate in commits lacking CVE associations and significantly distort evaluation metrics.

Key Results

• Reliance on Natural Language: Models primarily classify commits based on commit messages. Training on messages alone nearly matches full-commit performance, while code-only training results in a substantial drop in performance.
• Impact of Splitting Strategies: Group-stratified splits (separating projects between train/test) reduce F1 scores by 16–28% compared to random splits, revealing that previous high performance was driven by project-specific memorization. Temporal splits degrade further, but sliding window analysis suggests this is due to shifting project compositions rather than genuine temporal shifts in vulnerability patterns.
• Model Capacity and Context: Larger decoder-based models (e.g., Qwen3-Coder-Next) show clear gains over smaller encoders and baselines, particularly on CVE-mapped data. However, these gains are insufficient under strict false positive budgets. Neither intra-procedural context enrichment (control-flow/data-flow) nor inter-procedural repository context provided reliable improvements. At an FPR of 0.5% (0.005), every evaluated fine-tuned code-only model missed at least 80% of vulnerability fixes.
• Label Noise: Manual auditing revealed that label accuracy for CVE-associated commits is high (~91%), but accuracy for VFCs without CVE associations drops to ~29% (strict definition) or ~67% (lenient definition). This noise disproportionately affects evaluation, penalizing models for correctly rejecting mislabeled samples.
• Context Limitations: Error analysis indicates that the decisive evidence for a fix often lies outside the diff (e.g., in system interactions or attacker capabilities), and models fail to effectively utilize added semantic context even when it is provided.

Significance and Claims
The paper argues that current reported SPD performance largely reflects dataset correlations (specifically commit messages and project-specific patterns) rather than genuine code understanding. The authors claim that under strict evaluation conditions—code-only inputs and group-stratified splits—no measured configuration reliably detects VFCs from code changes alone with high confidence, despite larger models showing clear but insufficient gains over baselines.

The significance of this work lies in establishing that:

• Evaluation Protocols: Faithful estimates of code reasoning require code-only inputs and group-stratified splits; temporal splits on aggregated corpora are unreliable due to compositional shifts.
• Data Quality: Evaluations should prioritize CVE-confirmed positives, as models perform better on these cleaner datasets than on noisy data, and authors must explicitly state which definition of a "security fix" their labels capture.
• System Design: The primary bottleneck for SPD is not model capacity or the amount of context, but the ability to select and effectively use the specific context that experts rely on (often outside the diff). Future systems may require retrieval-augmented approaches, codebase-specialized models, or agentic systems to gather necessary context, rather than relying solely on fine-tuned classifiers on static diffs.

The authors conclude that while specialized classifiers remain relevant for screening, the field must move beyond current benchmark limitations to develop systems capable of grounding code changes in broader repository context.