Technical Summary: JavaVulBench

Problem Statement

Deep learning-based vulnerability detection has historically relied on C/C++ corpora (e.g., Big-Vul, Devign), leaving Java—a widely deployed enterprise language—under-evaluated. Existing Java-focused benchmarks (Vul4J, VJBench, CWE-Bench-Java, CVEfixes) suffer from critical methodological gaps: they often lack explicit non-vulnerable samples, line-level granularity, multiple realistic data splits, and unified evaluation harnesses. Crucially, prior evaluations frequently utilize random splits on multilingual corpora, a practice shown by Chakraborty et al. [5] to inflate performance metrics (F1 scores) by 15–40 points due to data leakage between training and testing sets (e.g., near-duplicate methods from the same project). There is a lack of infrastructure to audit pre-training contamination, making it difficult to distinguish genuine generalization from memorization.

Methodology

1. Dataset Construction (JavaVulBench)

The authors constructed a corpus of approximately 30,600 Java methods spanning 1,740 CVEs and over 700 projects (2015–2025).

• Mining: CVE records were merged from NIST, GitHub Security Advisories, and CVEfixes. Fix commits were resolved, and pre/post-fix snapshots were extracted using a tree-sitter grammar to preserve method, class, and file provenance. A noise filter removed test-only, formatting-only, and refactoring-only commits.
• Labelling: Vulnerable methods were paired with patched counterparts. Non-vulnerable (negative) samples were drawn from three strategies per vulnerable method: same file (3:1), same project (2:1), and random project (1:1).
• Granularity: Labels exist at both method and line levels, including CVE publication dates and CWE tags.
• Split Strategies: Five realistic partition strategies are provided as JSONL:
  1. Random: Optimistic reference.
  2. Project-Disjoint: No project overlap between train and test.
  3. Temporal: Trains on CVEs before 2023; tests on 2023–07 onwards.
  4. Deduplicated: Removes near-duplicates (Jaccard 0.8) before splitting.
  5. Unseen CWE-Family: Holds out entire CWE families to test cross-class generalization.

2. Unified Evaluation Harness

A single LlmPrediction schema unifies evaluation across three backend families:

• Encoder Classifiers: HuggingFace Transformers (CodeBERT, GraphCodeBERT, UniXcoder).
• Local Generative Models: Ollama-served models (e.g., DeepSeek-Coder-1.3B) in zero-shot and LoRA-fine-tuned configurations.
• API-Served LLMs: OpenRouter-routed models (GPT-4o, GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder, CodeLlama-13B).

The harness supports zero-shot, few-shot, and Retrieval-Augmented Generation (RAG) using a Java-specific knowledge base of MITRE CWE descriptions and OWASP/CERT API misuse patterns. To manage costs, API evaluations use a stratified 200-sample subset per split.

3. Leakage-Aware Evaluation

A pre-training contamination audit partitions test sets into "risky" (CVEs published before the model's training cutoff) and "clean" (published after) subsets. This allows metrics to be reported separately to prevent memorization from being conflated with generalization.

Key Contributions

1. The Dataset: A CVE-grounded corpus of ~30,600 methods with method/line labels, 225 CWE categories, and five realistic split strategies.
2. The Harness: A unified, single-command evaluation pipeline supporting encoder classifiers, local LLMs, and API LLMs under identical conditions.
3. Contamination Audit: A per-model audit mechanism to identify and separate potentially memorized test samples.
4. Reproducibility: A Dockerized, one-command reproduction pipeline with fine-tuned checkpoints and data archived on Zenodo.

Results

Impact of Split Strategy

Switching from random to project-disjoint splits reproduces the performance drop observed in C/C++ by Chakraborty et al. [5]:

• UniXcoder: F1 drops from 0.446 (random) to 0.298 (project-disjoint).
• GraphCodeBERT: F1 drops from 0.343 to 0.256.
• CodeBERT: F1 collapses to 0.000 on project-disjoint and temporal splits, as the model defaults to predicting the majority class due to a lack of overlapping API surfaces in its pre-training data.
• Conclusion: Project-disjoint splits should be treated as the default headline metric; random splits serve only as optimistic references.

Model Performance

• Encoders: Fine-tuned encoders (UniXcoder, GraphCodeBERT) achieve moderate F1 scores (~0.25–0.30) on project-disjoint splits.
• API LLMs: On a cost-matched 200-sample probe, GPT-4o (F1=0.419) and Claude Sonnet 4 (F1=0.416) outperform all encoder baselines. GPT-4o also demonstrates the best calibration (ECE=0.147).
• DeepSeek-Coder-v2: Shows a negative MCC (-0.108), indicating performance below random chance on Java vulnerability detection in its zero-shot configuration.
• CodeLlama-13B: Exhibits the highest PR-AUC (0.540), suggesting useful confidence ranking despite weaker threshold-based F1.

Contamination Audit

On the temporal split, models with 2024–2025 cutoffs (e.g., Claude Sonnet 4, GPT-4.1-mini) flag up to 59% of the test set as "risky." Interestingly, the strongest API models show higher F1 scores on the "clean" subset compared to the full set, suggesting the risky subset contains harder CVE variants rather than easily memorized ones.

Significance and Claims

The paper claims that split strategy is a first-class property of any vulnerability detection benchmark number. The primary empirical finding is that without project-disjoint or temporal splits, reported F1 scores for Java vulnerability detection are significantly inflated due to data leakage.

JavaVulBench provides the necessary infrastructure (unified harness, realistic splits, and contamination audits) to make these studies reproducible and to ensure that reported improvements reflect genuine generalization rather than memorization or data leakage. The authors position this work as filling the gap between existing Java benchmarks and the rigorous evaluation standards required for enterprise vulnerability detection.