💻 computer science

CrossCommitVuln-Bench: A Dataset of Multi-Commit Python Vulnerabilities Invisible to Per-Commit Static Analysis

This paper introduces CrossCommitVuln-Bench, a dataset of 15 real-world Python vulnerabilities that evade per-commit static analysis because their exploitable conditions span multiple benign commits, revealing that current SAST tools miss 87% of such cross-commit vulnerabilities even when analyzing the full codebase.

Original authors: Arunabh Majumdar

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arunabh Majumdar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a massive, busy office building. Your job is to stop bad guys from getting in. You have a very strict rule: You only look at one person at a time as they walk through the front door.

If a person walks in carrying a harmless-looking briefcase, you let them pass. If they walk in wearing a normal suit, you let them pass. You never see the whole picture because you only check the snapshot of the moment they arrive.

This is exactly how most modern software security tools work today. They are called SAST tools (Static Application Security Testing). They look at code changes (called "commits") one by one, like checking one person at the door.

The Problem: The "Slow-Motion" Heist

The paper introduces a new problem called Cross-Commit Vulnerabilities. This is like a heist where the bad guys don't break in all at once. Instead, they sneak in over several days, months, or even years, one tiny piece at a time.

Here is how the "heist" works in our office analogy:

Day 1 (Commit A): A nice employee brings in a harmless-looking keycard reader. It looks like a legitimate upgrade to the security system. The guard (the software tool) checks it, sees it's just a reader, and says, "All clear."
Day 30 (Commit B): Another employee brings in a master key that opens every door. Again, it looks like a normal maintenance task. The guard checks it and says, "All clear."
Day 60 (Commit C): A third employee removes the security camera that was watching the keycard reader. The guard sees the camera is gone but thinks, "Maybe it's just being upgraded," and lets it pass.

The Result: If you look at Day 1, Day 30, or Day 60 individually, everything looks safe. But if you look at the building after all three days, you realize: The bad guys can now walk in, use the keycard, open the master key, and walk right past the missing camera.

The vulnerability wasn't created in one big mistake; it was built slowly, piece by piece. Because the security guard only checks "one person at a time," they miss the entire crime.

What the Paper Did

The researcher, Arunabh Majumdar, decided to test this theory. He created a special "training course" called CrossCommitVuln-Bench.

The Class: He gathered 15 real-world examples of Python software vulnerabilities (CVEs) where this "slow-motion heist" actually happened.
The Test: He took two popular security guards (tools named Semgrep and Bandit) and asked them to check these 15 cases.
- Test 1 (Per-Commit): "Check each day's work individually."
- Test 2 (Cumulative): "Check the whole building after all the work is done."

The Shocking Results

The results were scary for software developers:

The "One-by-One" Blind Spot: When the tools checked the code day-by-day, they found the problem in only 1 out of 15 cases (13%). In the other 14 cases, the tools said, "Everything is fine!" because they couldn't see the connection between the different days.
Even the "Whole Building" Check Failed: Even when the tools looked at the entire codebase at once (the cumulative check), they still missed 73% of the problems.
- Why? Because the tools are bad at spotting "missing things" (like a missing security camera) or "hidden things" (like a secret keycard reader that looks like a normal tool).

Why Does This Matter?

Think of it like building a house.

Old Way: You check the bricks as you lay them. If a brick looks fine, you move on.
New Reality: The house is being built by a team where one person lays a brick, another paints it, and a third removes a support beam. If you only check the brick when it's laid, you miss the fact that the house will collapse because the beam is gone.

The paper argues that our current security tools are like the "check-the-brick" method. They are great at catching obvious mistakes, but they are blind to complex, multi-step attacks that happen over time.

The Takeaway

We need new security tools that don't just look at "snapshots" of code. We need tools that remember the history of the code, like a detective who looks at the whole timeline of events, not just the crime scene on a single Tuesday.

The author has released this dataset (the 15 examples) to the public so that other researchers can build these "detective-style" tools to catch the slow-motion heists before they happen.

In short: If you only check the code one change at a time, you are leaving the back door wide open for clever hackers who know how to sneak in slowly.

1. Problem Statement

Current Static Application Security Testing (SAST) tools (e.g., Semgrep, Bandit, CodeQL) operate on snapshots (single commits, pull request diffs, or point-in-time file trees). They assume that dangerous code is introduced in a single change and is visible in isolation.

The paper challenges this assumption by identifying Cross-Commit Vulnerabilities: security flaws where the exploitable condition is introduced across multiple commits over time (days, months, or years).

Mechanism: A vulnerability chain typically consists of:
- Commit A: Introduces a "taint source" (e.g., a new user-controlled parameter, a relaxed permission check) which appears benign in isolation.
- Commit B: Introduces a "sink" (e.g., a dangerous operation, a missing guard) or removes a guard, which also appears benign in isolation.
The Gap: The dangerous data flow only exists when both commits are present. Consequently, per-commit analysis tools miss these vulnerabilities because they lack state across commit history. Existing datasets (Juliet, D2A, Devign) focus on single-function or single-file vulnerabilities, leaving a gap in research regarding multi-commit introduction patterns.

2. Methodology

The authors constructed a benchmark dataset and evaluated standard SAST tools against it using the following methodology:

A. Dataset Construction (CrossCommitVuln-Bench)

Source: Mining the GitHub Security Advisory Database (GHSA) via the OSV API for high/critical PyPI advisories.
Filtering Pipeline:
1. Identified 1,200 advisories; 696 had verifiable fix commits.
2. Selected candidates where the fix modified $\ge$ 2 Python files.
3. Used git blame to trace vulnerable lines to their introducing commits.
4. Manual Archaeology: Researchers manually reviewed the diffs of introducing commits to verify they were individually benign and that the vulnerability required $\ge$ 2 commits to manifest.
Selection Criteria: A CVE was included only if it satisfied five criteria:
1. Multi-commit: $\ge$ 2 distinct commits introduce the flaw.
2. Individually Benign: Each commit looks like a legitimate feature change.
3. Collectively Critical: The combination is exploitable (CVSS $\ge$ 7.0).
4. Open Source: Public repository with full history.
5. Reproducible: The vulnerable state can be reconstructed.
Final Dataset: 15 real-world Python CVEs (9 Critical, 6 High) spanning 15 unique repositories.

B. Annotation Schema

Each CVE is annotated with a structured JSON file containing:

The sequence of contributing commits (hash, date, role: Source/Sink/Guard Removal).
Rationale for why each commit evades detection in isolation.
Baseline SAST results (Semgrep/Bandit findings) for both per-commit and cumulative modes.

C. Experimental Evaluation

The authors evaluated two popular SAST tools: Semgrep (v1.154.0) and Bandit (v1.9.4).

Per-Commit Mode (CCDR): Scanned each contributing commit individually.
Cumulative Mode (CDR): Scanned the full codebase immediately before the fix commit (all commits present).
Metrics:
- CCDR: Fraction of CVEs caught on the specific commit that introduced the element.
- CDR: Fraction of CVEs caught when the full codebase is available.

3. Key Contributions

CrossCommitVuln-Bench: The first curated dataset of 15 real-world Python CVEs specifically characterized by multi-commit introduction chains.
Open Annotation Schema: A structured format capturing the commit chain, the rationale for evasion, and SAST performance data.
Baseline Evaluations: Reproducible benchmarks showing that standard tools fail to detect these vulnerabilities even in cumulative mode.
Public Release: The dataset, scripts, and annotations are released under open-source licenses (CC-BY-4.0/MIT) to facilitate research.

4. Key Results

The evaluation revealed a significant failure in current SAST paradigms:

Per-Commit Detection Rate (CCDR): 13%
- Only 2 out of 15 CVEs were detected on the specific commit that introduced the vulnerability.
- Qualitative Failure: Even the 2 "detections" were practically useless:
  - One was flagged on a commit framed as a "security fix," leading developers to suppress the alert.
  - The other detected a minor hardcoded key (LOW severity) but completely missed the primary vulnerability (200+ unprotected API endpoints).
- Conclusion: 87% of multi-commit chains are invisible to per-commit SAST.
Cumulative Detection Rate (CDR): 27%
- Even with the full codebase present, 73% of vulnerabilities remained undetected.
- Reasons for Failure:
  1. Custom Wrapper Opacity: Dangerous operations wrapped in internal helper functions (e.g., exec_cmd()) are not recognized by name-pattern rules.
  2. Absent-Guard Invisibility: Tools cannot detect the absence of authentication or validation dependencies (e.g., missing @auth_required).
  3. Temporal Separation: The source and sink are separated by 21 to 1,342 days, requiring cross-commit state tracking which snapshot-based tools lack.

5. Significance and Future Directions

Paradigm Shift: The paper demonstrates that snapshot-based SAST is fundamentally insufficient for modern software development where vulnerabilities are often the result of incremental, seemingly benign changes over time.
Research Impact: The dataset provides a necessary ground truth for developing Cross-Commit Detection approaches, such as graph-based analysis (e.g., POSTURA) or commit-sequence anomaly models.
CI/CD Implications: Current CI/CD security gates (per-PR scans) are likely missing critical vulnerabilities. The authors suggest a need for systems that maintain persistent security state across the commit history.
Limitations & Extensions: The current dataset is Python-specific and annotated by a single researcher. Future work aims to expand to other languages (JS/TS, Java) and formalize inter-annotator agreement studies.

In summary, CrossCommitVuln-Bench proves that the industry's reliance on per-commit static analysis leaves a massive blind spot for vulnerabilities that evolve over time, necessitating a new generation of security tools capable of reasoning across commit histories.