💻 computer science

RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code

The paper introduces RealVuln, the first open-source benchmark evaluating 15 security scanners on 26 real-world Python repositories, revealing a consistent three-tier performance hierarchy where security-specialized tools outperform general-purpose LLMs, which in turn significantly surpass rule-based SAST solutions.

Original authors: John Pellew, Faizan Raza

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: John Pellew, Faizan Raza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a massive, complex house (your software code). You know there are hidden traps, loose floorboards, and unlocked windows (vulnerabilities) that burglars could use to break in. Your job is to find and fix them before the burglars arrive.

To help you, you hire three different types of security inspectors:

The Rule-Book Inspector (Rule-Based SAST): This guy carries a giant checklist. He only looks for things that match his list exactly. If a window is open, he checks it off. But if a burglar is hiding behind a curtain that looks like a window, he misses it. He's fast, but he's easily fooled by anything that doesn't look exactly like the picture in his book.
The Smart Generalist (General-Purpose LLM): This is a brilliant, well-read detective who has read every book in the library. They don't just look for open windows; they understand how a house works. They can guess where a burglar might hide based on the layout. However, they are generalists. They haven't specialized in "house security" specifically, so they sometimes get distracted, miss details, or get tired and stop working on certain rooms.
The Specialized Security Pro (Security-Specialized Scanner): This is a detective who has spent their entire life studying only house security. They know the specific tricks burglars use, they have a custom-built tool for every type of lock, and they are obsessed with finding every single flaw, even if it means they have to check a few harmless spots just to be sure.

The Big Experiment: RealVuln

The authors of this paper, Kolega, decided to put these three types of inspectors to the ultimate test. They didn't use fake, made-up houses (which is what most previous tests did). Instead, they built 26 real, intentionally broken houses (educational apps and Capture-The-Flag games) filled with 796 known traps and 120 "fake traps" designed to trick the inspectors.

They asked: Who finds the most real traps without getting too confused by the fake ones?

The Results: A Clear Three-Tier Race

The results were surprising and created a clear "Three-Tier" ranking:

🥇 The Winner: The Specialized Pro (Kolega.Dev)
This tool found 80% of the real traps. It was the best at catching the bad guys. However, it was a bit paranoid: it flagged some harmless spots as suspicious (false alarms). But in security, the authors argue, it's better to be paranoid than to miss a real break-in.
- Analogy: It's like a guard dog that barks at every squirrel, but it never misses a burglar.
🥈 The Runner-Up: The Smart Generalist (Claude Sonnet 4.6)
The best "general" AI found about 50% of the traps. It was much smarter than the Rule-Book Inspector, but it still missed half the real dangers. It was also more accurate (fewer false alarms) but didn't catch enough to be the top choice for high-stakes security.
- Analogy: A smart human detective who is good at solving crimes but gets distracted by a interesting conversation and misses a few clues.
🥉 The Loser: The Rule-Book Inspector (Semgrep, Snyk, SonarQube)
The traditional tools that rely on simple checklists performed the worst. They only found about 18% of the real traps. They were easily fooled by anything that didn't look exactly like their checklist.
- Analogy: A guard who only checks the front door. If the burglar comes through the chimney, the guard doesn't even look up.

Why Does This Matter?

The paper argues that in the world of cybersecurity, missing a danger is much worse than raising a false alarm.

If you miss one real vulnerability, a hacker can steal your data.
If you raise a false alarm, a human just has to spend 5 minutes checking it and saying, "Oh, that's fine."

Because of this, the authors created a special scoring system (called F3) that punishes missing a trap much more heavily than raising a false alarm. Under this system, the Specialized Pro wins by a huge margin.

The Catch (and the Future)

The paper admits a few things:

The Conflict of Interest: One of the authors works for the company that built the winning tool (Kolega). To fix this, they released everything—the code, the test data, and the scoring system—so anyone can check their work and prove they aren't cheating.
It's Just the Beginning: This test was only on Python code (one programming language) and "broken" test houses. The next version will test on real-world code and other languages like Java and JavaScript.

The Bottom Line

If you want to secure your software, don't just rely on old checklists or generic AI. The future of security lies in specialized tools that combine the smarts of AI with a deep, specific understanding of how hackers actually break into systems.

The paper concludes that while generic AI is a huge step up from old tools, it's not enough on its own. We need tools built specifically for security to keep our digital houses safe.

1. Problem Statement

Static Application Security Testing (SAST) tools are essential for secure software development but suffer from high false-positive rates (noise) and low recall (missed vulnerabilities).

The Noise Problem: Industry reports indicate false-positive rates as high as 99.5% for certain vulnerability classes, leading practitioners to ignore scanner alerts.
The Detection Gap: Existing fully automated tools often detect less than 25% of known vulnerabilities.
Benchmark Limitations: Current benchmarks (e.g., OWASP Benchmark, Juliet) rely on synthetic code that lacks real-world complexity. Others (e.g., SastBench) use closed datasets or focus on triage agents rather than raw detection. Vendor benchmarks often lack transparency and independent reproducibility.
The LLM Question: While Large Language Models (LLMs) promise better semantic understanding of code, there is no open, systematic benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on identical real-world code to determine if LLMs truly outperform traditional methods.

2. Methodology: The RealVuln Benchmark

The authors introduce RealVuln, the first open-source benchmark designed to evaluate SAST tools on real-world, intentionally vulnerable Python code.

Dataset Construction

Scope: 26 intentionally vulnerable Python repositories (educational and Capture-The-Flag applications like PyGoat, DVPWA, VAmPI).
Ground Truth: 796 hand-labeled entries, comprising:
- 676 True Vulnerabilities: Spanning 18 CWE families across 5 web frameworks (Flask, Django, FastAPI, etc.).
- 120 False-Positive Traps: Code patterns that appear suspicious but are safe (e.g., ORM parameterized queries), used to test specificity.
Labeling: All findings were manually reviewed with detailed metadata (CWE, severity, evidence, file/line location). The code is confirmed to be human-authored to prevent data contamination from LLM training sets.

Evaluation Framework

Scanners Tested: 15 configurations across three categories:
1. Rule-Based SAST (3): Semgrep, Snyk Code, SonarQube.
2. General-Purpose LLMs (10): 10 distinct models (Anthropic, Google, xAI, etc.) operating in "agentic mode" (using tools to read files, grep, and shell execute) with a shared prompt.
3. Security-Specialized (2): Kolega.Dev (purpose-built) and GitHub SecLab Taskflow Agent (multi-stage agentic auditor).
Matching Algorithm: Findings are matched against ground truth based on file path, CWE (allowing for acceptable alternatives), and line number (±10 lines).
Metrics:
- Primary Metric: F3 Score ( $\beta=3$ ). This weights Recall 9x higher than Precision. The authors argue that in security, the cost of a missed vulnerability (breach) far exceeds the cost of a false positive (analyst time).
- Secondary Metrics: F1 and F2 scores are reported for comparison.
- Cost Analysis: Token usage and provider pricing (as of March 2026) are tracked to calculate cost-per-100k lines of code (LOC).

3. Key Contributions

Systematic Comparison: The first head-to-head evaluation of Rule-Based, General-Purpose LLM, and Security-Specialized scanners on identical, real-world ground truth.
Open-Source Benchmark: A fully transparent framework including ground-truth labels, raw scanner outputs, scoring scripts, and an interactive dashboard. All artifacts are version-locked via a manifest hash to ensure reproducibility.
Living Benchmark: A community-driven roadmap to expand to multi-language support (JS, Go, Java) and production-grade targets (Type 2–5) in future versions.

4. Key Results

The evaluation reveals a distinct three-tier hierarchy in performance, which holds across F1, F2, and F3 metrics, though rankings within tiers shift based on the recall/precision weighting.

Tier 1: Security-Specialized Scanners

Leader: Kolega.Dev (F3 = 73.0).
- Achieved the highest recall (80.9%) and the best cost-efficiency ($25/100k LOC).
- Demonstrates that combining LLM reasoning with domain-specific architecture yields superior detection.
Outlier: GitHub SecLab Agent (F3 = 8.4).
- Despite using GPT-5.4 and a sophisticated threat-modeling pipeline, it scored below all Rule-Based tools.
- Reason: Its design focuses on depth (auditing only 3–5 threat categories per repo) rather than breadth, resulting in very low recall (7.7%).

Tier 2: General-Purpose LLMs

Leader: Claude Sonnet 4.6 (F3 = 51.7).
- Outperformed all Rule-Based tools by nearly 3x.
- Balanced precision (~~78%) and recall (~~50%).
Performance Variance: Performance varied significantly by model. Some frontier models (e.g., Opus 4.6) suffered from high timeout/failure rates (completing only 19/26 repos), dragging down their strict scores.
Cost: The best GP-LLMs were significantly more expensive (e.g., Claude Sonnet at $83/100k LOC) compared to Kolega.Dev.

Tier 3: Rule-Based SAST

Performance: All three tools (Semgrep, Snyk, SonarQube) scored below F3 = 18.
Semgrep: F3 = 17.7 (Recall ~17.5%).
SonarQube: F3 = 7.1 (Recall ~6.5%).
Insight: Rule-based tools struggle significantly with semantic vulnerabilities (e.g., SQL Injection, Command Injection) where data flow analysis is required, achieving only ~24–32% recall compared to ~83–95% for LLMs.

Cost-Efficiency

Kolega.Dev offers the best trade-off: highest detection at the lowest cost among non-free tools.
GP-LLMs are 3.3x to 5.4x more expensive than Kolega.Dev for 29–30% lower detection performance.
Rule-Based tools are cheap but provide negligible security coverage.

5. Significance and Conclusion

LLMs are a Major Advance: General-Purpose LLMs significantly outperform traditional Rule-Based SAST tools, particularly in detecting semantic and logic-based vulnerabilities that pattern matching misses.
Specialization is Key: However, simply prompting a general LLM is not enough. Security-Specialized architectures (like Kolega.Dev) that integrate domain-specific pipelines with LLM reasoning achieve the highest recall and reliability, proving that "breadth of coverage" is as critical as "analytical depth."
Metric Shift: The paper argues for F3 (recall-weighted) as the industry standard for security scanning, as it aligns with the reality that missing a vulnerability is far more costly than investigating a false positive.
Transparency: By open-sourcing the entire benchmark, the authors invite the community to audit, challenge, and extend the results, setting a new standard for objective security tool evaluation.

Limitations: The current version (v1.0) is limited to Python and intentionally vulnerable applications (high density). Future versions aim to include production code with realistic vulnerability densities and support for other languages.