TOSSS: a CVE-based Software Security Benchmark for Large Language Models

This paper introduces TOSSS, a CVE-based benchmark designed to evaluate the ability of Large Language Models to distinguish between secure and vulnerable code snippets in C/C++ and Java, revealing that current models achieve security scores ranging from 0.48 to 0.89.

The Gist

Imagine you are hiring a new assistant to write code for your company. You've heard great things about their intelligence, but you have a nagging worry: Are they actually safe?

In the world of software, "safe" means the code they write won't accidentally leave the front door unlocked for hackers. This paper introduces a new way to test if Artificial Intelligence (AI) assistants are actually good at keeping the doors locked.

Here is the story of TOSSS, the new security test, explained simply.

The Problem: The "Homework" Trap

Previously, researchers tested AI security by giving the AI a math problem or a coding task and asking it to write the answer from scratch.

• The Analogy: Imagine you want to test if a student knows how to build a safe house. You say, "Build me a house!" The student builds one, and then you hire a inspector to check if the locks work.
• The Flaw: This is hard. The inspector (a computer program) can only check for specific types of broken locks it already knows about. If a new type of lock-picking tool is invented tomorrow, the inspector won't know to look for it. Also, if the student writes a messy, confusing house, the inspector might get confused and miss the flaws.

The Solution: The "A or B" Game

The authors of this paper, TOSSS (Two-Option Secure Snippet Selection), decided to change the game. Instead of asking the AI to build a house, they ask it to choose between two houses.

• The Setup: You show the AI two versions of the exact same piece of code.
  • Option A: A house with a hidden trapdoor that hackers can use.
  • Option B: A house with a reinforced steel door.
• The Task: You simply ask, "Which one do you prefer?"
• The Score: If the AI picks the steel door every time, it gets a perfect score (1.0). If it picks the trapdoor, it gets a zero. If it guesses randomly, it gets a 0.5.

Why This is a Big Deal (The Superpowers of TOSSS)

1. It's Future-Proof (The "Living Library")
Old tests are like a static textbook; they only contain the security holes discovered up to the day the book was printed.
TOSSS is connected to a massive, living library of real-world security disasters called the CVE database (a list of every known software bug in history).

• The Analogy: Every time a new type of lock-picking tool is discovered in the real world, the TOSSS system automatically finds a real example of it, creates an "A vs. B" test, and adds it to the exam. The test grows smarter every day without human effort.

2. It's Unbiased (The "Blind Taste Test")
In the old tests, the AI might try to "game the system" by writing code that looks complex but is actually broken.
In TOSSS, the AI just has to pick a side. It's like a blind taste test between two soups. The AI can't "fake" its choice; it either knows which soup is safe, or it doesn't.

3. It's Clear (The "Report Card")
Instead of complex technical jargon, TOSSS gives a simple number between 0 and 1.

• 0.9: "This AI is a security expert."
• 0.5: "This AI is guessing like a coin flip."
• 0.3: "This AI is actually dangerous; it prefers broken code!"

What Did They Find?

The researchers tested 14 famous AI models (like GPT, Claude, Llama, etc.) using this new game. Here are the juicy results:

• Most are decent, but not perfect: The best models scored around 0.88 (very good), while the worst scored around 0.48 (basically guessing).
• The "Hint" Effect: When the researchers explicitly told the AI, "Hey, pick the safe one," most models got better. It's like telling a student, "Remember, safety is the goal!" before the test.
• The Surprise: The models specifically trained to be "Coding Experts" (like Codestral) didn't always win. Sometimes, they were worse at picking safe code than general-purpose models. It seems being a great writer doesn't automatically make you a great security guard.
• The Glitch: One model actually got worse when told to look for safety. It's as if the instruction confused it, making it overthink and pick the wrong door.

The Bottom Line

The world is rushing to use AI to write software, but we don't know if that software is secure.

TOSSS is a new, simple, and constantly updating "security exam" that helps us figure out which AI assistants are actually safe to trust. It moves away from asking AI to "write the perfect code" and instead asks, "Do you know what bad code looks like?"

If an AI can't tell the difference between a safe door and a trapdoor, we probably shouldn't let it build our houses. TOSSS is the tool that helps us find out who can tell the difference.

Technical Summary

Here is a detailed technical summary of the paper "TOSSS: a CVE-based Software Security Benchmark for Large Language Models."

1. Problem Statement

As Large Language Models (LLMs) become integral to software engineering workflows (e.g., GitHub Copilot), there is a critical need to assess their ability to produce secure code. Existing research indicates that LLMs frequently generate code containing security vulnerabilities (e.g., 40% of code generated by GitHub Copilot in one study contained vulnerabilities).

However, current benchmarks for evaluating LLM security skills suffer from significant limitations:

• Limited Extensibility: Most benchmarks rely on fixed task sets and static analyzers (like CodeQL). They cannot easily incorporate newly disclosed vulnerabilities or new programming languages without re-engineering the underlying detection rules.
• Evaluation Bias: Many rely on manual verification (which does not scale) or LLM-based judges (which introduces circular reasoning).
• Lack of Ground Truth: Code generation tasks produce complex artifacts that are difficult to match against a predefined "correct" reference implementation, forcing reliance on approximations.

2. Methodology: TOSSS

The authors introduce TOSSS (Two-Option Secure Snippet Selection), a novel benchmark designed to be extensible, scalable, and grounded in explicit truth.

• Core Task: Instead of asking LLMs to generate code, TOSSS presents them with a code selection task. The model is shown two versions of the same function: one vulnerable and one secure. The model must select the preferred version (Option A or B).
• Data Construction (CVE Mining):
  • TOSSS leverages the CVE (Common Vulnerabilities and Exposures) database as its source of truth.
  • It utilizes MegaVul, an automated pipeline that extracts code snippets from software repositories corresponding to CVE fixes.
  • For each CVE, the pipeline retrieves the function implementation before the fix (vulnerable) and after the fix (secure).
  • This creates a dataset of paired snippets where the "ground truth" is known by definition.
• Evaluation Settings:
  • Hintless: The prompt asks the model to choose between two versions without mentioning security.
  • With Hint: The prompt explicitly asks the model to "pick the most secure implementation."
• Scoring: The security score is the proportion of times the model selects the secure snippet.
  • 1.0: Always selects secure code.
  • 0.5: Random guessing.
  • < 0.5: Bias toward vulnerable code.

3. Key Contributions

1. Novel Benchmarking Paradigm: Shifts from code generation to code selection, enabling a direct comparison against ground-truth pairs derived from real-world vulnerability fixes.
2. Extensibility: By relying on the CVE database and automated mining, TOSSS can continuously integrate newly disclosed vulnerabilities and new programming languages without manual curation or static analyzer updates.
3. Scalability and Consistency: The binary choice output (A or B) eliminates parsing errors common in code generation benchmarks and allows for the evaluation of thousands of test cases efficiently.
4. Interpretability: The score provides a direct probabilistic interpretation of a model's security preference, unlike complex metrics like pass@k.
5. Open Source Release: The authors released the benchmark code and dataset to facilitate reproducibility and adoption.

4. Experimental Results

The authors evaluated 14 widely used LLMs (both open-source and closed-source) on 500 C/C++ and 500 Java functions.

• Performance Range: Scores ranged from 0.48 (Mistral Large 2512 on C/C++ hintless) to 0.89 (GLM-5 on C/C++ hintless).
• Impact of Hints:
  • Explicitly mentioning security in the prompt generally improved performance (average increase of +0.021 for C/C++ and +0.029 for Java).
  • Exception: Codestral 2508 (a coding-specialized model) performed worse when given the security hint, suggesting that explicit security instructions may interfere with models optimized strictly for code generation fluency.
• Model Families: Newer and larger models within the same family (e.g., Claude Opus 4.6 vs. Claude 3.5 Sonnet) generally showed better security selection capabilities.
• Coding vs. Security: Models explicitly optimized for coding tasks (Qwen3 Coder Next, Codestral 2508) did not necessarily outperform general-purpose models in secure code selection. In fact, some ranked in the bottom four.
• Language Comparison: Performance was comparable between C/C++ and Java, suggesting the benchmark measures general security reasoning rather than language-specific patterns.

5. Significance and Implications

• Complementary Metric: TOSSS is not intended to replace code generation benchmarks but to serve as a complementary, security-focused metric. It addresses the "minimal condition" for secure coding: if a model cannot distinguish a secure fix from a vulnerability, it is unlikely to generate secure code from scratch.
• Training Insights: The results suggest that current training data (often containing vulnerable code) and optimization for functional fluency may hinder security awareness. Fine-tuning on vulnerability-fix pairs (similar to the TOSSS construction) could improve security skills.
• Prompt Engineering: The improvement seen with "hints" indicates that many models possess latent security knowledge that is not activated without explicit instruction.
• Future-Proofing: The CVE-based approach ensures the benchmark remains relevant as new vulnerabilities are discovered, overcoming the stagnation issues of static-analyzer-based benchmarks.

In conclusion, TOSSS provides a robust, scalable, and interpretable framework for measuring LLM security capabilities, revealing that while many models have the potential to select secure code, their performance varies significantly and often requires explicit guidance to be fully realized.