How Reliable Are NVD CWE Labels? A Large-Scale Semantic Audit with Seclometry
This paper presents a large-scale semantic audit using the validated CWEAgent tool to reveal that nearly half of the CWE labels in the National Vulnerability Database (NVD) do not exactly match code-grounded vulnerability semantics, identifying structural error patterns and demonstrating that label reliability varies significantly by assigning organization and weakness type.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, every software flaw that could be exploited by a malicious actor is supposed to be cataloged in a massive public library called the National Vulnerability Database. Think of this database as a central filing system for the world's software problems. When a flaw is found, it gets a unique ID and a label that describes what kind of mistake caused it. This label is crucial because it acts like a sorting tag for security teams, researchers, and automated tools. If a tag says a problem is a "broken lock," security teams know to check for weak authentication. If it says "overflowing bucket," they look for memory errors. For years, everyone has assumed these tags are accurate and reliable, treating them as the absolute truth for building better security systems and measuring how well new tools work.
However, just because a label exists does not mean it is correct. The challenge lies in the fact that the people who write the initial reports often describe what happened—the symptom, like data being stolen—rather than the root cause, like a specific coding error that allowed the theft. A report might say "an attacker stole data," leading to a generic label, while the actual code reveals a very specific mechanism, such as a reused encryption key. If the label is wrong, it misleads everyone who relies on it, causing tools to miss real dangers or waste time on false alarms. Until now, no one had systematically checked the accuracy of these millions of labels at a large scale, largely because doing so requires reading the actual code and patches to understand the true nature of the error, a task too complex for simple automated checks.
A team of researchers set out to solve this problem by building a new kind of auditing tool. They created a system that does not just read the text of a vulnerability report but instead examines the actual code changes that fixed the problem. The system translates both the vulnerability and the official labels into a structured description of the underlying mechanics: what triggered the error, what security rule was broken, and how the code failed. By comparing the official label against this code-grounded description, the system can determine if the label is exactly right, if it is a defensible but broader description, or if it is simply wrong. The researchers tested this tool on a carefully curated set of one hundred known vulnerabilities to ensure it worked correctly, achieving a high level of accuracy. They then applied it to a massive collection of over fifteen thousand open-source vulnerabilities discovered between 2017 and 2026.
The results revealed a landscape far more nuanced than a simple list of right or wrong answers. The study found that nearly half of the official labels matched the code evidence perfectly. Another significant portion were not technically wrong but were imprecise, offering a broader category that was defensible but less specific than the evidence allowed. However, a small but critical fraction of the labels—about 3.6 percent—were directly inconsistent with the evidence, meaning the label described a different type of weakness than the one actually present in the code. The researchers discovered that the reliability of a label depended heavily on who assigned it. Some organizations consistently provided precise, accurate tags, while others frequently used broad or incorrect labels. Surprisingly, the severity of the vulnerability did not predict the accuracy of its label; the most dangerous flaws were just as likely to be mislabeled as the less critical ones.
Over time, the quality of these labels has shifted. While the rate of perfect matches has remained relatively stable, the number of labels that contradict the code evidence has grown in recent years, rising from about one to three percent in the early years of the study to three to six percent in the later years. The researchers identified six recurring patterns in these errors. The most common mistake was confusing the consequence of a flaw with its cause, such as labeling a vulnerability as "information exposure" when the root cause was actually a specific cryptographic error. Other frequent errors involved mixing up similar sub-types of memory errors or confusing different types of injection attacks. These mistakes were not random; they often stemmed from the way the labeling system itself is structured, where broad categories are easier to assign than specific ones, or where the initial report lacked the technical details needed to make the correct choice.
The study also highlighted that these errors are not isolated incidents but structural issues within the metadata ecosystem. Sometimes, a correct label is added by the original reporter, but a later update by the database administrators introduces a conflicting, incorrect label that remains on the record. In other cases, the original report simply omits the necessary technical details, forcing the labeler to guess, which leads to an error that is technically consistent with the report but wrong based on the code. The researchers concluded that while the database is a vital resource, users cannot treat every label as ground truth. Instead, they should look at who assigned the label and understand that a significant portion of the data requires human verification or a deeper look at the code to be truly reliable. The work suggests that while automated tools can help manage the growing backlog of vulnerabilities, the final judgment on what a flaw really is must remain grounded in the evidence of the code itself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.