A Cautionary Evaluation of LLMs for TLS Normative Requirement Classification
This paper evaluates large language models for automating TLS normative requirement classification using NIST SP 800-52 Rev. 2, finding that while ensemble methods achieve 93.75% accuracy, systematic confusion between "SHOULD NOT" and "MAY" language renders current LLMs insufficiently reliable for standalone compliance auditing.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The internet relies on a digital handshake called Transport Layer Security, or TLS, to keep our private messages, bank transactions, and emails safe from prying eyes. This system works by agreeing on a specific set of rules for how to scramble and unscramble data. These rules are not invented on the fly; they are written down in massive, complex documents known as technical standards. These documents tell engineers exactly which rules are mandatory, which are merely suggestions, and which are forbidden. The language used is precise and legalistic, relying on specific words like "must," "should," and "may" to define the strength of a rule. A "must" means a rule is non-negotiable, while a "may" means it is optional. However, the text often gets tricky, mixing these words with conditions, such as saying a rule is discouraged unless a very specific emergency occurs. For security experts, getting these rules right is a matter of safety; misreading a suggestion as a permission could leave a system wide open to attack.
For years, the only way to turn these thousands of pages of text into a checklist a computer could understand was for human experts to read every line and manually translate the rules. This process is slow, expensive, and prone to human error. Recently, a new type of computer program called a large language model has emerged, capable of reading and understanding human language with remarkable skill. These models can answer questions, summarize texts, and even write code. This raised a hopeful question for the cybersecurity world: could these smart programs read the security standards and automatically figure out which rules apply to which digital connections, saving experts months of work? A team of researchers at the Fondazione Bruno Kessler in Italy set out to test this idea, not to see if the technology could write a story, but to see if it could perform a task where a single mistake could compromise the security of the entire internet.
The researchers focused on a specific section of a major security standard known as NIST SP 800-52, which governs how TLS should be configured. They chose a dataset of 144 specific digital connection patterns, known as cipher suites, which are the building blocks of secure communication. Their goal was to see if large language models could correctly identify the status of each pattern: was it a hard requirement, a recommendation, a permission, or a prohibition? To make the task fair and rigorous, they did not just ask the models to guess. Instead, they built a system that forced the models to think step-by-step. They provided the models with the exact definitions of the key words and asked them to follow a six-stage logic path, examining the text, finding keywords, checking for negative words, and then making a final decision. To ensure the results were robust, they ran this test on four different large language models and combined their answers using a voting system that gave more weight to the models that performed best.
The results were a mix of impressive capability and sobering limitation. The system worked well enough to get the answer right in nearly 94 percent of the cases. This was a slight improvement over using any single model on its own, proving that combining the "opinions" of multiple models can smooth out some of their individual quirks. However, the researchers found that this level of accuracy, while good for many tasks, is not good enough for security compliance. In the world of cybersecurity, a 6 percent error rate is catastrophic. If a tool is used to audit a system and it misses even one forbidden rule, that system could be deployed with a known vulnerability. The study showed that the models were not failing randomly; they were failing in a very specific, predictable way.
The trouble spot was a subtle but critical distinction in the language of the standards. The models struggled to tell the difference between a rule that says "do not do this, unless you absolutely have to" and a rule that says "you are allowed to do this." In the technical language of the standards, the first is a discouraged rule that is only allowed under strict conditions, while the second is a simple permission. The researchers found that more than half of the mistakes the models made involved confusing these two categories. The models would read a sentence that discouraged a certain practice but allowed it in emergencies, and they would interpret the "allowed" part as the main message, ignoring the "discouraged" warning. They saw the permission but missed the condition.
This failure was not fixed by the step-by-step thinking process the researchers had built into the system. Even when the models were forced to write out their reasoning, they still made the same mistake. They could correctly identify the words "not recommended" and "may be used," but they could not weigh them against each other correctly to reach the right conclusion. The researchers concluded that while these computer programs are powerful tools for reading text, they do not yet possess the deep understanding of nuance and conditional logic required to replace human experts in security auditing. The technology is not ready to work alone. The study suggests that these tools might be useful as a first pass to flag potential rules for a human to review, but they cannot be trusted to make the final call. Until the models can reliably understand the difference between a discouraged exception and a simple permission, the task of securing the internet's rules must remain in human hands.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.