Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making
This paper demonstrates that while Large Language Models can approximate structured cybersecurity reasoning when provided with explicit attack graph structures, their decision-making capabilities are fragile, highly sensitive to prompt framing, and fail to robustly scale with complexity compared to formal optimization baselines.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the high-stakes world of cybersecurity, organizations face a constant, invisible war. Attackers probe for weaknesses in networks, looking for a way to slip past defenses and steal data or disrupt services. Defenders must decide how to spend their limited resources to stop them. They cannot protect everything with the strongest possible shield; they must choose which locks to install, which alarms to set, and which doors to reinforce. This is a problem of strategy, not just technology. It requires mapping out every possible path an attacker might take and figuring out which specific barriers will block the most dangerous routes. For decades, experts have used complex mathematical models to solve this puzzle, treating the network as a map and the defense as a game where the defender moves first, trying to anticipate the attacker's best response.
Recently, a new kind of tool has entered this arena: large language models. These are the powerful artificial intelligence systems capable of writing code, answering questions, and summarizing vast amounts of text. They have been trained on nearly all the public information available on the internet, including countless security reports and technical manuals. Because they know so much about the field, many hope they can act as expert advisors, helping defenders make better choices faster. But a critical question remains: do these systems truly understand the logic of the attack, or are they simply recognizing familiar words and guessing based on what they have read before? If a defender relies on an AI that only mimics expertise without grasping the underlying structure of the threat, the consequences could be severe.
A team of researchers set out to test this question with a controlled experiment. They did not ask the AI to write a poem about a hack or summarize a news article. Instead, they placed the models in a rigorous decision-making scenario. They presented the AI with seven different, realistic attack scenarios derived from real-world incidents, such as ransomware campaigns, supply-chain compromises, and attacks on industrial control systems. For each scenario, the researchers provided a detailed map of the network, showing how an attacker could move from an initial entry point to a critical target. The AI was given a list of available security controls, like firewalls or multi-factor authentication, each with a specific cost and a rating of how effective it would be. The task was simple but difficult: choose a combination of these controls that fit within a specific budget while minimizing the chance that an attacker would succeed.
To know if the AI was doing a good job, the researchers compared its choices against a gold standard. This standard was a specialized computer program built by mathematicians to find the absolute best possible defense for any given map. This program does not guess; it calculates every possibility to find the single strategy that offers the most protection. The researchers then asked the AI models to act as judges, rating the quality of different defense strategies, including their own, those of the mathematical program, and some deliberately poor strategies. They also tested whether the AI could write its own computer code to solve the problem from scratch.
The results revealed a picture of conditional competence. When the researchers gave the AI a clear, structured map of the attack paths, the models often produced strategies that were very close to the mathematically perfect solution. They seemed to understand that blocking a critical junction in the middle of an attack path was more valuable than patching a minor hole at the start. In these controlled settings, the AI appeared to be reasoning through the problem, not just recalling facts. However, this ability was fragile. The moment the researchers changed how the problem was presented, the AI's performance became erratic.
The study found that the AI's judgment was highly sensitive to how the options were labeled. In one striking test, the researchers took a deliberately weak defense strategy—one that would leave the network wide open—and simply renamed it "optimal strategy." When the AI evaluators saw this label, they frequently rated this terrible defense as one of the best options, often scoring it higher than the genuine, mathematically perfect solution. Conversely, when a strong strategy was given a negative label, its score dropped. This suggests that the AI was not deeply analyzing the security logic of the defense but was instead reacting to the suggestive words used to describe it. The models were easily swayed by framing, prioritizing the label over the actual content.
The researchers also discovered that the AI struggled when the complexity of the network increased. As the maps grew larger and the number of possible attack paths multiplied, the AI's ability to find the best solution faded. In some cases, the models began to ignore the specific structure of the map entirely. Instead of looking at the unique paths an attacker could take, they fell back on general knowledge, choosing controls that are famous in the industry, like encryption or multi-factor authentication, regardless of whether those controls actually blocked the specific paths in the scenario. When the researchers removed the map entirely and asked the AI to choose defenses based only on a text description, the models abandoned the structural reasoning completely. They simply picked the controls with the highest general effectiveness ratings, ignoring the unique topology of the attack.
Even when asked to write their own computer code to solve the problem, the AI showed a gap between understanding and execution. The models could generate code that looked correct and used the right high-level approach, but when tested on larger networks, the code was too slow to be useful. It took the AI-generated programs significantly longer to find a solution than the specialized mathematical solver, and on the largest maps, the AI's code often failed to finish the task at all. The AI could mimic the logic of the solution, but it could not build a robust engine to carry it out at scale.
These findings suggest that while large language models can approximate the behavior of an expert security analyst, they do not possess the reliable, structured reasoning required for high-stakes decision-making. They can produce good answers when the problem is clearly defined and the context is simple, but their performance collapses under pressure, ambiguity, or misleading labels. They are not yet capable of replacing human experts or formal mathematical tools. Instead, they function best as assistants that can offer suggestions, provided those suggestions are checked against a rigorous, structured analysis. The study concludes that in the world of cybersecurity, where a single error can lead to a massive breach, we cannot rely on the AI's intuition alone. We must ensure that the tools we use are grounded in the hard logic of the attack path, not just the soft power of a convincing label.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.