🤖 AI

Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

This paper presents an empirical study revealing that the primary vulnerabilities of LLMs in Cyber Threat Intelligence stem from the domain's inherent heterogeneity and volatility rather than generic model flaws, introducing a human-in-the-loop framework to identify and mitigate three specific cognitive failure modes through targeted defenses.

Original authors: Yuqiao Meng, Luoxi Tang, Feiyang Yu, Jinyuan Jia, Guanhua Yan, Ping Yang, Zhaohan Xi

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuqiao Meng, Luoxi Tang, Feiyang Yu, Jinyuan Jia, Guanhua Yan, Ping Yang, Zhaohan Xi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Overworked Detective

Imagine a massive, chaotic library where new books about dangerous criminals (cyber threats) are being written every single second. Security analysts are the detectives trying to read these books to stop the criminals. But there are too many books, too fast.

To help, they hired a team of super-smart AI assistants (Large Language Models, or LLMs). These AIs can read thousands of pages in a blink. However, the researchers in this paper discovered that while these AI detectives are fast, they are making dangerous mistakes because the library itself is messy, contradictory, and constantly changing.

The paper argues that the problem isn't just that the AI is "hallucinating" (making things up); the problem is that the threat landscape (the library) is too messy for the AI to handle correctly.

The Three Main Ways the AI Gets Confused

The researchers found that the AI fails in three specific ways, which they call "cognitive failures."

1. The "Coffee Shop Gossip" Problem (Spurious Correlations)

The Analogy: Imagine you walk into a coffee shop and see two people, Bob and Alice, sitting at the same table. You assume they are best friends. But in reality, they just happened to sit there at the same time; they don't even know each other.
The AI Mistake: The AI sees two things mentioned together in a report (like a specific computer virus and a specific country) and assumes they are connected. It creates a false link.

Real Example: The AI sees a virus and a specific type of server mentioned in the same news article. It concludes the virus must be attacking that server, even though the article was just listing them as unrelated examples. The AI mistakes "being mentioned together" for "being related."

2. The "He Said, She Said" Problem (Contradictory Knowledge)

The Analogy: Imagine you are trying to figure out the weather. One friend says, "It's raining!" Another says, "It's sunny!" A third says, "It's snowing!" If you ask an AI to decide the weather based on these three friends, it gets confused and might say, "It's raining-snowing-sunny," or it might flip-flop between answers.
The AI Mistake: Cyber threat reports often disagree. One security company says a patch fixes a bug; another says the bug is still there. The AI tries to blend these conflicting stories, resulting in a confused, unstable answer that doesn't help anyone.

Real Example: One report says a hacker group is using a specific tool. Another says they switched tools. The AI gets stuck in the middle, unable to decide who is right, leading to a weak defense plan.

3. The "Old Map" Problem (Constrained Generalization)

The Analogy: Imagine a GPS that was trained only on roads from 2010. If you try to drive it to a brand-new city built in 2025 with flying cars and underground tunnels, the GPS will try to force your new route onto the old 2010 roads. It will tell you to turn left where a building now stands.
The AI Mistake: The AI has studied millions of past attacks. When a brand-new, never-before-seen type of attack happens (a "zero-day"), the AI tries to force it into an old category. It misses the new threat because it's too focused on what it has seen before.

Real Example: A hacker uses a brand-new trick to break into a system. The AI, having never seen this trick, assumes it's an old, common trick and suggests an outdated defense that doesn't work.

How They Found These Problems

The researchers didn't just guess; they built a special "Detective Training Camp."

The Human-in-the-Loop: They knew that asking an AI to grade its own homework is a bad idea (it will just say, "I'm perfect!"). So, they created a system where humans checked the AI's work.
The Filter: They took thousands of AI failures and sorted them into buckets. They found that the "Coffee Shop" and "He Said/She Said" problems were the most common.
The Stress Test: They tried to trick the AI by changing small, unimportant details (like the date or the name of a city) in a report. If the AI changed its answer just because the date changed, they knew the AI was relying on the wrong clues (like the date) instead of the real evidence (the virus code).

The Solution: Teaching the AI to Think Causally

The paper suggests that instead of just feeding the AI more data, we need to change how it thinks.

The "What If?" Test (Causal Intervention):
The researchers taught the AI to ask itself: "If I change this date or this location, does my conclusion still hold?"

If the answer is "No, my conclusion changed," the AI realizes it was relying on a coincidence (like the date) rather than the real cause.
If the answer is "Yes, my conclusion stays the same," the AI knows it found the real truth.

The Result:
When they applied this "What If?" test, the AI made significantly fewer mistakes. It stopped guessing based on gossip and started looking for solid proof.

Summary

The paper concludes that AI is a powerful tool for cybersecurity, but it is currently too easily confused by the messy, contradictory nature of real-world threat reports.

To fix this, we can't just build bigger AI models. We have to build smarter processes that force the AI to:

Ignore coincidental connections.
Handle conflicting reports without panicking.
Recognize when it is facing a brand-new threat it hasn't seen before.

By doing this, we can turn these AI assistants from confused interns into reliable, expert detectives.

Technical Summary: Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

Problem Statement

Large Language Models (LLMs) are increasingly deployed to automate Cyber Threat Intelligence (CTI) tasks, ranging from vulnerability assessment to incident response. However, significant reliability gaps persist in operational workflows. While existing literature often attributes these failures to generic model limitations such as hallucination or jailbreaking, this paper argues that the dominant bottleneck is the unique nature of the threat landscape itself. CTI data is characterized by heterogeneity, volatility, and fragmentation. Evidence is often intertwined, crowdsourced, and temporally unstable—properties that standard LLM evaluations rarely capture. Consequently, LLMs struggle to reason effectively under these conditions, leading to substantial failure rates in critical security tasks.

Methodology

The authors propose a comprehensive empirical study and a novel methodological framework to identify and categorize LLM failures in CTI reasoning.

1. Large-Scale Empirical Evaluation

The study evaluates a diverse suite of models, including industry-leading general-purpose LLMs (e.g., GPT-5, Claude-Sonnet-4, Gemini-2.5) and cybersecurity-specialized agents (e.g., DeepHat, SecGPT, Cyber-Zero). The evaluation spans the complete CTI lifecycle across four stages:

Contextualization: Enriching raw observations with context (e.g., mapping logs to CVEs).
Attribution: Identifying adversaries and linking events to actor profiles.
Prediction: Forecasting exploitation likelihood and campaign evolution.
Mitigation: Recommending patches, detection rules, and response playbooks.

Datasets include established benchmarks (CTIBench, SevenLLM-Bench, SWE-Bench, CyberTeam) and real-world threat reports from authoritative sources (NVD, CISA, vendor advisories).

2. Human-in-the-Loop Failure Categorization

To address the challenge of reliably classifying failure modes without hard labels, the authors introduce a stratified, human-in-the-loop framework:

Stratified Sampling: Instances are partitioned into quantile strata based on similarity scores between model outputs and reference materials. Manual inspection of "anchor" samples within each stratum establishes score ranges for "correct" vs. "failed" outputs, minimizing manual workload to <1.5% of total instances.
Iterative Taxonomy Refinement: An autoregressive process alternates between LLM-assisted classification and human inspection. New failure patterns identified by humans are added to the taxonomy until convergence.
Multi-Agent Deliberation: To ensure stability, multiple LLMs independently classify instances, followed by a second round of deliberation. Human inspectors only intervene when agent consensus is lacking or predictions fluctuate across rounds (<1.8% of cases).

Key Contributions

1. Identification of Three Domain-Specific Cognitive Failures

The study identifies three dominant categories of vulnerabilities that constrain LLM effectiveness in CTI, distinct from generic LLM flaws:

Spurious Correlations: LLMs over-attribute based on superficial or co-occurring features rather than causal links. Examples include mistaking commodity tools for actor-specific evidence (Co-mention bias) or over-attributing infrastructure reuse based on deliberately recycled Indicators of Compromise (IoCs).
Contradictory Knowledge: Inconsistencies across CTI sources (e.g., conflicting reports, temporal contradictions between outdated and recent evidence, semantic conflicts in naming) confuse models, leading to unstable or conflicting predictions.
Constrained Generalization: LLMs struggle to extend beyond familiar distributions, failing on emerging (zero-day) threats or unseen patterns. This includes distributional bias (over-reliance on common training data) and environmental unawareness (ignoring specific deployment constraints).

2. Root Cause Analysis Across CTI Stages

The paper maps these vulnerabilities to specific CTI stages and techniques. For instance, Co-mention bias often arises during Contextualization when retrieval systems treat co-occurring entities as causally related. Contradictory Knowledge frequently destabilizes Attribution and Mitigation when sources disagree on actor identity or patch efficacy. Constrained Generalization failures are most prominent in Prediction and Mitigation when models encounter novel attack vectors or environmental constraints not present in training data.

3. Validation via Causal Interventions

The authors validate these root causes through causal intervention studies. By perturbing non-causal contextual tokens (e.g., dates, sectors) while preserving causal attributes (e.g., TTPs, exploit mechanisms), they demonstrate that LLM predictions often flip, confirming reliance on spurious correlations.

4. Defensive Strategies

The paper proposes and evaluates targeted mitigation strategies:

Causal Intervention ( $C^3$ ): An inference-time counterfactual consistency check that forces models to verify if conclusions remain stable when non-causal context is altered.
Fine-Grained Mitigations: Specific techniques for each vulnerability subtype, such as retrieval-time evidence filtering (to remove co-mention bias), temporal-aware reranking (to resolve contradictions), and novelty-aware inference (to handle unseen patterns).

Results

Performance Gaps: Evaluation reveals universal gaps across all models. General-purpose LLMs excel in understanding-heavy tasks (e.g., source reliability scoring) but struggle with operational outputs. Cyber-specialized agents show better performance in specific operational rankings but suffer from constrained generalization on emerging threats.
Failure Rates: Without mitigation, failure rates are significant across all stages. For example, general-purpose models show high failure rates in False Flag Detection and Infrastructure Reuse.
Effectiveness of Defenses:
- The proposed $C^3$ causal intervention yields universal performance gains, with notable improvements in False Flag Detection (up to +0.115 F1) and Event Timeline Construction (avg Δ ≈ 0.08).
- Fine-grained mitigations significantly reduce vulnerability ratios. A "Full-fledged" approach (combining retrieval filtering and inference constraints) reduces failure rates across all categories. For instance, Temporal Contradiction failures in general LLMs dropped from 17.53% to 4.25% with retrieval-only mitigation, while Spurious Correlation failures were best addressed by inference-only checks.

Significance and Claims

The paper claims to uncover "blind spots" in current LLM vulnerabilities specific to the CTI domain. It argues that standard model adaptation or prompt engineering is insufficient because the failures are rooted in the intrinsic properties of the threat landscape (heterogeneity, volatility, fragmentation).

The significance of this work lies in:

Shifting the Focus: Moving the discourse from generic LLM flaws (hallucination) to domain-specific reasoning failures driven by the nature of cyber threat data.
Providing a Roadmap: Offering a concrete, taxonomy-informed path for building resilient, domain-aware CTI agents.
Demonstrating Feasibility: Showing that lightweight, targeted defenses (causal consistency checks and retrieval filtering) can substantially mitigate these specific failure modes without requiring massive retraining.

The authors conclude that robust LLM-based CTI analysis requires protections explicitly tailored to the heterogeneity of the threat landscape, providing foundational insights for future principled adaptations in cyber-focused LLM systems.