Proactively Detecting Threats: A Novel Approach Using LLMs

Technical Summary: Proactively Detecting Threats Using Large Language Models

Problem Statement
Enterprise security is increasingly challenged by sophisticated malware and expanding digital attack surfaces, exacerbated by remote work and cloud adoption. Current cybersecurity paradigms are predominantly reactive, relying on known signatures or post-compromise detection, which leaves organizations vulnerable to zero-day attacks and polymorphic malware families. Furthermore, while structured threat intelligence standards (like STIX 2.0) exist, industry adoption remains fragmented. Consequently, 69% of security teams rely on unstructured, free-form threat reports from diverse web sources. These reports contain Indicators of Compromise (IOCs) in inconsistent formats (e.g., images, hashes, text), making automated extraction and classification difficult. There is a critical gap in proactive systems capable of parsing unstructured web-based intelligence to identify IOCs before they are cataloged in traditional databases.

Methodology
The authors developed an automated system to evaluate the efficacy of Large Language Models (LLMs) in proactively identifying IOCs from unstructured web sources. The methodology involved three primary stages:

1. Data Collection and Parsing: An automated web crawler harvested raw data from 15 reputable threat intelligence sources (including Sophos, SentinelOne, and Abuse.ch) via RSS/Atom feeds. A parser utilizing regular expressions extracted specific patterns corresponding to IOCs: IPv4/IPv6 addresses, domains, URLs, and file hashes (SHA1, MD5, SHA256). The system also mapped extracted terms to the MITRE ATT&CK framework to enrich the context.
2. Dataset Construction: The study analyzed 479 webpages containing 2,658 IOCs (711 IPv4, 502 IPv6, 1,445 domains). A specific test subset of 303 webpages was curated, containing 251 confirmed malicious indicators (verified via context and VirusTotal) and benign indicators.
3. LLM Evaluation: Six pre-trained LLM variants (Gemini 2.0 Flash-Lite, Gemini 1.5 Pro, Qwen3 30B, Qwen3 32B, Llama 8B, and Llama 70B) were evaluated without fine-tuning. The models were prompted to act as cybersecurity experts, analyzing the full text of a webpage (context) to determine if a specific extracted entity was an Indicator of Compromise. The evaluation focused on classification metrics: Precision, Recall, Specificity, and F1-score.

Key Contributions

• Systematic Evaluation of LLMs for Proactive Detection: This paper presents the first systematic evaluation of LLMs for extracting IOCs from unstructured web-based threat intelligence, distinguishing the approach from reactive malware detection.
• Automated Unstructured Data Pipeline: The authors engineered a pipeline that converts diverse, non-standardized threat reports into structured data, addressing the lack of unified approaches for multi-source threat intelligence.
• Comprehensive Benchmarking: The study provides a comparative analysis of six distinct LLM architectures across a real-world dataset of nearly 2,700 IOCs, offering empirical data on the trade-offs between recall and precision in security contexts.

Results
The evaluation revealed significant performance variations among the models:

• Gemini 1.5 Pro: Achieved the highest performance balance, with a precision of 0.958 and specificity of 0.788 for malicious IOC identification. Notably, it demonstrated perfect recall (1.0) for actual threats, meaning it successfully identified all malicious indicators in the test set.
• Qwen3 32B: Exhibited the lowest performance, characterized by a high number of false negatives (51) when classifying non-malicious data, resulting in a low F1-score.
• Llama 70B: Successfully identified all malicious indicators (TP=251) but suffered from higher false negatives compared to Gemini models, leading to lower recall for the malicious class.
• General Trends: Models generally excelled at high recall (catching threats) but struggled with specificity (correctly identifying benign data). The study noted that models often misidentified benign context, such as domain registrars or research repositories, as malicious due to the presence of keywords like "abuse" or "killer" within the text.

Significance and Claims
The paper claims that LLMs offer a viable mechanism for shifting cybersecurity from reactive to proactive measures by automating the extraction of IOCs from the vast volume of unstructured web reports. The authors emphasize that while high recall is paramount for threat detection, the observed trade-off with precision (false positives) highlights the need for improved model refinement and context-aware classification.

The authors modestly conclude that while LLMs demonstrate cutting-edge performance in identifying malicious indicators through contextual understanding, they currently face challenges in accurately distinguishing between malicious indicators and benign entities that share similar linguistic features (e.g., domain names containing "abuse" or "dark"). The work establishes a foundation for future automation in enterprise security systems but acknowledges that further research is required to handle complex indicator types and reduce hallucinations in unstructured data parsing.