CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

This paper introduces CyberThreat-Eval, an expert-annotated benchmark derived from real-world Cyber Threat Intelligence workflows that addresses the limitations of existing evaluations by assessing Large Language Models across the full triage-to-reporting pipeline using analyst-centric metrics, revealing significant gaps in current models' ability to handle nuanced, actionable security insights.

The Gist

Imagine you are the captain of a massive ship (a global tech company) sailing through a stormy ocean filled with invisible pirates (cybercriminals). Your job is to spot these pirates, figure out their plans, and warn your crew before they attack.

For years, this job has been done by human "Lookouts" (Security Analysts). They spend hours reading thousands of news reports, forums, and blogs to find clues. It's exhausting, slow, and prone to human error.

Enter Large Language Models (LLMs)—the AI assistants that promise to do this work for us. But here's the catch: Can these AI assistants actually do the job, or are they just fancy parrots repeating what they've heard?

This paper, titled "CyberThreat-Eval," is like a rigorous "driving test" for these AI assistants. The authors (from Microsoft and HKUST) built a new, realistic exam to see if AI can truly replace or help human lookouts.

Here is the breakdown of their findings, explained simply:

1. The Old Tests Were Like "Multiple Choice" Quizzes

Previous tests for AI in cybersecurity were like school quizzes: "Who is the bad guy in this story? A) Bob, B) Alice, C) Charlie."

• The Problem: Real security work isn't a quiz. A human lookout doesn't get a list of options. They get a messy pile of news articles and have to figure out: "Is this important? Who is attacking us? How do they do it?"
• The New Test: The authors created CyberThreat-Eval, a test that mimics the real job. It has three stages, just like a real human analyst's day:
  1. The Triage (The Filter): Sifting through a mountain of trash mail to find the few letters that might be threats.
  2. The Deep Dive (The Detective): If a threat is found, the AI must go hunting for more clues on the internet to build a full picture.
  3. The Report (The Storyteller): Writing a clear, actionable report for the captain that explains who did it, how they did it, and what to do next.

2. The Results: AI is a Great Researcher, but a Bad Detective

The authors tested four different AI models. Here is what they found:

• The "Triage" Problem (Too Many False Alarms):
  Imagine a smoke detector that goes off every time you toast bread. The AI was great at finding potential threats (it rarely missed a real one), but it also flagged harmless articles as dangerous.

  • Analogy: It's like a security guard who stops everyone entering the building, even the people just delivering pizza. It creates too much work for the humans to clean up.

• The "Deep Search" Surprise:
  When asked to find more clues, the standard AI models (like GPT-4o) were actually better at finding new useful links than the specialized, "fine-tuned" models.

  • Analogy: The specialized models were like experts who only read one specific type of book. They knew a lot about that one book but didn't look outside their library. The general models were like curious kids who ran around the whole library and found hidden gems.

• The "Storytelling" Struggle:
  When asked to write the final report:

  • Good at "How": The AI was excellent at explaining the mechanics of an attack (e.g., "They used a virus to break the door").
  • Bad at "Who" and "Why": The AI struggled to explain who the bad guys were and their motives. It often gave vague answers or made things up (hallucinations).
  • Analogy: The AI is great at describing the crime scene (broken window, muddy footprints) but terrible at identifying the criminal or their motive. It might say, "The thief is a guy named Bob," when the real thief is a group called "The Shadow Syndicate."

3. The Solution: The "Threat Research Agent" (TRA)

Since the AI isn't perfect on its own, the authors built a hybrid system called TRA. Think of this as a Team of Two: The AI and a Human Expert.

• How it works:

  1. The AI does the heavy lifting: It reads the articles, finds the clues, and drafts the report.
  2. The "Fact-Checker" Step: Before the report is sent to the captain, the system automatically checks the AI's facts against trusted databases (like a digital encyclopedia of known bad guys). If the AI says, "The bad guy is Bob," the system checks: "Is Bob actually in the database?" If not, it flags it.
  3. The Human Loop: A human expert reviews the AI's draft. They don't start from scratch; they just fix the AI's mistakes and add the "human touch" (context, nuance, and intuition).

• The Result: This team approach turned the AI's "rough drafts" into "publish-ready" reports. The AI found details the human missed, and the human corrected the AI's mistakes.

The Big Takeaway

AI is not ready to replace human security analysts yet. It's too prone to making up facts and getting the "big picture" wrong.

However, AI is an incredible "Super-Assistant." If you pair it with human experts and give it tools to check its own facts, it can do 80% of the boring work, allowing humans to focus on the 20% that requires deep thinking and intuition.

In short: Don't let the AI drive the ship alone. But do let it be the first mate who scans the horizon, as long as the Captain (the human) is there to steer and verify the course.

Technical Summary

Here is a detailed technical summary of the paper "CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?"

1. Problem Statement

The paper addresses the critical gap between current Large Language Model (LLM) benchmarks and the actual workflows of cybersecurity analysts. While LLMs show promise in automating Cyber Threat Intelligence (CTI), existing evaluation methods suffer from three primary limitations:

• Unrealistic Task Formats: Current benchmarks (e.g., CTIBench, CyberBench) rely heavily on multiple-choice questions or simple fact-recall Q&A. These do not reflect the open-ended, decision-making nature of real analyst work, such as triaging articles or synthesizing complex narratives.
• Model-Centric Metrics: Evaluations often use metrics like ROUGE or BERTScore, which prioritize lexical overlap over factual accuracy, actionable insights, or operational costs (time/token usage). A sparse summary might score higher than a detailed, actionable one under these metrics.
• Incomplete Workflow Coverage: Existing benchmarks typically evaluate isolated subtasks (e.g., summarization or IoC extraction) rather than the complete, end-to-end threat research workflow, which consists of Triage, Deep Search, and TI Drafting.

2. Methodology

A. CyberThreat-Eval Benchmark

The authors introduce CyberThreat-Eval, a novel, expert-annotated benchmark derived from the daily operations of a world-leading technology company's security division. It evaluates LLMs across the three-stage CTI workflow:

1. Triage:
   • Task: Determine if an article is relevant ("accepted" vs. "rejected") and assign a priority score (1–5) based on threat severity and target impact.
   • Data: 488 articles with expert-annotated ground truth.
2. Deep Search:
   • Task: Given a seed URL, the model must retrieve supplementary URLs containing new, valuable information (e.g., novel facts, technical details) not present in the source.
   • Evaluation: An LLM-as-Judge compares retrieved content against the seed to verify "additional information."
   • Data: 55 input URLs.
3. TI Drafting:
   • Task: Synthesize information into structured reports.
   • Sub-tasks:
     • IoC Extraction: Extract concrete artifacts (IPs, hashes, domains).
     • TTP Identification: Map adversary behaviors to MITRE ATT&CK techniques.
     • Narrative Generation: Generate coherent profiles of Threat Actors and Root Cause analyses.
   • Data: 1,310 IoCs, 1,565 TTPs, and 412 articles for narrative generation.

B. Evaluation Metrics

The benchmark shifts from model-centric to analyst-centric metrics:

• Factual Accuracy: Precision/Recall for extraction tasks; Pass Rate and Bias for scoring.
• Content Quality: Evaluated via an LLM-as-Judge (calibrated against human experts with >95% agreement) on six dimensions: Relevance, Accuracy, Comprehensiveness, Clarity, Coherence, and Attribution.
• Operational Efficiency: Processing time and token consumption.

C. Threat Research Agent (TRA)

To address LLM limitations, the authors propose TRA, an end-to-end framework integrating:

• Human-in-the-Loop: Expert feedback is embedded via prompt engineering to refine drafts and validate findings.
• External Ground Truth: Integration with authoritative databases (e.g., VirusTotal) to verify technical details (IoCs, TTPs) and prevent hallucinations.
• Iterative Refinement: A cycle of retrieval, selection, evaluation, and synthesis.

3. Key Contributions

1. CyberThreat-Eval Benchmark: The first benchmark to cover the full end-to-end CTI workflow (Triage, Deep Search, Drafting) using real-world data and analyst-centric metrics.
2. Comprehensive Evaluation: A systematic assessment of four models (GPT-4o, o3-mini, and their fine-tuned variants) revealing specific strengths and weaknesses in threat research.
3. TRA Framework: A practical, deployable architecture demonstrating how to combine LLMs with external knowledge and human expertise to achieve "publish-ready" intelligence.
4. Insights on LLM Limitations: Detailed analysis showing that while LLMs are good at retrieval and root-cause explanation, they struggle with precision in triage, complex TTP reasoning, and generating comprehensive threat actor profiles without external grounding.

4. Experimental Results

General Performance Trends

• Triage: Models exhibit high recall (>0.90) but low precision (<0.40), leading to an "over-acceptance" of irrelevant articles, which increases analyst workload. Fine-tuning did not significantly improve precision.
• Deep Search: Base models (GPT-4o, o3-mini) retrieved more URLs with additional information compared to fine-tuned models, suggesting fine-tuning may make models too conservative or reliant on internal knowledge.
• TI Drafting:
  • IoC Extraction: High precision (~0.82–0.88) but variable efficiency.
  • TTP Mapping: A major bottleneck. All models showed low precision and recall (<0.35), indicating a failure in complex reasoning about adversarial intent.
  • Narrative Generation: Models performed well on Root Cause analysis (factual "how") but struggled with Threat Actor profiling (inferential "who"), often producing sparse or hallucinated content.

The Trade-off Dilemma

The results highlight a fundamental trade-off:

• Increasing Recall (finding all potential threats) often decreases Precision and increases Cost (time/tokens).
• Increasing Precision (e.g., via o3-mini) often leads to lower Recall or significantly higher resource consumption.
• Fine-tuning improves specific tasks (like actor profiling) but can degrade performance in exploratory tasks (like deep search) and does not solve fundamental reasoning gaps.

TRA Performance Improvements

Integrating the TRA framework yielded significant gains:

• IoC Precision: Increased by 26 percentage points across base models.
• TTP Precision: Improved from ~0.28 to 0.42 (o3-mini) and 0.31 (GPT-4o FT).
• Content Quality: Threat actor and root cause narratives achieved expert scores >4.5/5 on relevance and coherence, transforming sparse drafts into "publish-ready" reports.
• Efficiency: TRA reduced latency for GPT-4o by ~70 seconds in certain tasks while improving quality.

5. Significance and Conclusion

The paper demonstrates that while LLMs can automate parts of the threat research workflow, they are not yet ready for fully autonomous deployment due to issues with hallucination, reasoning depth, and precision.

• Practical Impact: The CyberThreat-Eval benchmark provides a rigorous standard for the community to develop more capable, analyst-aligned models.
• Operational Value: The TRA framework proves that a "Human-in-the-Loop" approach combined with external verification is essential for high-stakes security operations. It bridges the gap between raw LLM output and actionable intelligence, reducing analyst workload while maintaining accuracy.
• Future Direction: The authors conclude that future research must focus on enhancing multi-document reasoning, grounding assertions in verifiable evidence, and developing cost-efficient architectures that balance recall and precision without sacrificing operational viability.

The code and benchmark are released on GitHub and HuggingFace to facilitate further research in analyst-oriented CTI automation.