🤖 AI

CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence

This paper introduces CTIConnect, a comprehensive benchmark and evaluation harness designed to systematically assess retrieval-augmented LLMs across nine heterogeneous Cyber Threat Intelligence tasks, revealing that effective performance requires domain-specific structural interventions rather than generic retrieval improvements.

Original authors: Yutong Cheng, Yang Liu, Changze Li, Dawn Song, Peng Gao

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yutong Cheng, Yang Liu, Changze Li, Dawn Song, Peng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a cybersecurity detective trying to solve a complex crime. You have a massive library of clues, but they are written in five completely different languages and formats: some are rigid, technical spreadsheets (like a database of software bugs), while others are long, messy news articles written by different reporters who use slang and nicknames for the same criminals.

This is the world of Cyber Threat Intelligence (CTI). The problem is that there is too much information for any human to read and connect the dots manually.

Enter Large Language Models (LLMs). Think of these AI models as brilliant, super-fast detectives who can read and understand almost anything. But there's a catch: these AIs have a "memory limit." They can't memorize every single new threat report that comes out every day. If you ask them a question about a brand-new virus, they might guess wrong because they haven't "read" the latest report yet.

To fix this, we use Retrieval-Augmented Generation (RAG). This is like giving the AI detective a magical library card. When asked a question, the AI first runs to the library, grabs the most relevant documents, and then answers based on what it just read.

The Problem: The "Lost in Translation" Gap

The authors of this paper, CTIConnect, discovered that simply handing the AI a library card isn't enough. The library is a mess.

The Vocabulary Mismatch: One document might call a hacker group "APT29," while another calls them "Cozy Bear," and a third calls them "Nobelium." If you ask the AI to find all reports on "APT29," a standard search might miss the reports that only use the other names.
The Format Mismatch: One document might say "The attacker stole passwords from memory," while a technical database lists this as "T1003.001 – LSASS Memory." The AI might not realize these are the same thing.

The paper argues that standard search tools (which just look for similar words) fail here because they can't bridge these gaps. They are like a translator who only knows literal word-for-word translation but doesn't understand the meaning or the nicknames.

The Solution: CTIConnect

The team built a new benchmark (a standardized test) called CTIConnect to see how well AI detectives perform when they have to search through this messy, multi-language library.

They created 1,860 expert-verified questions covering three main types of detective work:

Entity Linking (Connecting the Dots):
- The Task: "This software bug (CVE) is caused by which specific weakness (CWE)?"
- The Analogy: Matching a specific car model to its engine type. The AI has to translate between two different technical catalogs.
- The Fix: The AI needs to "translate" the question into the exact technical terms the database uses before searching.
Entity Attribution (Naming the Culprit):
- The Task: "A report says 'the bad guys encrypted files with a .cuba extension.' Which specific hacking technique does this describe?"
- The Analogy: Reading a witness description ("He wore a red hat and ran fast") and matching it to a police file ("Suspect: John Doe, known for running").
- The Fix: The AI must break the sentence down into small, atomic actions and translate each one into the official police code.
Multi-Document Synthesis (Building the Story):
- The Task: "Combine reports from five different news outlets to build a profile of the hacker group 'APT29'."
- The Analogy: You have five different witnesses describing the same party. One calls the host "Bob," another "Bobby," and another "The Boss." The AI must realize they are all talking about the same person and combine their stories into one timeline.
- The Fix: The AI needs to be smart enough to realize "Bob" = "Bobby" = "The Boss" before it starts reading.

What They Found

The researchers tested 10 different AI models (both free/open-source and expensive/proprietary ones) using this new test. Here are their key discoveries:

One Size Does Not Fit All: A "one-size-fits-all" search strategy failed miserably. The best way to search for a technical bug (Entity Linking) is totally different from the best way to search for a hacker's nickname (Multi-Document Synthesis).
Specialized Tools Win: When they gave the AI specialized strategies (like "translate the question first" or "break the sentence into parts"), the AI's performance jumped dramatically—sometimes by 35%. Standard search tools only improved performance by a tiny bit (1-5%).
The Bottleneck Shifts:
- For technical linking tasks, the problem was the search tool. Once the AI found the right document, even a "smaller" AI could answer correctly.
- For naming and story-building tasks, the problem was the AI's brain. Even with the right documents, the AI needed to be very smart to understand the context and connect the aliases.
Smarter Search > Bigger Brains: For some tasks, using a "smaller" AI with a specialized search tool worked better than using a "massive, expensive" AI with a basic search tool. This means you don't always need the most expensive AI; you just need the right way to find information.

The Bottom Line

The paper concludes that to build effective AI security tools, we can't just rely on bigger AI models or generic search engines. We need to build specialized retrieval systems that understand the unique "dialects" and "nicknames" of the cybersecurity world.

Think of it this way: You don't need a bigger library to solve a crime; you need a librarian who knows exactly how to find the right book, even if the title is written in a code you don't speak. CTIConnect provides the map and the test to build that librarian.

Technical Summary: CTIConnect

Problem Statement

Cyber Threat Intelligence (CTI) is essential for proactive defense, yet the ecosystem is characterized by massive volume and extreme heterogeneity. It spans structured knowledge bases (CVE, CWE, CAPEC, MITRE ATT&CK) and unstructured vendor threat reports, which utilize fundamentally different schemas, vocabularies, and abstraction levels. While Large Language Models (LLMs) offer strong reasoning capabilities, existing benchmarks fail to evaluate them in realistic, retrieval-augmented settings. Current benchmarks like CTIBench and SEvenLLM either rely on closed-book settings (testing only parametric memory) or focus on single-report extraction, ignoring the critical need to retrieve and synthesize information across diverse, cross-referenced sources.

A specific challenge identified is the cross-source semantic gap: queries expressed in one vocabulary (e.g., analyst narrative prose) systematically fail to match relevant evidence encoded in another (e.g., formal taxonomy entries). This gap causes standard embedding-based retrieval to break down, as general-purpose Retrieval-Augmented Generation (RAG) strategies cannot bridge the lexical and structural mismatches inherent in the CTI domain.

Methodology

Benchmark Construction (CTIConnect)

The authors constructed CTIConnect, a unified evaluation environment integrating five heterogeneous CTI sources into 1,860 expert-verified QA pairs. The construction pipeline involves three stages:

Seed Correlation Annotation: Ground-truth correlations are established from authoritative sources (MITRE/NVD mappings) or through dual expert annotation for unstructured data.
Template-Constrained QA Synthesis: Correlations are transformed into task-specific questions and answers using prompt templates to ensure format consistency and reduce hallucination.
Multi-layered Quality Control: A combination of LLM-based filtering, independent verification by domain practitioners, and final adjudication by a senior CTI analyst ensures high data quality.

Task Landscape

The benchmark covers nine tasks across three categories, representing all cross-source directions in CTI analysis:

Entity Linking (Structured $\to$ Structured): Mapping entries between taxonomies (e.g., CVE to CWE, CAPEC to ATT&CK). Tasks include Root Cause Mapping (RCM), Weakness Instantiation (WIM), Attack Technique Derivation (ATD), and Exploitation Surface Discovery (ESD).
Entity Attribution (Unstructured $\to$ Structured): Grounding narrative descriptions in formal taxonomy entries. Tasks include Attack Technique Attribution (ATA) and Vulnerability Catalog Attribution (VCA).
Multi-Document Synthesis (Unstructured $\to$ Unstructured): Aggregating intelligence scattered across multiple vendor reports. Tasks include Threat Actor Profiling (TAP), Malware Lineage Analysis (MLA), and Campaign Storyline Construction (CSC).

Retrieval Strategies

The study evaluates three retrieval configurations:

Closed-Book (CB): Baseline using only parametric knowledge.
Vanilla RAG (VR): Direct embedding of raw queries without transformation.
Domain-Specific (DS) Strategies: Tailored approaches designed to address specific semantic gaps:
- Extract-then-Retrieve (EtR): For Entity Linking, the LLM extracts semantic content and canonicalizes it into keyphrases aligned with the target knowledge base before retrieval.
- Decompose-then-Retrieve (DtR): For Entity Attribution, the input is decomposed into atomic behaviors, each canonicalized and retrieved independently before aggregation.
- CSKG-Guided RAG: For Multi-Document Synthesis, a Cybersecurity Knowledge Graph (CSKG) is used to resolve entity aliases (e.g., "APT29" $\equiv$ "Cozy Bear") and rank reports based on canonical entities rather than raw text chunks.

Experimental Results

Evaluation Setup

Ten state-of-the-art LLMs (spanning open-source and proprietary families, e.g., GPT-5, LLaMA-3-405B, Claude-Sonnet-4) were evaluated on a 691-pair subset, with verification on the full 1,860-pair benchmark for representative models.

Key Findings

Performance Gains: Domain-specific strategies significantly outperform Vanilla RAG. Gains were up to +35.2% for Entity Linking, +16.0% for Entity Attribution, and +11.3% for Multi-Document Synthesis.
Task-Specific Bottlenecks:
- Entity Linking: The bottleneck is primarily retrieval infrastructure. Once the correct evidence is retrieved via canonicalization, even smaller models can perform the task with near-ceiling accuracy (average DS: 97%).
- Entity Attribution: The bottleneck shifts to model reasoning. This task requires decomposing narratives and inferring root causes; scaling model size yields significant gains (up to +29.5% from 8B to 405B models), and incorrect retrieval directly degrades precision.
- Multi-Document Synthesis: Performance is limited by both retrieval (handling aliasing) and model reasoning (synthesizing conflicting accounts).
Semantic Gap Analysis: Vanilla RAG fails due to a "cross-source semantic gap."
- Entity Linking has a small similarity gap (0.06) but suffers from candidate discrimination issues.
- Entity Attribution has a moderate gap (0.31) due to vocabulary mismatch.
- Multi-Document Synthesis has the largest gap (0.43) due to entity aliasing and chunk-level noise.
General-Purpose vs. Domain-Specific: Strong general-purpose retrieval paradigms (Retrieve-then-Rerank, IRCoT) only close 1–5% of the performance gap. This confirms that bridging the CTI semantic gap requires structural interventions (canonicalization, decomposition, entity resolution) rather than generic retrieval improvements.
Model Scaling: Scaling model size benefits Entity Attribution disproportionately (+19.2% average gain) compared to Entity Linking (+3.4%) or Synthesis (+2.7%). Smaller open-source models can match frontier proprietary models on retrieval-bottlenecked tasks if the retrieval strategy is optimized.

Significance and Claims

The paper claims that CTIConnect is the first retrieval-augmented benchmark for CTI that systematically evaluates LLMs across heterogeneous sources. Its primary contribution is demonstrating that a "one-size-fits-all" retrieval approach is insufficient for CTI. Instead, effective CTI systems require task-aware retrieval architectures that apply specific structural interventions (canonicalization, decomposition, alias resolution) to bridge the semantic gaps between diverse data sources.

The authors assert that these findings provide actionable guidance for designing scalable knowledge retrieval architectures, suggesting that for resource-constrained environments, investing in domain-specific retrieval infrastructure may yield greater returns than simply scaling up model size. The results remain stable across temporal splits (2008–2025), indicating the benchmark's relevance to continually evolving threat landscapes.