Technical Summary: HackerSignal

Problem Statement

Modern applied cybersecurity faces a central challenge in understanding how cyber threats emerge, propagate, and relate to vulnerabilities. Signals tracking this trajectory are distributed across heterogeneous text-based sources, including hacker forums, public exploit databases, official vulnerability advisories (e.g., NVD, CISA KEV), and open-source fix commits. Existing public datasets in this domain typically span only a single layer or a subset of these sources and are often proprietary. Furthermore, structured datasets linking hacker community discourse to formal vulnerability records are largely absent from public literature. Current benchmarks often lack temporal rigor, failing to evaluate how models generalize to threats that did not exist at training time, and frequently suffer from data leakage issues that inflate performance claims.

Methodology

The authors introduce HackerSignal, a large-scale, temporally structured, multi-source corpus and benchmark designed to link hacker community discourse to the full CVE vulnerability lifecycle.

Dataset Construction

• Scale and Scope: The dataset aggregates 7.45 million exact-deduplicated documents from 64 public forum and source identifiers spanning an 8-layer source hierarchy and a 36-year window (1990–2026).
• Source Layers: The corpus integrates:
  1. Hacker and security communities (6.97M rows, including darkweb marketplaces and security boards).
  2. Vulnerability references (NVD).
  3. Exploit archives (e.g., ExploitDB).
  4. Exploit Q&A references.
  5. Advisory references.
  6. Bug-bounty disclosures.
  7. Fix-commit references.
  8. Exploitation references (CISA KEV).
• Unification: All sources are normalized into a common JSONL schema unified by a shared CVE identifier namespace. A separate CVE index maps records to references via foreign keys, enabling cross-source joins.
• Preprocessing: Data collection involved official APIs, ingestion of existing public datasets, and ethical web scraping. Processing steps included Unicode normalization, HTML stripping, URL replacement, whitespace collapsing, and filtering for non-English content and short posts (<20 chars). Exact deduplication (SHA-256) and near-deduplication (MinHash-LSH) were applied.
• Temporal Splits: To ensure prospective generalization, the data is split temporally:
  • Task 1 & 3: Train (<2022), Validation (2022–2023), Test (2024+).
  • Task 2: Train (<2020), Validation (2020–2022), Test (≥2023).

Benchmark Tasks

The paper defines three benchmark tasks to evaluate temporal out-of-distribution (OOD) generalization:

1. CVE Linkage Retrieval (CVE-R): A cross-source entity grounding task where the model retrieves the correct CVE description from the NVD corpus (340K entries) given an exploit/advisory text query.
   • Baselines: BM25, dense bi-encoders (MiniLM, mpnet, BGE, E5), and a hybrid sparse+dense approach.
2. Exploit Type Classification (ETC): An 8-class classification task predicting vulnerability types (e.g., Injection, XSS, Memory Corruption, RCE) from exploit text.
   • Baselines: Bag-of-words classifiers (Decision Tree, TF-IDF+LR, SVM), RNNs (RNN, GRU, LSTM, BiLSTM), and a domain-specific transformer (SecBERT).
3. Temporal Generalization (TG): A strict retrieval task identical to Task 1 but with a CVE-disjoint constraint ($C_{train} \cap C_{test} = \emptyset$). Models must generalize to entirely unseen vulnerability descriptions, preventing memorization of specific CVE entries.

Evaluation Protocols

• Leakage Audits: The authors explicitly audit for temporal and source leakage, verifying CVE disjointness programmatically.
• Baseline Ladder: Following BEIR and WILDS methodologies, the evaluation employs a heterogeneous ladder of models ranging from lexical (BM25) to dense and hybrid architectures.
• Metrics: Retrieval tasks use Recall@k and Mean Reciprocal Rank (MRR); classification uses Macro-F1 and per-class F1.

Key Results

• CVE Linkage Retrieval (Task 1): The E5-base-v2 model achieved the highest performance with an MRR of 0.587, outperforming the hybrid BM25+mpnet approach (0.567) and other symmetric bi-encoders. This suggests that asymmetric query/passage prefix training is particularly effective for cross-source CVE grounding.
• Exploit Type Classification (Task 2): The BiLSTM model achieved the best Macro-F1 of 0.871, outperforming the best bag-of-words baseline (SVM at 0.799) and the domain-specific SecBERT (0.846). This indicates that bidirectional context is crucial for exploit text, while domain-specific pre-training offered only modest benefits over standard architectures.
• Temporal Generalization (Task 3): Under strict CVE-disjoint conditions, E5-base-v2 again led with an MRR of 0.602. Models generally maintained or improved performance compared to Task 1, confirming that the temporal splits effectively prevented meaningful memorization of training data.

Key Contributions

1. A Temporally Grounded CTI Corpus: A 7.45M-document dataset spanning eight source layers with CVE linkage metadata, supporting diverse NLP tasks beyond the three benchmarks (e.g., exploit code summarization, topic modeling).
2. Three Benchmark Tasks: A suite of tasks with temporal train/val/test splits designed to test progressively harder generalization, including cross-source retrieval, 8-class classification, and prospective CVE-disjoint evaluation.
3. Governance and Transparency Artifacts: A full model ladder, source-shortcut diagnostics, manual-audit packets, a datasheet, and a release-governance addendum to ensure integrity and prevent data leakage.

Significance and Claims

The paper positions HackerSignal as a foundational resource for temporal out-of-distribution cyber threat intelligence (CTI). Its primary significance lies in:

• Bridging the Gap: It is among the first public benchmarks to map the full trajectory from hacker community discourse to formal vulnerability records and software fixes.
• Rigorous Evaluation: By enforcing temporal splits and CVE-disjoint constraints, it addresses the common issue of inflated benchmark claims caused by data leakage and non-prospective evaluation.
• Defensive Research Focus: The dataset is explicitly curated for defensive research, with governance rules prohibiting the fine-tuning of automated exploit models or de-anonymization of users.

The authors acknowledge limitations, noting that CVE linkages are metadata-derived rather than expert-adjudicated (introducing noise), that cross-lingual retrieval was not evaluated, and that large language model rerankers were not included due to compute costs. However, the release of the code, data, and Croissant metadata aims to support future extensions in exploit code summarization, community interaction analysis, and cross-lingual CTI extraction.