Contrastive Learning and Correlation Clustering for Sequences of Network Telescope Data

Technical Summary: Contrastive Learning and Correlation Clustering for Sequences of Network Telescope Data

Problem Statement
The paper addresses the challenge of analyzing Internet scanner activities within the vast, heterogeneous, and largely unlabeled data generated by network telescopes. Traditional approaches to network scan detection often rely on manually designed statistical models, heuristics, or supervised learning, all of which face significant limitations. Supervised methods are hindered by the scarcity of high-quality annotations and the dynamic nature of scanning behavior, which varies even within the same organizational context. Furthermore, existing unsupervised representation learning methods (e.g., Word2Vec-based approaches like IP2Vec or DarkVec) often tie embeddings to specific training data, limiting their ability to generalize to unseen sources or to capture semantic relationships without synthetic data augmentation. The core question investigated is whether semantically meaningful pairwise relationships between sequences of network flow records can be estimated using contrastive learning in a fully unsupervised setting, without pretraining or annotations.

Methodology
The authors propose a framework combining a transformer-based architecture with contrastive learning and correlation clustering:

1. Data Preprocessing: Network flow records (source IP, destination IP, port, protocol, timestamp) collected from the UCSD network telescope are grouped by source IP. These are sorted chronologically and segmented into fixed-length subsequences of 128 records.
2. Transformer Encoder Model:
   • Record Embedding: Individual records are embedded by separately processing their components. IP address octets are embedded into $\mathbb{R}^8$ to capture IP space structure. Destination ports, protocols, and log-scaled time differences are embedded into $\mathbb{R}^{16}$, $\mathbb{R}^4$, and $\mathbb{R}^{16}$, respectively. These are concatenated and mapped via a linear layer to a 128-dimensional vector.
   • Sequence Encoding: A standard transformer encoder (4 layers, 4 attention heads) processes the sequence of record embeddings to capture dependencies. Mean pooling is applied to generate a fixed-size sequence representation, which is further processed by a projection head to produce the final sequence embedding.
3. Contrastive Learning Objective:
   • The model is trained using the NT-Xent loss (Normalized Temperature-scaled Cross-Entropy).
   • Positive Pairs: Defined as different subsequences originating from the same source IP address. This leverages naturally occurring consistency rather than synthetic data augmentation.
   • Negative Pairs: Defined as subsequences from different source IP addresses.
   • Sampling Strategy: To handle the long-tail distribution of source IP activity, a weighted sampling strategy is employed where the probability of selecting a source is proportional to the number of records raised to a power ($\alpha=0.5$), balancing the influence of frequent and infrequent sources.
4. Correlation Clustering:
   • To group sequences based on learned similarities, the authors formulate a correlation clustering problem. Unlike metric-based clustering (e.g., k-means), this does not require prior assumptions about the number or size of clusters.
   • A cost function is defined based on the cosine similarity of embeddings, shifted by an offset parameter $\delta$ to distinguish similar from dissimilar pairs.
   • The problem is solved locally using a greedy additive edge contraction algorithm.

Key Results
Experiments were conducted on datasets derived from the UCSD telescope, including training data, unseen sequences from seen sources, and unseen sequences from unseen sources.

• Source Discrimination: The model successfully learns source-specific information. Sequences from the same source IP exhibit significantly higher cosine similarity (mean $\approx$ 0.93 for seen sources, $\approx$ 0.89 for unseen sources) compared to sequences from different sources (mean $\approx$ 0.03 and $\approx$ 0.02, respectively). This holds true for sequences from sources not seen during training.
• Semantic Consistency with Annotations: When evaluated against sparse scanner network annotations (e.g., Censys, Shodan), sequences from the same scanner network show higher intra-class similarity than inter-class similarity. However, the separation is less pronounced than for source identity, and the intra-class distribution is bimodal. The authors attribute this to scanner networks not being perfectly homogeneous groups and different infrastructures potentially employing similar scanning strategies.
• Clustering Performance: Correlation clustering applied to the learned embeddings achieves high performance. On test sets with 50% unannotated background traffic, the weighted F1-scores for recovering scanner network groups were 0.913 (unseen sequences from seen sources) and 0.917 (unseen sequences from unseen sources).
• Generalization: The learned representations generalize effectively to unseen sources and unseen sequences, demonstrating that the model captures structural properties of scanner behavior rather than merely memorizing specific IP addresses.

Significance and Claims
The paper claims that contrastive learning can extract semantically meaningful structure from minimally preprocessed network flow sequences without any supervision. The primary significance lies in the ability to:

1. Recover source-specific information and scanner-related structure purely from raw data.
2. Perform exploratory analysis and clustering of scanner activity even in the presence of substantial unlabeled background traffic.
3. Generalize to previously unseen sources and behaviors, addressing a key limitation of supervised approaches.

The authors maintain a modest tone, noting that the scanner annotations used are sparse and coarse-grained. They acknowledge that the overlap between intra-class and inter-class similarity distributions suggests the embedding space contains structure not fully captured by the available annotations. Consequently, the work is presented as exploratory, establishing the feasibility of unsupervised representation learning for network traffic analysis while highlighting the need for further research to fully understand the semantic structure within the embedding space.