Technical Summary: Application of Tabular Transformer Architectures for Operating System Fingerprinting

Problem Statement

Operating System (OS) fingerprinting is critical for network management, inventory, and cybersecurity, yet traditional rule-based tools (e.g., Nmap, p0f) struggle in modern, dynamic environments. Frequent OS updates, obfuscation techniques, and the heterogeneity of connected devices degrade the accuracy of static rule-based signatures. While Machine Learning (ML) has been applied to this domain, existing research largely relies on classical algorithms (e.g., Decision Trees, Random Forests, k-Nearest Neighbors). Deep Learning (DL) approaches, particularly Transformer architectures, remain largely unexploited for OS fingerprinting despite their proven success in Natural Language Processing (NLP) and Computer Vision. The core challenge addressed is improving the accuracy and robustness of OS identification in complex, evolving network conditions by leveraging advanced DL architectures designed for structured tabular data.

Methodology

The study investigates the application of Tabular Transformer architectures to OS fingerprinting, treating network flow data as structured tabular inputs.

Datasets

Three publicly available datasets were utilized to ensure diversity in data types, feature distributions, and classification granularity:

1. DAT1 (lastovicka_2019): IPFIX flow records from a university backbone (2019), focusing on OS family classification (5 classes) using TLS and flow features.
2. DAT2 (lastovicka_2023): IPFIX web traffic data (2023) supporting classification at family (12 classes), major (50 classes), and minor (88 classes) levels.
3. DAT3 (nmap-7.94_2023): Active signatures collected via Nmap probes, containing 263 features for family (7 classes) and minor (91 classes) classification.

Data Preparation

• Preprocessing: Handling of missing values (NaN/Null), removal of irrelevant features (e.g., timestamps), and deduplication.
• Feature Engineering: Categorical features (e.g., protocol types) and numerical features (e.g., TTL, window size) were standardized. Hexadecimal strings were split into bytes.
• Class Imbalance: Addressed via random undersampling for majority classes and Synthetic Minority Over-sampling Technique (SMOTE) for minority classes.
• Encoding: One-Hot Encoding applied to categorical targets; class weights computed as the inverse of normalized frequencies.

Models

The study compared two specific Tabular Transformer architectures against three classical ML baselines:

• Baselines: k-Nearest Neighbors (kNN), Random Forest (RF), and Multi-Layer Perceptron (MLP).
• TabTransformer (TabT): Replaces traditional categorical embeddings with context-aware representations using Transformer layers, applying self-attention primarily to categorical features.
• FT-Transformer (FT-T): Generalizes the self-attention mechanism to both categorical and numerical features, treating them uniformly to capture interactions across heterogeneous data.

Experimental Setup

• Hyperparameter Optimization: Conducted using Optuna with NSGA-II sampler and Stratified 10-Fold Cross-Validation.
• Training: Models were trained on the FinisTerrae III supercomputer (CESGA) using CUDA-enabled GPUs. Early stopping and AdamW optimization were employed.
• Evaluation: Metrics included Accuracy, Balanced Accuracy, Precision, Recall, and F1-score (weighted average to handle class imbalance). Confusion matrices were generated for detailed error analysis.

Key Results

The experiments demonstrated that Transformer-based models generally outperform traditional ML methods, with FT-Transformer emerging as the most robust architecture overall.

• DAT1 (Family Level): FT-Transformer achieved the highest F1-score (90.80%), outperforming TabTransformer (89.46%), MLP (89.45%), and RF (87.93%).
• DAT2 (Family, Major, Minor):
  • Family: FT-Transformer (95.09%) slightly outperformed RF (95.33% accuracy, but lower F1) and significantly outperformed other baselines.
  • Major: FT-Transformer achieved the best performance (F1: 79.32%), surpassing RF (75.64%) and TabTransformer (74.99%).
  • Minor: FT-Transformer (F1: 69.76%) and TabTransformer (F1: 68.28%) outperformed classical baselines, though the task remained challenging.
• DAT3 (Family, Minor):
  • Family: FT-Transformer led with an F1-score of 92.23%.
  • Minor: TabTransformer achieved the best result (F1: 75.90%), slightly outperforming FT-Transformer (75.74%) and the RF baseline (75.66%).

Comparison with Prior Work

The proposed methods set new benchmarks when compared to previous studies on the same datasets:

• On DAT1, FT-Transformer improved the F1-score by approximately 12% over the previous state-of-the-art (Decision Tree).
• On DAT2, FT-Transformer achieved an F1-score of 95.09% for family classification, substantially outperforming the best previous model (Decision Tree at 81.60%). It also provided the first reported results for the "major" classification level on this dataset.
• On DAT3, FT-Transformer outperformed the MLP baseline in family classification, while TabTransformer improved upon the RF baseline for minor classification.

Significance and Claims

The paper claims three primary contributions:

1. First Application of Transformers to OS Fingerprinting: This study introduces attention-based Deep Learning models (specifically Tabular Transformers) to the domain of OS identification, marking a shift from classical ML and rule-based systems.
2. Comprehensive Evaluation: The research provides a rigorous assessment across three diverse datasets and multiple classification granularities (family, major, minor), establishing a robust baseline for future research.
3. Reproducibility: The authors emphasize transparency by releasing the full experimental code, preprocessing pipelines, and model implementations under a GNU GPL v3.0 license.

The authors conclude that the self-attention mechanism allows these models to learn intricate feature dependencies within heterogeneous network traffic, leading to improved adaptability and accuracy. They posit that FT-Transformer, in particular, offers a significant advantage in capturing complex interactions between numerical and categorical features, establishing a new benchmark for DL-based OS fingerprinting.

Future Directions

The paper outlines modest future work, including:

• Expanding the evaluation to additional real-world datasets to validate generalizability.
• Investigating hybrid approaches, such as combining Transformers with Graph Neural Networks (GNNs) to capture temporal-spatial relationships.
• Evaluating performance on newly released or modified OS versions against traditional tools like Nmap.
• Exploring transfer learning by pre-training on larger, related datasets.
• Integrating the developed models into practical cybersecurity tools (e.g., Nmap, Intrusion Detection Systems) to enhance operational effectiveness.