Architectural Selection Framework for Synthetic Network Traffic: Quantifying the Fidelity-Utility Trade-off

1. Problem Statement

The generation of synthetic network traffic is critical for cybersecurity research (e.g., training Intrusion Detection Systems) due to privacy restrictions and the scarcity of labeled real-world data. However, existing methods face three core challenges that are often treated independently:

1. Fidelity: The inability to replicate complex statistical and temporal dependencies of real traffic.
2. Utility: The failure to produce diverse data variations effective for downstream AI training (avoiding mode collapse).
3. Scalability: The prohibitive computational costs of state-of-the-art models when applied to large-scale datasets.

The paper argues that these failures stem from a fundamental architectural mismatch: generative models are often selected without considering the intrinsic structure of the target dataset (e.g., categorical-heavy vs. continuous-flow-heavy data). There is a lack of an evidence-based framework to guide model selection based on data structure.

2. Methodology

The study establishes a rigorous Architectural Selection Framework to empirically evaluate how data structure dictates the optimal trade-off between fidelity and utility.

• Datasets: Two distinct network traffic datasets were selected to represent heterogeneous structures:
  • NSL-KDD: Represents older benchmarks dominated by categorical/discrete features (e.g., protocols, flags).
  • CIC-IDS2017: Represents modern traffic dominated by continuous, high-dimensional flow metrics (e.g., bytes/sec, packet variance).
• Feature Selection: Mutual Information (MI) was used to select the top 25% of features, ensuring the selection captured non-linear dependencies rather than just linear correlations.
• Models Evaluated: Twelve generative architectures were tested across three categories:
  • Non-AI/Statistical: ROS, SMOTE, ADASYN, Cluster Centroids, GMM, Bayesian Networks.
  • Deep Generative (VAE/Diffusion): Tabular VAE (TVAE), Tabular Diffusion (TABDDPM).
  • Generative Adversarial Networks (GANs): CTGAN, CopulaGAN, GANBLR++, CasTGAN.
• Evaluation Metrics:
  • Fidelity: Assessed via three structural metrics:
    1. Data Structure (DS): Binary check for logical constraints (e.g., binary columns must remain 0/1).
    2. Correlation (Corr): Comparison of correlation heatmaps between real and synthetic data.
    3. Probability Distribution (PD): Quantification of distribution differences using Kernel Density Estimation (KDE).
  • Utility: Measured via TSTR (Train on Synthetic, Test on Real) performance against a TRTR (Train on Real, Test on Real) baseline. Metrics included Accuracy and F1-Score.
  • Scalability: Assessed via training time, memory usage, and failure rates on large datasets.
• Experimental Rigor: All experiments were run 20 times (N=20) to calculate mean and standard deviation, ensuring statistical significance via paired t-tests.

3. Key Contributions

1. Evidence-Based Framework: Proposes a framework linking generative model architecture directly to dataset structure, moving beyond generic benchmarking.
2. Identification of Optimal Architectures: Empirically demonstrates that GAN-based models (specifically CTGAN and CopulaGAN) offer the superior balance of fidelity and utility across both categorical and continuous data types.
3. Quantification of Failure Modes:
   • Reveals that statistical methods (e.g., SMOTE) achieve high utility but suffer from compromised fidelity (failing structural checks like Data Structure and Correlation).
   • Demonstrates that Diffusion Models (TABDDPM) and Bayesian Networks face prohibitive computational barriers on large-scale continuous data, rendering them impractical for current security deployments.
4. Unified Benchmark: Provides a comprehensive comparison of 12 methods with a reproducible baseline, including the implementation of "Fidelity Gatekeeper Metrics."

4. Key Results

• GAN Dominance: CTGAN and CopulaGAN consistently occupied the "optimal performance frontier" (high utility, high fidelity, low variance).
  • On CIC-IDS2017, CopulaGAN achieved an F1-score of 0.9756 ± 0.0005 with a 0% Probability Distribution difference, proving its ability to model continuous flows perfectly.
  • These models successfully handled mixed data types (discrete and continuous) without structural collapse.
• Statistical Methods (The "Compromised Fidelity" Trap): Methods like SMOTE and ROS achieved near-perfect utility (F1 ≈ 0.999) but failed structural integrity checks (Data Structure = "No"). They generate interpolated samples that are easy to classify but lack the generative realism required for robust security training.
• Scalability Failures:
  • Diffusion Models (TABDDPM) and Bayesian Networks failed to execute on the large CIC-IDS2017 dataset due to memory and computational constraints.
  • GANBLR++ exhibited extreme instability (high standard deviation), making it unreliable for production.
• Dataset Influence: Models that performed well on categorical data (NSL-KDD) often failed or degraded significantly on continuous data (CIC-IDS2017), except for the robust GAN architectures.

5. Significance and Implications

• Practical Guidance: The paper provides security practitioners with a clear decision matrix: use CTGAN or CopulaGAN for synthetic traffic generation in intrusion detection systems. It explicitly advises against using Diffusion models for large-scale tabular network data due to current scalability limits.
• Redefining Utility: The study highlights that high classification accuracy (Utility) does not guarantee data quality (Fidelity). Relying on statistical oversampling (SMOTE) may lead to models that overfit to synthetic artifacts rather than learning real attack patterns.
• Future Directions: The authors suggest future work should focus on integrating Differential Privacy into GANs, developing conditional dependency metrics (beyond simple correlation), and optimizing Diffusion models for discrete/tabular data to overcome current computational barriers.

In conclusion, this study establishes that architectural compatibility with data structure is the primary determinant of synthetic data success. It validates GANs as the current gold standard for scalable, high-fidelity synthetic network traffic generation in cybersecurity.