PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems

1. Problem Statement

The paper addresses critical barriers hindering the progress and reproducibility of Provenance-based Intrusion Detection Systems (PIDSs). While PIDSs (which use machine learning on system provenance graphs) show promise in detecting Advanced Persistent Threats (APTs), the field suffers from:

• Inconsistent Evaluation: Prior work uses non-standard preprocessing, dataset splits, ground-truth labeling (e.g., labeling entire neighborhoods vs. specific nodes), and metrics. This prevents fair "apples-to-apples" comparisons.
• High Re-implementation Costs: Researchers often build systems from scratch, duplicating effort in parsing data, constructing graphs, and writing evaluation scripts. Small implementation differences can skew results.
• Methodological Gaps: Many studies report single-run results despite neural network instability, rely on manual thresholds (introducing data snooping), and lack rigorous hyperparameter tuning or ablation studies.
• Reproducibility Crisis: A recent study noted that many published PIDS results cannot be reproduced due to missing code, data, or implementation details.

2. Methodology: The PIDSMAKER Framework

The authors present PIDSMAKER, an open-source, modular framework designed to standardize the development and evaluation of PIDSs. It integrates eight state-of-the-art systems into a unified architecture.

Core Architecture

The framework models the detection pipeline as seven modular stages, allowing components to be mixed and matched:

1. Construction: Parses raw provenance logs to build graphs and extract attributes (e.g., process paths, IPs).
2. Transformation: Applies graph transformations (e.g., converting to DAGs, removing redundant edges, creating pseudo-graphs).
3. Featurization: Converts attributes to numerical representations using text embeddings (Word2Vec, FastText, Doc2Vec) or domain-specific encodings (Hierarchical Feature Hashing).
4. Batching: Partitions large graphs into temporal subgraphs (Global, Intra-graph, or Inter-graph batching) to fit memory constraints.
5. Training: Trains an encoder (typically a Graph Neural Network like GAT, GraphSAGE, or TGN) and a decoder using self-supervised objectives (e.g., edge/node type prediction, reconstruction) on benign data.
6. Evaluation: Computes anomaly scores from prediction errors, applies thresholds (fixed, validation-based, or clustering), and calculates metrics (Precision, Recall, F1, AUC-ROC) at node, edge, or window levels.
7. Triage: Prioritizes alerts and identifies root causes (e.g., using DepImpact).

Key Technical Features

• YAML-Based Configuration: Systems are defined entirely via YAML files. Researchers can create new variants by inheriting from base systems and overriding specific parameters without writing code.
• On-Disk Caching: To ensure efficiency, the framework hashes stage arguments and caches outputs. If a stage's configuration hasn't changed, it skips execution and loads cached results, enabling rapid iteration.
• Standardized Data: It includes preprocessed versions of standard datasets (DARPA TC E3/E5 and OpTC) with consistent ground-truth labels (focusing on node-level detection following Jiang et al.).
• Experimental Utilities: Built-in support for:
  • Hyperparameter Tuning: Grid search and sweep capabilities integrated with Weights & Biases (W&B).
  • Ablation Studies: Systematic isolation of component contributions.
  • Instability Measurement: Running multiple iterations to quantify run-to-run variance (mean, standard deviation, relative std dev).
  • Visualization: Real-time monitoring of metrics and anomaly score distributions.

3. Key Contributions

1. Unified Architecture: Consolidates eight diverse PIDSs (THREATRACE, NODLINK, MAGIC, KAIROS, FLASH, R-CAID, ORTHRUS, VELOX) into a single codebase, enabling fair comparison.
2. Standardized Protocols: Provides consistent preprocessing pipelines and ground-truth labels, eliminating major sources of evaluation inconsistency.
3. Config-Driven Prototyping: Enables rapid design-space exploration by allowing users to compose new systems by mixing components from different existing systems via YAML.
4. Methodological Rigor: Addresses gaps in prior work by integrating tools for ablation studies, hyperparameter tuning, and multi-run instability measurement.
5. Open-Source Release: Releases the framework, preprocessed datasets, and labels to serve as a common evaluation platform for the community.

4. Results and Demonstrations

The paper demonstrates PIDSMAKER's capabilities through several use cases:

• Running Existing Systems: Successfully executes the eight integrated systems on DARPA datasets.
• Rapid Prototyping: Shows how to create a custom variant (e.g., changing an encoder from GraphSAGE to GAT or featurization from Word2Vec to FastText) in minutes via configuration files.
• Hyperparameter Tuning: Performs grid searches over learning rates and hidden dimensions, logging results to W&B to identify optimal settings.
• Instability Analysis: Quantifies the variance in performance across multiple runs, highlighting that single-run results can be unrepresentative for certain architectures.
• Visualization: Generates plots showing the separation of benign vs. attack anomaly scores and rank-based discrimination plots.

Note: The paper focuses on the framework's capability to facilitate these analyses rather than presenting a single "best" new detection algorithm. The value lies in the ability to rigorously compare existing methods.

5. Significance

• Reproducibility: PIDSMAKER directly tackles the reproducibility crisis in PIDS research by providing a shared codebase and standardized data, ensuring that reported results are comparable.
• Efficiency: It drastically reduces the engineering overhead for researchers, allowing them to focus on algorithmic innovation rather than data parsing and pipeline construction.
• Scientific Rigor: By enforcing consistent evaluation protocols (tuning, ablation, multi-run testing), it promotes more reliable and trustworthy security research.
• Community Standard: The authors aim for PIDSMAKER to become the de facto standard for evaluating provenance-based security, fostering collaboration and accelerating progress in detecting APTs.

The framework is implemented in Python using PyTorch and PyTorch Geometric, with a recommendation for high-memory servers (100GB+) and GPUs (20GB+) to handle large-scale provenance graphs.