A Deterministic Forensic Preprocessing Framework for Heterogeneous Network Datasets: Formal Foundations, Implementation, and Empirical Validation

Technical Summary: A Deterministic Forensic Preprocessing Framework for Heterogeneous Network Datasets

Problem Statement
Digital forensic investigations increasingly rely on correlating evidence from heterogeneous sources, including enterprise traffic logs, IoT device telemetry, and Intrusion Detection System (IDS) alerts. A critical bottleneck exists in the preprocessing phase: existing approaches are predominantly ad hoc, relying on custom scripts that lack mechanisms to verify consistency across runs. This creates reproducibility gaps that undermine evidence integrity and challenge legal admissibility. Furthermore, current tools (e.g., Wireshark, Zeek) are designed for real-time capture rather than post-hoc integration, while general-purpose Extract, Transform, Load (ETL) systems lack formal determinism guarantees, cryptographic provenance, and the ability to handle datasets larger than available memory. The core problem is the absence of a preprocessing pipeline that is simultaneously formally correct, reproducible, cryptographically verifiable, and scalable.

Methodology
The authors propose a deterministic forensic preprocessing framework that transforms heterogeneous network datasets into a reproducible canonical form. The methodology is grounded in set-theoretic definitions and implemented via a chunk-based architecture.

1. Formal Foundations: The framework defines three core transformations:

   • Schema Normalization ($T_s$): Maps heterogeneous field names to a canonical schema while preserving original field values to ensure information preservation.
   • Temporal Normalization ($T_t$): Converts diverse timestamp formats (Unix epoch, ISO 8601, etc.) into a canonical ISO 8601 UTC format. Crucially, it preserves raw timestamps and handles ambiguous cases (e.g., Daylight Saving Time transitions) deterministically without discarding data.
   • Provenance Tracking ($T_p$): Generates cryptographic hashes (SHA-256) for data chunks and creates provenance records documenting the chain of custody, including chunk indices, record counts, and timestamps.

2. Theoretical Guarantees: The paper establishes four theorems proving:

   • Determinism: The transformations produce identical outputs for identical inputs regardless of execution context.
   • Information Preservation: No data is lost during schema or temporal normalization; original values remain recoverable.
   • Provenance Completeness: Every record is covered by a provenance record, ensuring full auditability.

3. Implementation Architecture:

   • Chunk-Based Processing: To handle datasets exceeding system memory, the framework processes data in fixed-size chunks ($c$). This ensures $O(c)$ bounded memory complexity, independent of the total dataset size ($n$).
   • Deterministic Ordering: The system enforces deterministic behavior by processing records in file order, sorting field names alphabetically during serialization, and using fixed parsing orders for timestamps.
   • Verification: The implementation includes a multi-run verification mechanism that compares SHA-256 hashes of the final datasets and provenance chains across independent executions.

Key Contributions
The paper claims six primary contributions:

1. Formal Preprocessing Framework: A set-theoretic formalization of forensic preprocessing as a deterministic transformation problem with explicit definitions for $T_s$, $T_t$, and $T_p$.
2. Theorem-Backed Guarantees: Four formal theorems (with proofs) establishing determinism, information preservation, and provenance completeness.
3. Provenance-Aware Preprocessing: Integration of cryptographic provenance tracking (SHA-256) directly into the preprocessing transformation level, supporting chain-of-custody requirements.
4. Bounded-Memory Architecture: A chunk-based design enabling the processing of datasets substantially larger than available RAM on standard hardware.
5. Reproducibility Verification: A methodology for multi-run SHA-256 equivalence testing to empirically validate formal guarantees.
6. Empirical Validation: Evaluation across three public datasets (UNSW-NB15, IoT-23, TON_IoT) demonstrating scalability and consistency.

Experimental Results
The framework was evaluated on three datasets ranging from 2.5 million to 325 million records:

• Determinism: Achieved 100% output consistency (both dataset and provenance hashes) across five independent runs for all datasets.
• Temporal Normalization: Achieved 99.99% to 100% completeness. Malformed timestamps were preserved as raw values with canonical fields marked as undefined, ensuring no evidence loss.
• Provenance Coverage: 100% of records were covered by provenance records with cryptographic verification.
• Scalability and Memory: The framework processed 325 million records (IoT-23) with a peak memory usage of 2.3 GB, confirming $O(c)$ bounded memory. In contrast, a Pandas-based baseline failed on large datasets due to memory constraints and lacked determinism verification.
• Performance: Throughput ranged from 20,000 to 66,850 records per second depending on the dataset complexity.

Significance and Claims
The paper positions this work as the first system to combine formal preprocessing guarantees, determinism proofs, cryptographic verification, bounded memory, and large-scale empirical validation. The authors argue that this framework addresses the "reproducibility gaps" in current forensic workflows, providing a legally defensible basis for evidence admissibility. By formalizing preprocessing as a deterministic transformation, the work enables analysts to scientifically demonstrate that evidence has not been altered or inconsistently processed. The authors note that while the framework provides a robust foundation, future work is required for automated schema inference, distributed processing, and machine-checked formal verification (e.g., in Coq).