SynthChain: A Synthetic Benchmark and Forensic Analysis of Advanced and Stealthy Software Supply Chain Attacks

1. Problem Statement

Advanced Software Supply Chain (SSC) attacks are increasingly sophisticated, operating primarily at runtime and leaving fragmented evidence across heterogeneous telemetry sources (hosts, networks, build pipelines, and containers).

• The Gap: Existing research and benchmarks focus heavily on static artifact inspection (e.g., scanning malicious packages) or limited sandboxed dynamic analysis. They fail to capture the multi-stage, cross-environment nature of modern SSC intrusions.
• The Challenge: In real-world deployments, defenders face strict observability constraints (budget, access, retention). Evidence is often non-redundant; a step visible in process logs may be absent in network traces. Consequently, single-source telemetry is inherently insufficient for reconstructing full compromise chains, yet there is no public dataset that allows researchers to evaluate detection systems under these realistic, constrained, multi-source conditions.

2. Methodology: The SynthChain Framework

The authors constructed SynthChain, a near-production testbed and a multi-source runtime dataset designed to bridge the gap between static analysis and full-chain forensic reconstruction.

A. Testbed and Data Collection

• Environment: A hybrid setup involving Windows and Linux hosts, Docker containers, and cloud-integrated CI/CD pipelines.
• Telemetry Sources: Synchronized collection of:
  • Host-level: Process lineage, system/audit logs (Sysmon, Syslog, Windows Event Logs).
  • Network-level: Packet captures and flow data (Zeek, Suricata).
  • Container-level: eBPF-based instrumentation (Tracee) for kernel-level visibility.
  • C2 Orchestration: Use of the Mythic C2 framework to generate ground-truth tasking logs.
• Benign Noise: The testbed injects realistic, randomized benign activity (web browsing, dev workflows, updates) to simulate operational noise and prevent trivial signature-based detection.

B. Scenario Design

Seven representative attack scenarios were instantiated, covering diverse ecosystems (PyPI, npm, C/C++) and platforms (Windows, Linux):

1. Stegano: Typosquatting with steganographic payload delivery.
2. Starter: Typosquatting with persistence via startup folders.
3. Parallel: Multi-stage script chains in npm.
4. NPMEX: Sequential dependency chain attacks.
5. 3CX: Trojanized installer with DLL side-loading (inspired by the real 3CX incident).
6. CloudEX: CI/CD pipeline compromise via leaked credentials.
7. LayerInj: ML model backdoor injection.

C. Ground Truth and Annotation

• Chain-Level Ground Truth: Derived by combining C2 tasking logs (Mythic) with payload actions extracted via an LLM-assisted pipeline and human verification.
• Mapping: All adversarial actions are mapped to MITRE ATT&CK tactics and techniques (covering 14 tactics and 161 techniques across the scenarios).
• Alignment: Defender-visible events are aligned to these ground-truth actions to enable quantitative analysis of detectability.

3. Key Contributions

1. First Public End-to-End SSC Dataset: SynthChain is the first dataset to jointly capture end-to-end execution traces, multi-stage post-compromise activity, and synchronized multi-source telemetry with explicit cross-source alignment.
2. Observability Constraint Quantification: The authors provide a rigorous methodology to measure how much of an attack chain can be reconstructed given specific "source budgets" (e.g., single-source vs. multi-source fusion).
3. Realistic Attack Scenarios: The scenarios are grounded in real-world incidents (e.g., 3CX, Lazarus group tactics) and cover emerging vectors like AI/ML model poisoning and cloud control-plane abuse.
4. Open Release: The dataset, ground truth, and analysis artifacts are released to support reproducible, forensic-aware runtime defenses.

4. Key Results and Findings

The paper evaluates detection performance under varying telemetry constraints, yielding three critical insights:

A. Single-Source Monitoring is Inherently Insufficient

• Performance: No single telemetry source (e.g., just host logs or just network traffic) can reconstruct a complete compromise chain.
• Metrics: The best single source achieves only 0.391 weighted tag/step coverage and 0.403 mean chain reconstruction.
• Failure Modes: Single sources suffer from "evidence incompleteness" (missing phases like download or exfiltration) and "semantic ambiguity" (inability to link network events to specific processes).

B. Multi-Source Fusion is Essential but Not Monotonic

• The Gain: Even a minimal two-source fusion (e.g., host audit + network traffic) significantly boosts performance, raising coverage to 0.636 and reconstruction to 0.639 (a ~1.6x gain).
• Diminishing Returns: Adding more sources does not linearly improve results. If additional sources do not provide complementary join keys (e.g., process-to-network attribution) or fail to expose missing phases, they introduce noise without improving chain reconstruction.
• Optimal Strategy: Effective detection depends on selecting complementary sources rather than maximizing collection volume.

C. Structural Observability Gaps

• Some attack phases (e.g., cloud control-plane actions or fileless in-memory execution) may be structurally unobservable if the specific telemetry schema lacks the necessary fields (e.g., IAM audit logs or egress proxy data). In these cases, adding more standard host logs cannot recover the missing chain steps.

5. Significance and Impact

• Paradigm Shift: The paper challenges the assumption that "more logs equal better security." It demonstrates that correlation quality (joining heterogeneous data) is more critical than data volume.
• Benchmarking Standard: SynthChain provides a standardized benchmark for evaluating graph-based provenance systems, multi-source correlation algorithms, and runtime detection tools under realistic constraints.
• Defensive Guidance: It offers actionable guidance for defenders:
  • Prioritize telemetry that provides stable join keys (e.g., process-to-socket linkage).
  • Target specific missing-phase gaps (e.g., adding cloud audit logs for CI/CD attacks) rather than indiscriminately logging everything.
  • Recognize that single-stream detectors will inevitably miss parts of advanced SSC chains.

In conclusion, SynthChain establishes that reconstructing advanced supply chain attacks requires a multi-source, cross-layer approach grounded in realistic observability constraints, moving the field beyond static analysis toward dynamic, forensic-aware runtime defense.