Technical Summary: A Study of Kernel Telemetry Options for Security-Oriented Provenance

Problem Statement

Provenance systems aim to capture the origins, transformations, and interactions of kernel objects to support security tasks such as intrusion detection, forensic auditing, and threat hunting. While the value of provenance graphs is well-established, existing capture approaches face significant hurdles in production environments. These challenges span three architectural layers:

1. Capture Layer: Issues include performance overhead, lack of event integrity and availability, and difficulty in determining which kernel objects to capture.
2. Reduction Layer: The need to filter telemetry to achieve specific granularities (e.g., per-container) without incurring prohibitive costs.
3. Infrastructure Layer: The exponential growth of provenance graphs complicates storage and analysis.

These challenges are exacerbated in cloud-native environments where containers share a single kernel, requiring the capture layer to distinguish host events from container-specific events. Prior work has largely treated the capture layer as a "given," assuming faithful and complete recording, and has failed to empirically compare the performance trade-offs of different capture mechanisms (particularly eBPF program types) and filtering strategies. Furthermore, existing tools often lack guarantees regarding the integrity and availability of captured events, rendering them unsuitable for rigorous security use cases.

Methodology

The authors employ a multi-faceted approach combining theoretical classification, empirical benchmarking, and macro-evaluation of existing tools:

1. Classification of Capture Approaches: The paper categorizes kernel telemetry capture methods into user-space (ptrace, file system snapshots) and kernel-space (integrated systems like auditd, out-of-tree kernel modules, and eBPF). These are evaluated against five security-relevant properties: portability, efficiency, visibility, safety, and robustness.
2. Empirical Evaluation of eBPF: Recognizing eBPF as the most promising approach, the authors conduct micro-benchmarks on a 24-core Debian 12 host. They evaluate four eBPF program types (LSM, tracepoint, kprobe, and tracing) across three workloads (network, file, and process).
   • Overhead Measurement: They measure CPU-time increases relative to an untraced baseline.
   • Filtering Granularity: They compare three filtering strategies: pre-filtering (via cgroup attachment), in-filtering (logic within the eBPF program), and post-filtering (user-space filtering).
   • TOCTOU Analysis: They analyze Time-Of-Check to Time-Of-Use vulnerabilities, specifically regarding system calls accepting user-space pointers.
3. Macro-Benchmarking of Tools: The authors classify eight provenance systems and five capture agents based on their capture layers. They macro-benchmark the open-source subset (CamFlow, ConProv, ProvBPF, eAudit, Spade) to measure performance overhead, log loss, and provenance graph size.

Key Contributions

1. Identification of eBPF LSM Programs as the Optimal Capture Mechanism

The study identifies eBPF LSM (Linux Security Module) programs as the most suitable mechanism for security-oriented provenance.

• Efficiency: Unlike other eBPF types (kprobe, tracepoint, tracing) that require dual attachments (entry and exit) to capture both arguments and the final verdict (allowed/denied), LSM programs require only a single attachment. They execute only for operations allowed by all preceding LSMs, naturally filtering out denied events.
• Robustness: LSM programs operate after arguments are copied into kernel space, offering superior resistance to TOCTOU race-condition attacks compared to system-call hooking.
• Granularity: LSM programs support pre-filtering via direct cgroup attachment (available since Linux 6.0), allowing the capture layer to isolate container events before the program is triggered, significantly reducing overhead compared to in-filtering or post-filtering.

2. Empirical Quantification of Overhead and Filtering Costs

The micro-benchmarks reveal that:

• Program Type: While tracing programs have the lowest per-hook cost for system calls, LSM programs are the most efficient for provenance capture overall because they avoid the cost of a second attachment required to capture the verdict.
• Filtering Strategy: Pre-filtering (cgroup attachment) introduces the lowest overhead. In-filtering is more expensive as logic executes for every event, and post-filtering yields the highest overhead and data volume. Notably, LSM programs with pre-filtering outperform tracing programs with in-filtering.
• Workload Impact: File workloads incur the highest overhead due to the inherent latency of kernel operations like inode_init_security and the volume of events.

3. Critical Assessment of Existing Provenance Systems

The macro-benchmark of open-source provenance systems reveals a fundamental disconnect between performance and completeness:

• Log Loss vs. Overhead: Systems that build provenance graphs (CamFlow, ProvBPF, ConProv, Spade) suffer from massive log loss (90–100% in many workloads) to manage overhead, resulting in incomplete graphs.
• Efficiency vs. Modeling: eAudit is the only system that avoids log loss, but it incurs moderate overhead (5.9% on network, 12% on file workloads) and does so by emitting a raw stream of system calls rather than constructing a provenance graph.
• Integrity Gaps: Most tools fail to guarantee the integrity and availability of captured events. System-call-based tools remain vulnerable to TOCTOU attacks, and graph-building tools drop significant portions of the event stream.

4. Classification of Tools

The paper provides a comprehensive classification of eight provenance systems and five capture agents (e.g., LTTng, Sysdig, Tracee, Tetragon) based on capture approach, cloud-native awareness, granularity, and maintenance status. It highlights that while capture agents offer better portability and maintenance, they are often slow and lossy when used for provenance.

Results

• eBPF LSM programs are the most efficient and robust option for provenance capture, particularly when combined with cgroup-based pre-filtering.
• TOCTOU vulnerabilities affect 69% of system calls in Linux 6.14 that accept user-space pointers, compromising the integrity of capture layers relying on system-call entry points. LSM programs mitigate this by inspecting arguments after they are copied to kernel space.
• Current provenance systems are not ready for production security use cases. They either sacrifice completeness (high log loss) to achieve performance or sacrifice performance to attempt graph construction. None successfully reconcile low overhead, zero log loss, and complete graph modeling.
• Capture agents (like Tetragon and Tracee) show promise due to their active maintenance and portability but currently lack the specific optimizations needed for efficient, complete provenance capture.

Significance and Claims

The paper claims to be the first work to systematically study and compare the capture layers across provenance systems and their specific impact on security-oriented provenance.

The authors modestly assert that while eBPF is the de facto mechanism, the choice of program type and filtering strategy is critical. They conclude that existing tools are largely unsuitable for security use cases because they cannot guarantee event integrity and availability. The study suggests that LSM programs offer the most attractive foundation for future systems due to their stability, low overhead, and ability to provide strong integrity guarantees. However, the authors note that a fair evaluation must ultimately extend beyond capture mechanics to assess the semantic quality of the resulting graphs for downstream tasks like intrusion detection accuracy.