💻 computer science

Operational Runtime Behavior Mining for Open-Source Supply Chain Security

This paper presents HeteroGAT-Rank, an industry-oriented system that leverages attention-based graph learning on runtime execution behaviors to generate interpretable, actionable signals for security analysts to investigate and prioritize suspicious open-source supply chain components.

Original authors: Zhuoran Tan, Ke Xiao, Jeremy Singer, Christos Anagnostopoulos

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhuoran Tan, Ke Xiao, Jeremy Singer, Christos Anagnostopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard trying to catch a thief who has sneaked into a massive, automated warehouse (the software supply chain). The thief doesn't just steal a box; they might tamper with the instructions, change the locks, or leave a fake note.

For a long time, security guards only looked at the blueprints (static code analysis) to find the thief. But modern thieves are smart; they hide their plans or write them in invisible ink that only appears when the warehouse starts running. So, guards started watching the live video feeds (runtime telemetry) of the packages as they install and run.

However, there's a problem: some cameras are high-definition and show exactly what the thief is holding (a specific IP address or command), while others are blurry and only show "someone moved a box" or "a door opened." This paper, HetHunt, is a new tool designed to help security guards make sense of these different camera feeds to find the bad guys.

Here is how it works, broken down into simple concepts:

1. The Problem: Too Many Cameras, Different Angles

The authors noticed that security systems collect data in two very different ways:

The "High-Def" Camera (Event-level): This records every single action: "The package opened a file called secret.txt," or "It called the number 192.168.1.1." It's detailed but often noisy (full of harmless background chatter).
The "Summary" Camera (Aggregated): This doesn't show specific names. It just says, "The package opened 50 files" or "It made 3 network connections." It's less detailed but very stable.

The challenge is that a standard AI model might say, "This package is 90% likely to be bad," but it can't tell the human guard what to look at. It's like a doctor saying, "You have a fever," but not pointing to the specific infection.

2. The Solution: HetHunt (The "Detective's Map")

HetHunt is a framework that turns these messy video feeds into a standardized map (a graph).

The Map: It takes actions from both high-def and summary cameras and draws them on the same type of map. It connects the "Package" to the "Process," then to the "File," "Command," or "Network."
The Detective (The AI): It uses a special type of AI (a Graph Neural Network) to look at this map and decide, "Yes, this package is suspicious."

3. The Magic Trick: From "Important" to "Suspicious"

This is the most critical part of the paper.

The Trap: When the AI says, "This connection is important to my decision," it doesn't always mean it's a crime. Sometimes, the AI is just paying attention because every package connects to a common library (like a grocery store). That's important for the math, but not a crime.
The Fix (Rarity Calibration): HetHunt adds a second step. It asks: "Is this action rare among good packages?"
- If a package connects to a weird, unknown server that no one else connects to, HetHunt flags it as highly suspicious.
- If a package connects to a common server that 10,000 good packages also use, HetHunt ignores it, even if the AI thought it was "important."

Think of it like a metal detector at an airport.

The detector beeps for any metal (the AI's "importance").
But the security guard knows that 99% of people have keys or coins (common, benign behavior).
HetHunt is the guard who filters out the coins and only stops you if you have a weird, rare piece of metal that no one else carries.

4. The Results: What Did They Find?

The authors tested HetHunt on two different "warehouses" (datasets):

Warehouse A (QUT-DV25): This had "Summary" cameras.
- Result: HetHunt was excellent at ranking the bad packages. It couldn't point to a specific file name (because the camera was blurry), but it could say, "This package is acting strangely compared to the others." It found the "behavioral patterns" (TTPs) rather than specific "clues" (IOCs).
Warehouse B (OSPTrack): This had "High-Def" cameras.
- Result: HetHunt could point to specific clues like "This IP address" or "This command." However, because the cameras recorded everything, there was so much noise (harmless background activity) that it was hard to set a perfect alarm. If you set the alarm too sensitive, it goes off for every harmless file read. If you set it too strict, you miss the thief.

5. The Big Takeaway

The paper concludes that you cannot judge a security tool just by how well it detects bad packages. You also have to judge how well it helps a human guard investigate.

Telemetry is the limit: If your camera is blurry, you can only find behavioral patterns. If your camera is high-def, you can find specific clues, but you have to deal with more noise.
Context matters: HetHunt doesn't just say "This is bad." It says, "Here is a list of things to investigate, ranked by how weird they are compared to normal packages."

In short, HetHunt is a tool that translates raw, messy computer logs into a prioritized "To-Do" list for security analysts, helping them ignore the noise and focus on the truly suspicious activities, regardless of how detailed their data is.

Technical Summary: Operational Runtime Behavior Mining for Open-Source Supply Chain Security

Problem Statement
Open-source software (OSS) supply-chain attacks increasingly rely on malicious logic executed during installation, import, or post-installation phases, which static analysis often misses due to obfuscation or dynamic generation. Defenders rely on runtime telemetry, but this data is heterogeneous: some sources preserve concrete artifacts (domains, IPs, commands), while others expose only aggregated behavioral summaries (syscall counts, file-access buckets). This creates a mismatch between model-centric evaluations (which treat detection as a binary package-level classification) and analyst-facing needs (which require specific, actionable runtime pivots). Furthermore, existing graph explainers identify features that preserve model predictions but do not necessarily correspond to actionable investigation leads, as high-importance edges may reflect common benign behaviors or environmental noise. The core challenge is observability-bounded hunting: the specificity of investigation evidence is strictly limited by what the telemetry source records.

Methodology: HetHunt Framework
The authors present HetHunt, an observability-aware runtime graph hunting framework designed to bridge package-level risk prediction and analyst-facing investigation. The framework operates in three stages:

Canonical Runtime Evidence Graphs:
- HetHunt maps heterogeneous runtime telemetry from different sources (event-level traces and aggregated summaries) into a shared, typed heterogeneous graph schema.
- The ontology includes six node types: PKG (package identity), PROC (process instance), FILE, NET (network endpoint), SYSCALL, and CMD (command string).
- Eleven edge types encode interactions (e.g., LOAD, DEPEND, EXEC, READ, CONNECT). A key design feature is the intermediate PROC layer, creating multi-hop paths (e.g., PKG → PROC → FILE) to distinguish behaviors like installation-time writes from import-time network connections.
Relational Graph Encoder (Risk Prediction):
- The framework uses a Relational Graph Convolutional Network (R-GCN) to learn package-level risk from the graph structure.
- While other encoders (GAT, HAN, HGT) were evaluated, R-GCN was selected for the final pipeline due to its balance of competitive classification performance, lower inference latency, and a direct typed-edge interface suitable for mapping scores back to runtime pivots.
From Importance to Suspiciousness (The Hunting Layer):
- Edge Importance Learning: A learnable edge-mask module (inspired by PGExplainer) identifies runtime interactions most influential to the model's prediction.
- Environmental Filtering: Rule-based filters suppress high-frequency background noise (e.g., common registries, system paths, routine probing commands) to prevent shared benign behaviors from dominating the ranking.
- Benign Rarity Calibration: To convert model-facing edge importance into analyst-facing suspiciousness, the framework calibrates scores against a benign reference population. An edge's suspiciousness score is calculated as:
  $s(e) = \text{importance}(e) \times (1 + \alpha \cdot \text{rarity}(e))$
  where rarity is derived from the document frequency of the edge's destination in benign packages. This ensures that influential edges are also unusual relative to normal behavior.
- Graph-Level Scoring: Packages are ranked by the mean suspiciousness of their top- $K$ retained edges, prioritizing graphs with concentrated high-suspiciousness evidence.

Key Contributions

Observability-Bounded Hunting Formulation: The paper formulates runtime supply-chain hunting as a graph problem where outputs are explicitly bounded by telemetry granularity, distinguishing between IOC-level, TTP-level, contextual, and telemetry-gap evidence.
Canonical Runtime Evidence Graphs: A compact, typed graph schema that unifies heterogeneous package-execution telemetry (from event-level and aggregated sources) into a shared representation.
HetHunt Suspiciousness Framework: An end-to-end pipeline that decouples package risk, model-facing edge importance, and analyst-facing suspiciousness, utilizing environmental filtering and relation-aware benign rarity calibration.
Hunting-Oriented Evaluation: A novel evaluation protocol that assesses not just package classification accuracy, but the quality of edge-derived rankings and the specificity of surfaced pivots across different telemetry granularities.

Evaluation and Results
The framework was evaluated on two datasets:

QUT-DV25: 14,271 Python packages with aggregated, stable dynamic-analysis telemetry.
OSPTrack: 9,758 cross-ecosystem sandbox executions with event-level telemetry (domains, IPs, commands).

Key Findings:

QUT-DV25 (Aggregated Telemetry): The rarity-calibrated HetHunt achieved strong ranking performance (0.876 AUROC, 0.880 AUPRC, and perfect Precision@100). However, the surfaced pivots were primarily TTP-level or contextual (e.g., "install-time network connection") rather than concrete IOCs, reflecting the aggregated nature of the data.
OSPTrack (Event-Level Telemetry): While this dataset preserved concrete artifacts (IOCs), the high volume of environmental noise (package manager activity, registry access) limited reliable low-FPR alerting. Rarity calibration improved ranking when restricted to network relations, but top-of-list precision remained weak.
Analyst-Usefulness: Annotation of top pivots confirmed that high ranking quality does not guarantee high pivot specificity. QUT-DV25 yielded abstract evidence, while OSPTrack yielded concrete but often package-local or environment-driven candidates.
Telemetry Volume vs. Observability: Higher telemetry volume did not correlate with better hunting utility; in OSPTrack, the highest-volume traces were dominated by benign activity, while cleaner, lower-volume traces yielded better ranking quality.

Significance and Claims
The paper claims that runtime graph explanations must be calibrated to telemetry observability before they can support operational hunting. HetHunt demonstrates that heterogeneous runtime graphs can effectively bridge the gap between package-level risk prediction and analyst investigation, provided that:

Explanations are treated as telemetry-bounded investigation pivots rather than confirmed Indicators of Compromise (IOCs).
Benign rarity calibration is applied to distinguish model-influential edges from common background noise.
The specificity of the output (IOC vs. TTP vs. Contextual) is understood as a function of the underlying telemetry source, not just the model's capability.

The authors emphasize that HetHunt does not replace package-level classifiers but provides the necessary "runtime pivots" to guide analyst inspection, acknowledging that missing concrete indicators may reflect telemetry gaps or untriggered behaviors rather than explanation failures.