Killing Two Birds with One Stone: Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence

Technical Summary: Cerebro – Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence

Problem Statement
The open-source software (OSS) supply chain has become a primary attack vector, with package registries like NPM (JavaScript) and PyPI (Python) serving as frequent targets for malicious package injections. Existing detection mechanisms face two critical limitations:

1. Ecosystem Silos: Most detectors are designed for a single ecosystem (either NPM or PyPI) and cannot leverage knowledge of malicious behaviors detected in one ecosystem to identify threats in the other. This is particularly problematic given that attackers often transcribe malicious packages across ecosystems, yet public datasets for malicious packages are small and fragmented.
2. Lack of Sequential Modeling: Current approaches typically model malicious behavior as a set of discrete, independent features (e.g., specific imports or metadata anomalies). They fail to capture the sequential nature of attacks, where malicious activities often occur in a specific order (e.g., reading sensitive data, encoding it, transmitting it, and executing a payload).

Methodology: Cerebro
To address these limitations, the authors propose Cerebro, a unified detection model that utilizes a "malicious behavior sequence" to detect threats in both NPM and PyPI. The system operates through three core components:

1. Feature Extractor (Static Analysis):

   • Cerebro employs static analysis to extract 16 language-independent features based on a high-level abstraction of malicious behavior.
   • The features are categorized into five dimensions: Metadata (e.g., suspicious names, though Cerebro focuses on code-level features to avoid registry-specific assumptions), Information Reading (OS/file system module usage), Data Transmission (network module usage, URLs), Encoding (obfuscation techniques like base64), and Payload Execution (process module usage, eval, bash scripts).
   • By relying on code-level abstractions rather than registry metadata, Cerebro enables the fusion of knowledge across different ecosystems.

2. Behavior Sequence Generator:

   • This component organizes extracted features into a sequential representation to model the execution flow of an attack.
   • Prioritization: Methods are prioritized based on their execution likelihood: Install-time (highest, e.g., setup.py, preinstall scripts), Import-time (e.g., __init__.py, index.js), and Run-time.
   • Call Graph Traversal: The system constructs a call graph for the package. It traverses the graph starting from prioritized methods, extracting feature instances and ordering them by line number and call dependencies. This preserves the inter-procedural sequential relationship of suspicious activities.

3. Maliciousness Classifier:

   • The generated behavior sequence is transformed into a textual description (e.g., "import network module, use network module call, read sensitive information").
   • This text is fed into a pre-trained language model (BERT, RoBERTa, or T5) which is fine-tuned as a binary classifier. The use of natural language descriptions allows the model to leverage the semantic understanding acquired during pre-training to interpret the context of security-related terms.

Key Contributions

• Unified Detection Model: The authors implemented Cerebro, the first model to detect malicious packages in both NPM and PyPI using a single architecture based on behavior sequences.
• Behavior Sequence Modeling: The paper introduces a novel approach to modeling maliciousness not as a bag of features, but as an ordered sequence of events derived from static analysis and call graphs.
• Empirical Validation: Extensive experiments were conducted on a dataset comprising 2,675 malicious and 7,391 benign packages.
• Real-World Deployment: The system was deployed to monitor newly published packages over 7–8 months, resulting in the detection and removal of hundreds of new malicious packages.

Evaluation Results

• Effectiveness:
  • Mono-lingual (Train/Test on same ecosystem): Cerebro outperformed state-of-the-art learning-based approaches (e.g., Amalfi, SAP) by an average of 10.0% in precision and 7.4% in recall.
  • Bi-lingual (Train on both, test on one): Cerebro achieved an average improvement of 9.9% in precision and 8.9% in recall over existing methods, demonstrating the value of cross-ecosystem knowledge fusion.
  • Cross-lingual (Train on one, test on the other): The model showed significant gains in F1-score compared to baselines, though performance was lower than mono-lingual settings, indicating that while knowledge transfer is possible, ecosystem-specific nuances remain.
• Efficiency: The average time to analyze a single package is 10.5 seconds, which the authors deem practically acceptable for registry vetting. Training the model on the full mixed dataset takes approximately 39.3 hours.
• Ablation Study: Removing the behavior sequence generator or the textual transformation step resulted in significant drops in recall (up to 33% in cross-lingual scenarios), confirming the necessity of sequential modeling and semantic representation.
• Real-World Impact:
  • Over a monitoring period, Cerebro flagged 5,976 potentially malicious package versions from 923,638 new releases.
  • After manual verification, the authors confirmed 683 malicious PyPI and 799 malicious NPM packages.
  • All confirmed malicious packages were reported to and removed by the official PyPI and NPM teams. The authors received 707 thank letters for these reports.
  • Analysis of these new threats revealed that 83.3% triggered malicious behavior at install-time, and 70.9% aimed to steal sensitive information.

Significance and Claims
The paper claims that Cerebro effectively addresses the dual challenges of ecosystem fragmentation and the lack of sequential modeling in current detection systems. By abstracting malicious behavior into a language-agnostic sequence and leveraging the semantic capabilities of large language models, Cerebro achieves higher accuracy than existing rule-based or discrete-feature learning approaches. The authors emphasize that their approach is not only theoretically sound but practically viable, as evidenced by the successful detection and removal of nearly 1,500 new malicious packages in the wild. The work suggests that future detection systems should move beyond static feature lists to model the process of an attack, and that cross-ecosystem learning is a viable strategy to mitigate data scarcity in security research.