Understanding NPM Malicious Package Detection: A Benchmark-Driven Empirical Analysis

1. Problem Statement

The NPM (Node Package Manager) ecosystem is a primary target for software supply chain attacks due to its open publication model and minimal gatekeeping. Despite the existence of various detection tools (static, dynamic, ML-based, and LLM-based), malicious packages remain a persistent threat. The field suffers from four critical gaps:

1. Lack of Comparative Benchmarks: Existing tools are evaluated on incompatible datasets, making cross-tool comparison unreliable.
2. Unexplained Performance Trade-offs: It is unclear why tools exhibit specific precision-recall trade-offs (e.g., high precision but low recall) and whether this is due to engineering quality or structural constraints.
3. Unclear Attack-Detection Interaction: The relationship between specific attack characteristics (behaviors, evasion techniques) and detection failure is not well understood.
4. Temporal Degradation: Machine Learning (ML) tools suffer from performance degradation over time, often attributed to "concept drift," but the specific mechanisms in the NPM ecosystem are unknown.
5. Combination Principles: There is no established theory on how to effectively combine multiple tools to improve detection without introducing excessive false positives.

2. Methodology

The authors conducted a large-scale empirical study combining quantitative evaluation with deep source-code inspection.

• Dataset Construction:
  • Malicious: Curated 6,420 malicious package versions from 4,286 unique packages. Sources included academic datasets (BKC, DONAPI, MalOSS, etc.) and security advisories (OSV, Snyk), retrieving code from mirrors for removed packages.
  • Benign: Collected 7,288 benign packages from the official NPM registry, prioritizing highly downloaded packages to rigorously test false-positive rates.
  • Taxonomy: Annotated the dataset with 11 malicious behavior categories (e.g., data exfiltration, command execution) and 8 evasion techniques (e.g., string obfuscation, hook abuse) using LLM-assisted context extraction and expert verification.
• Tool Selection: Evaluated 8 detection tools representing 13 variants across four paradigms:
  • Static Analysis: GuardDog, OSSGadget, GENIE.
  • Dynamic Analysis: Packj (trace mode).
  • Machine Learning: SAP (with DT, RF, XGB), MalPacDetector, Cerebro.
  • LLM-Based: SocketAI.
• Evaluation Strategy:
  • Black-box + White-box: Ran tools on the unified dataset and inspected their source code to understand detection rules, feature designs, and thresholds.
  • Temporal Analysis: Evaluated performance across time periods (2020–2025) to study degradation.
  • Combination Analysis: Tested "Union" (OR logic) and "Intersection" (AND logic) strategies for tool combinations.

3. Key Contributions

1. Unified Benchmark: The largest NPM malware dataset to date (6,420 malicious + 7,288 benign) with a fine-grained taxonomy of behaviors and evasion techniques.
2. Source-Code Level Analysis: Unlike prior work, the authors reverse-engineered the tools to identify the structural mechanisms behind their performance.
3. Novel Structural Findings: Identified five core principles governing detection effectiveness, including the root cause of precision-recall trade-offs and the mechanism of ML degradation.
4. Open Resources: Released the dataset, evaluation framework, and analysis scripts.

4. Key Results & Findings

RQ1: Detection Effectiveness & Structural Trade-offs

• Performance: GuardDog achieved the best balance with an F1-score of 93.32%. Other tools ranged widely (e.g., GENIE had 99.76% precision but only 46.14% recall; Packj_static had 98.69% recall but massive false positives).
• Root Cause: The fundamental challenge is distinguishing Code Capability (what code can do) from Code Intent (what code aims to do). Since benign and malicious packages often invoke identical OS APIs (e.g., fs.readFile), tools must resolve this ambiguity.
  • Capability-end tools (flag any sensitive API) have high recall but flood false positives.
  • Intent-end tools (require strict evidence chains) have high precision but miss attacks that don't meet specific criteria.
  • GuardDog bridges this by using taint analysis (tracking data flow from source to sink).

RQ2: Fine-Grained Behavioral Analysis

• Behavioral Coupling: Detection signals are amplified when behaviors co-occur. For example, a single os.hostname() call is ambiguous, but a chain (hostname → serialize → exfiltrate) creates unambiguous intent. This raised SAP_DT detection from 3.2% to 79.3%.
• Structural Blind Spots:
  • Web Injection and Credential Theft are hard to detect because the APIs used are identical to legitimate libraries.
  • Install Script Abuse: 72.21% of malicious packages exploit npm lifecycle hooks (preinstall/postinstall). 21.9% of malware resides entirely in package.json scripts with no JavaScript payload, making them invisible to code-analysis tools.
• Evasion: Most attackers (80.3%) use no evasion because the ecosystem lacks mandatory pre-publication scanning. Among those who do, Anti-analysis and Environment Detection are the hardest to detect. Interestingly, heavy obfuscation (4+ techniques) often backfires, creating detectable anomalies (high entropy).

RQ3: Temporal Evolution & ML Degradation

• Concept Convergence vs. Drift: ML tools (e.g., SAP_DT) degraded significantly (from 87.15% to 39.49% detection rate). This was not due to attackers becoming more sophisticated (concept drift). Instead, malware became simpler (minimal footprint, no obfuscation), making it statistically indistinguishable from benign code in feature space (Concept Convergence).
• API Evolution: Malicious code shifted from broad reconnaissance (collecting many system variables) to single-step execution (e.g., exec('curl attacker.com | sh')). This bypasses API-level sink definitions used by many tools.
• Resilience: Pattern-based tools (GuardDog) remained stable or improved through rule updates, while ML tools failed without retraining.

RQ4: Tool Complementarity

• Best Combinations: Strategic combinations outperformed individual tools.
  • Union (OR): GuardDog + SocketAI achieved 95.79% F1.
  • Intersection (AND): GuardDog + Packj_static achieved 93.04% F1 with 97.39% precision.
• Principle: Effectiveness depends on Complementarity minus False-Positive Introduction, not just paradigm diversity.
  • Combining tools that extract the same features (e.g., SAP_DT + SAP_RF) yields no benefit.
  • Combining a high-recall tool with a high-precision tool works only if the high-recall tool does not introduce excessive false positives that drown out the signal.

5. Significance & Implications

• For Practitioners:
  • Tool Selection: Use GuardDog + Packj_static (Intersection) for CI/CD pipelines requiring low false positives. Use GuardDog + SocketAI (Union) for maximum coverage.
  • Configuration: Organizations should consider disabling automatic script execution (npm install --ignore-scripts) for untrusted packages due to the prevalence of install-script attacks.
  • ML Maintenance: ML tools require frequent retraining; otherwise, they degrade as malware converges toward benign code patterns.
• For Researchers:
  • Future tools should focus on behavioral chains and taint analysis to resolve capability-intent ambiguity, rather than relying on surface-level features or static signatures.
  • Research is needed on detecting dynamic code execution and credential theft, which remain structurally underdetected.
• For Ecosystem Maintainers:
  • The current open model assumes publisher trust, which is broken. Mandatory pre-publication scanning with behavioral sandboxing or restricting lifecycle script execution by default is necessary to structurally reduce the attack surface exploited by 72% of malware.

Conclusion

The paper concludes that the ambiguity between code capability and malicious intent is the root cause of all detection challenges. Tools anchored to attack objectives (what malware must do) are stable, while those anchored to attack implementations (how malware currently does it) are fragile. The study provides a roadmap for moving from reactive, feature-based detection to proactive, behavior-chain-based defense.