The Role of Domain-Specific Features in Malware Detection: A macOS Case Study

Problem Statement
Despite the increasing adoption of macOS in enterprise environments and the rising sophistication of malware targeting this platform (including infostealers and arm64-specific variants), research into macOS malware detection remains underexplored compared to Windows and Android. Existing literature relies heavily on generic static features—such as strings, byte N-grams, and section counts—while largely overlooking the unique characteristics of the Mach-O binary format and macOS-specific security mechanisms. Furthermore, the field suffers from a lack of large-scale, up-to-date, publicly available datasets, with previous studies often relying on small datasets (e.g., fewer than 200 samples) that limit the training and generalization of machine learning models.

Methodology
The authors propose a machine learning-based malware detector that leverages a comprehensive set of domain-specific static features derived from the Mach-O file format and macOS security architecture.

• Feature Engineering: The detector utilizes a 4,578-dimensional feature vector categorized into eight groups:
  • Structural: Standard binary properties (size, load commands, segments, sections) extended from Windows detection literature.
  • Byte & String: Byte histograms, N-grams, and counts of specific strings (IPs, URLs, paths, libraries).
  • Packing: Indicators of obfuscation, specifically checking for UPX-related section names and high-entropy segments.
  • APIs: Presence of 4,009 common macOS APIs across 35 frameworks, focusing on security and privacy-related calls.
  • Entitlements: Boolean flags for 55 common and security-relevant entitlements (e.g., allow-jit, allow-unsigned-executable-memory) embedded in code signatures.
  • Persistence: Detection of login items and launch agents/daemons via specific API calls or embedded .plist strings.
  • Certificates: Status of the code-signing certificate chain (e.g., validated by Apple, self-signed, expired, revoked).
• Dataset Construction: To address the data scarcity issue, the authors curated a novel dataset of 41,129 Mach-O executables (11,413 benign, 29,716 malicious) spanning x86-64, arm64, and fat (universal) architectures. Samples were sourced from macOS installations, Homebrew, open-source repositories, and various malware repositories (Objective-See, MalwareBazaar, VirusShare, VirusTotal).
• Model Training: The authors evaluated tree-based models (Decision Tree, Random Forest, XGBoost) and a deep learning baseline (MalConv). Hyperparameters were tuned via 5-fold cross-validation.
• Evaluation Strategy: The study includes a primary evaluation on the constructed dataset and a real-world generalization test using a "fresh" dataset of 9,000 samples collected from VirusTotal over a three-month period (September–November 2024) to assess temporal robustness.

Key Contributions

1. Domain-Specific Feature Integration: This work is the first to systematically integrate macOS-specific static features (entitlements, certificate status, persistence mechanisms, and specific APIs) into a machine learning detector, moving beyond generic byte/structural analysis.
2. Large-Scale Public Dataset: The authors released the largest publicly available dataset of macOS binaries for malware detection to date (41,129 samples), significantly larger than previous benchmarks (e.g., 152 samples).
3. Comprehensive Empirical Analysis: The study provides a detailed analysis of feature importance, packing prevalence, API usage differences between benign and malicious software, and the distribution of certificate states.

Results

• Detection Performance: The proposed detector, using XGBoost and the full feature set, achieved a True Positive Rate (TPR) of 98.50% at a 1% False Positive Rate (FPR) on the main dataset. This represents an average improvement of 16.56% over state-of-the-art approaches relying on generic features.
• Feature Importance: Analysis revealed that macOS-specific features (particularly certificate validation status and entitlements like allow-jit) are the most critical for classification. However, generic features alone can still achieve high performance on known samples.
• Generalization and Robustness: In the real-world evaluation on fresh data, the detector maintained a 99.50% TPR. Crucially, the removal of domain-specific features caused a 15.92% drop in detection performance on these new samples, demonstrating that generic features (like N-grams) fail to generalize to novel malware variants as effectively as domain-specific features.
• Comparison: The solution outperformed existing detectors (Pajouh-based, Thaeler-based, and MalConv) by an average of 50.03% in TPR on the fresh dataset. MalConv, which relies on raw byte sequences, performed poorly compared to feature-engineered approaches.

Significance
The paper argues that while general machine learning techniques are consistent across operating systems, the specific features they leverage must be tailored to the OS to maximize efficacy. The authors demonstrate that leveraging semantically rich, domain-specific features (such as entitlements and certificate chains) is not only essential for achieving state-of-the-art detection rates but is also critical for the generalization of detectors against new, evolving malware variants. By releasing a large-scale dataset and demonstrating the superiority of OS-specific feature engineering, this work provides a concrete foundation for advancing macOS malware detection and highlights the limitations of relying solely on generic static analysis or raw byte sequences.