Can Small GenAI Language Models Rival Large Language Models in Understanding Application Behavior?

Technical Summary: Can Small GenAI Language Models Rival Large Language Models in Understanding Application Behavior?

Problem Statement
The proliferation of malware, which increasingly utilizes obfuscation, polymorphism, and code reuse, has rendered traditional signature-based detection methods ineffective. While Large Language Models (LLMs) have demonstrated significant potential in understanding code semantics and detecting malicious behavior, their deployment in practical cybersecurity environments is hindered by high computational requirements, substantial memory usage, and energy consumption. These constraints limit their feasibility for real-time analysis, edge computing, and resource-constrained security operations centers. Consequently, there is a critical need to evaluate whether Small Language Models (SLMs)—which are lightweight, efficient, and faster—can achieve comparable malware detection performance to their larger counterparts, thereby offering a viable balance between detection accuracy and operational efficiency.

Methodology
The study employs a systematic evaluation framework to compare the capabilities of SLMs and LLMs in binary malware classification.

• Dataset: The researchers utilized the SBAN dataset, a multi-dimensional benchmark containing both benign and malware source code from open-source repositories, security benchmarks, and AI-generated code. To ensure a balanced evaluation and prevent accuracy inflation, a subset of 10,000 samples (5,000 benign and 5,000 malware) was randomly selected.
• Model Selection: Five models were tested, representing both SLM and LLM categories based on parameter count and architecture:
  • SLMs: DeepSeek-1.3B and Phi-4-mini.
  • LLMs: Llama-3.1-8B, Qwen-2.5-7B, and Mistral-7B.
• Evaluation Strategies: The study compared two distinct inference approaches:
  1. Classification Head: Utilizing the model's pre-trained weights with a lightweight classification head added for binary classification, using a sigmoid function with a default threshold of 0.5.
  2. Prompt-Based (Zero-Shot): Directly prompting the models with a specific cybersecurity instruction to classify code as 'malware' or 'benign' without additional fine-tuning, using a temperature of 0 for deterministic outputs.
• Metrics: Performance was assessed using Accuracy, Precision, Recall, and F1-score for both classes (benign and malware), alongside inference time and computational feasibility. Experiments were conducted on a workstation equipped with a single NVIDIA H100 GPU.

Key Contributions

• Comparative Analysis: The study provides a detailed comparison of LLMs and SLMs specifically for malware code diagnosis, utilizing both classification heads and direct prompting.
• Metric Granularity: Results are reported with specific metrics (precision, recall, F1-score) for both benign and malware classes, highlighting the models' ability to distinguish between the two.
• Robust Evaluation: The use of a balanced 10,000-sample subset from the SBAN dataset ensures reliable and unbiased performance assessment.
• Dual-Strategy Evaluation: The paper uniquely compares the performance of the models' native classification heads against zero-shot prompting strategies, revealing significant differences in their effectiveness.

Experimental Results

• Prompt-Based Superiority: A significant performance gap was observed between the two strategies. Across nearly all evaluated models, the prompt-based approach consistently outperformed the classification-head strategy. This suggests that the instruction-tuning objectives inherent in these models enable stronger reasoning capabilities for this task than their default classification layers.
• SLM Competitiveness: Despite having fewer parameters, SLMs demonstrated competitive performance. Notably, Phi-4-mini achieved one of the highest F1-scores and maintained balanced precision and recall values, challenging the assumption that model size is the sole determinant of detection effectiveness.
• LLM Performance vs. Cost: Larger models (e.g., Qwen-2.5-7B, Mistral-7B) demonstrated strong detection capabilities but required substantially greater inference time and computational resources.
• Precision-Recall Trade-offs: While SLMs showed high precision, some exhibited lower recall for benign samples. However, the overall results indicate that carefully designed SLMs can provide meaningful detection capabilities.

Significance and Claims
The paper claims that small GenAI models can effectively complement large models, offering a practical balance between performance and resource efficiency. The findings suggest that:

1. Operational Feasibility: SLMs are particularly suitable for real-world deployment in environments with limited computational capacity, such as edge devices and security operations centers, where latency and resource consumption are critical.
2. Hierarchical Architectures: SLMs can serve as an effective first-stage filtering mechanism in malware analysis pipelines. Suspicious samples identified by SLMs could be forwarded to larger models for deeper analysis, reducing operational costs while preserving high detection accuracy.
3. Minimal Retraining: The strong performance of prompt-based methods suggests that extensive task-specific retraining may not always be necessary; pre-trained models with carefully designed prompts can perform malware detection with minimal additional training cost.

The authors conclude that while larger models generally achieve higher overall accuracy, smaller models like Phi-4-mini offer reliable, resource-efficient solutions. The study emphasizes that model size alone does not dictate effectiveness; training quality, instruction tuning, and domain knowledge play equally important roles. Future work is directed toward enhancing SLM performance through fine-tuning on diverse datasets and exploring model optimization techniques like pruning and distillation.