Technical Summary: Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysis

1. Problem Statement

The paper addresses three critical limitations in current Large Language Model (LLM) applications for event log analysis:

1. Resource and Security Constraints: LLMs require significant computational power (often necessitating expensive cloud GPUs) and pose data privacy risks when sensitive organizational logs are sent to external service providers.
2. Lack of Actionable Remediation: Existing tools and approaches primarily focus on identifying anomalies or problems but fail to provide actionable remediation steps or solutions.
3. Research Gap: While Small Language Models (SLMs) have shown promise in other domains, their independent application to event log analysis—specifically for generating solutions rather than just detection—remains underexplored. Prior work often combines SLMs with LLMs or restricts SLMs to anomaly detection only.

2. Methodology

The study employs a two-stage methodology involving dataset generation, fine-tuning, and expert evaluation.

Stage 1: Synthetic Dataset Generation

• Motivation: Existing public datasets (e.g., LogHub) were deemed unsuitable due to small size (<2,000 entries), lack of JSONL formatting, and absence of remediation solutions. Additionally, real-world logs were unavailable due to sensitivity and scope limitations.
• Process: The authors utilized a high-performing LLM (Claude 3.7 Sonnet) via the OpenRouter API to generate a large-scale synthetic dataset.
  • Input: Real-world Windows log samples covering system health, attacks, and normal activities were provided as context.
  • Generation: The LLM generated 10,000 synthetic Windows event logs in JSON format.
  • Labeling: The LLM was prompted to identify problems within these logs and generate detailed, step-by-step remediation actions.
  • Correlation: The DepCOMM tool was used to correlate logs into related groups, ensuring the dataset included both individual logs and correlated groups (essential for detecting attacks like DDoS that appear benign individually).
• Validation: Nine cybersecurity experts (from academia and industry) validated the dataset's realism, format, and the sufficiency of the generated solutions.

Stage 2: Fine-Tuning and Evaluation

• Models: The study fine-tuned three SLMs (BTLM-3b, Bloom-4b, Gemma-4b) and three LLMs (Mistral-7b, Gemma-7b, Bloom-7b).
• Technique: LoRA (Low-Rank Adaptation) was used for parameter-efficient fine-tuning to reduce computational requirements and training time.
• Environment: Experiments were conducted on a cloud-based A100 PCIe GPU to ensure consistency across all models, despite the goal of local SLM deployment.
• Testing: A real-world testing dataset of 20,000 logs (split into seven subsets) was used. DepCOMM correlated logs into groups.
• Handling Context Limits: For log groups exceeding seven entries, a specific method was implemented where the model receives the previous response and is asked to update it with new information, mitigating context window limitations.
• Evaluation: The same nine experts evaluated the fine-tuned models' outputs based on:
  • Correctness of problem identification.
  • Sufficiency and clarity of remediation steps.
  • Presence of hallucinations.
  • Ability to handle large correlated log groups.

3. Key Contributions

The paper claims three primary novel contributions:

1. Solution-Aware Synthetic Dataset: The creation of a large-scale (10,000 entries), fine-tuning-ready Windows event log dataset that includes paired problem identification and actionable remediation steps, addressing the common limitation of detection-only datasets.
2. Solution-Oriented Fine-Tuning: The application of fine-tuning to SLMs and LLMs specifically to generate remediation solutions, moving beyond simple anomaly detection.
3. Human-in-the-Loop Benchmarking: An expert-led evaluation comparing SLMs and LLMs on correlated log groups, providing evidence-based insights into the trade-offs between model sizes for diagnostic tasks.

4. Results

• Dataset Quality: Experts confirmed the synthetic dataset accurately reflected real-world scenarios, with 100% agreement on the correctness of problem identification and high scores for solution sufficiency.
• Efficiency of LoRA: The use of LoRA significantly reduced fine-tuning time (ranging from 35 minutes to 1 hour 13 minutes) and allowed training on lower-specification GPUs compared to traditional full fine-tuning.
• SLM vs. LLM Performance:
  • Overall: Fine-tuned SLMs consistently outperformed LLMs in identifying issues and providing relevant remediation.
  • Specific Models: Gemma-4b emerged as the best-performing model, producing clear, low-hallucination outputs capable of analyzing large log groups. Mistral-7b also performed well but was slightly less detailed than Gemma-4b in some aspects.
  • LLM Underperformance: Larger models (Gemma-7b and Bloom-7b) severely underperformed, exhibiting significant hallucinations and failing to analyze log groups larger than seven entries. They often prioritized instructions over previous context, leading to a loss of information in multi-turn or large-group analysis.
  • Comparison: The SLM versions of Bloom and Gemma outperformed their LLM counterparts, suggesting that smaller models may learn the specific task more effectively with the available dataset size.
• Time to Analyze: Interestingly, LLMs were faster at analyzing the testing dataset than SLMs, a result the authors attribute to the complexity of event log analysis and suggest future work could address via agentic frameworks.

5. Significance and Claims

The paper asserts that its study demonstrates a practical, secure, and cost-effective alternative to cloud-based LLMs for event log analysis. By proving that fine-tuned SLMs can rival or exceed LLM performance in generating actionable solutions while requiring fewer resources, the authors argue for a shift in deployment strategies.

The significance lies in:

• Security: Enabling local deployment of models that process sensitive logs without transmitting them to the cloud.
• Actionability: Moving the field from mere detection to providing users with clear, step-by-step remediation plans.
• Efficiency: Validating that LoRA fine-tuning on SLMs is a viable path for creating specialized, high-quality diagnostic tools without prohibitive computational costs.

The authors conclude that while SLMs are currently the superior choice for this specific task, future work should explore agentic frameworks to handle specific log types and potentially automate the implementation of the suggested solutions.