LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics

1. Problem Statement

Modern software systems generate massive volumes of heterogeneous log data, making manual inspection infeasible. Automated anomaly detection is critical for identifying system failures, security breaches, and performance degradation. However, existing solutions face significant challenges:

• Traditional Methods: Rely on a two-stage pipeline (log parsing followed by machine learning classification). These require significant engineering effort for parser configuration, depend heavily on labeled training data, and struggle to adapt when log formats evolve.
• LLM Potential: While Large Language Models (LLMs) possess inherent semantic understanding of logs, there is a lack of systematic, "apples-to-apples" comparisons between LLM-based approaches, fine-tuned transformers, and traditional methods across standardized benchmarks. Practitioners lack clear guidance on which approach to adopt based on constraints like accuracy, cost, latency, and label availability.

2. Methodology

The authors established a unified evaluation framework to compare three distinct paradigms across four public datasets (HDFS, BGL, Thunderbird, and Spirit).

A. Evaluation Categories

1. Traditional Pipeline:
   • Parsing: Uses classical parsers (Drain, Spell, AEL) to convert raw logs into structured event templates.
   • Classification: Trains machine learning classifiers (Logistic Regression, Random Forest, SVM, Isolation Forest) on the parsed sequences or event counts.
2. Fine-Tuned Transformers:
   • Directly fine-tunes pre-trained models (BERT, RoBERTa, DeBERTa-v3) on raw log messages, bypassing the parsing step.
   • Uses standard classification heads with early stopping based on validation F1-scores.
3. Prompt-Based LLMs:
   • Evaluates models (GPT-3.5, GPT-4, LLaMA-3) in zero-shot and few-shot settings.
   • Novel Contribution: Introduces Structured Log Context Prompting (SLCP), a strategy that augments prompts with:
     • System context (description of the generating system).
     • Temporal context (timestamps and frequency stats).
     • Semantic markers (keywords like "error," "fatal").
     • Explicit classification instructions.

B. Evaluation Metrics

• Primary: F1-Score (due to class imbalance).
• Secondary: Precision, Recall, AUC, Inference Latency (ms/prediction), Cost (API cost per 1000 predictions), and Label Efficiency (performance vs. amount of labeled data).

3. Key Contributions

• Comprehensive Benchmark: The first study to systematically compare traditional, fine-tuned, and prompt-based LLM approaches across four diverse datasets with consistent preprocessing.
• Structured Log Context Prompting (SLCP): A novel prompting technique that improves LLM zero-shot performance by 8–12% (specifically +2.9% to +3.1% in the main experiments) by providing structured context.
• Practical Guidelines: Provides actionable deployment strategies based on trade-offs between accuracy, cost, latency, and data availability.
• Reproducibility: All code, configurations, and processed datasets are publicly released.

4. Key Results & Findings

A. Accuracy Performance

• Fine-Tuned Transformers: Achieve the highest accuracy. DeBERTa-v3 consistently outperforms all other methods, reaching F1-scores between 95.3% and 98.9%.
• Prompt-Based LLMs: Demonstrate remarkable zero-shot capabilities. GPT-4 achieves F1-scores of 81.2%–88.3% without any training data, outperforming some traditional supervised methods.
• Impact of SLCP: Applying SLCP to GPT-4 in zero-shot mode boosts F1 by ~3%, and combining SLCP with 5-shot examples (F1: 87.9%–93.8%) approaches the performance of traditional supervised pipelines.
• Traditional Methods: Remain competitive (e.g., Drain + Random Forest: 86.4%–95.1%), proving that well-engineered pipelines are still viable.

B. Cost and Latency Trade-offs

• Cost: Traditional methods and local LLaMA-3 deployment are effectively free (compute cost only). GPT-4 is expensive (~$8.40 per 1,000 predictions).
• Latency: Traditional methods are fastest (0.3ms/pred). Fine-tuned transformers are moderate (12ms). API-based LLMs have high latency (340ms–890ms).
• Conclusion: For high-throughput production systems, fine-tuned transformers or traditional methods are more practical. For low-volume or exploratory tasks, LLMs are viable.

C. Label Efficiency

• Low-Label Regime: LLMs excel when labeled data is scarce. With only 1% of labels (used as few-shot examples), GPT-4 + SLCP (89.1% F1) significantly outperforms traditional methods (71.3% F1) and fine-tuned DeBERTa (82.4% F1).
• High-Label Regime: As labeled data increases (25%+), fine-tuned transformers surpass LLMs.

D. Failure Modes

• Traditional: Fail on novel log templates and subtle anomalies in normal event sequences.
• Fine-Tuned: Robust to new templates but struggle with sequences exceeding context windows.
• LLMs: Exhibit inconsistent behavior (temperature-dependent) and struggle with numerical anomalies requiring domain-specific thresholds.

5. Significance and Practical Guidelines

This paper bridges the gap between theoretical LLM capabilities and practical system diagnostics. It offers specific recommendations for practitioners:

Scenario	Recommended Approach	Rationale
High Accuracy + Labeled Data	Fine-tuned DeBERTa-v3	Best accuracy with reasonable latency.
No Labeled Data + Moderate Cost	GPT-4 + SLCP (Zero-shot)	Strong baseline without training; handles format changes well.
No Labeled Data + Cost Sensitive	LLaMA-3 (Local) + SLCP	Cost-effective alternative with competitive performance.
High Throughput + Low Latency	Traditional (Drain + RF)	Minimal latency; competitive accuracy for stable systems.
Evolving Log Formats	LLM or Transformer	More robust to format changes than rule-based parsers.

Conclusion: While fine-tuned transformers remain the gold standard for accuracy when data is available, LLMs (especially with SLCP) offer a powerful, label-free alternative for real-world scenarios where anomalies are rare and labeled data is scarce. The study emphasizes that the "best" method depends entirely on the specific constraints of the deployment environment.