Technical Summary: FAME – Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection

Problem Statement

Production distributed systems generate millions of log lines daily, yet existing anomaly detectors typically operate at the window- or session-level. This coarse granularity assigns a single anomaly label to a contiguous group of lines, forcing operators to manually inspect hundreds of routine messages to locate the specific fault. While message-level (line-level) detection offers the necessary precision for direct fault localization, it faces three significant challenges:

1. Template Ambiguity (C1): A single event template (derived from log parsing) can contain both normal and anomalous instances depending on parameter values, making template identity insufficient for classification.
2. Heterogeneous Failure Modes (C2): Anomalies span diverse subsystems (e.g., memory errors, network partitions, filesystem faults). A single global classifier struggles to fit one decision boundary across these heterogeneous populations without sacrificing precision or recall.
3. Label Scarcity (C3): Manually labeling millions of lines at the message granularity is operationally impractical. While Large Language Models (LLMs) can reason over log semantics, invoking them for every line in a continuous stream incurs prohibitive API costs and latency.

Methodology: FAME Framework

FAME (Failure-Aware Mixture-of-Experts) addresses these challenges through a label-efficient, failure-aware Mixture-of-Experts (MoE) architecture. The core insight is that LLM reasoning is required only once during an offline setup phase; all subsequent inference runs on lightweight, on-premise models.

1. Offline Setup Phase

• Log Parsing & K-Shot Sampling: Raw logs are parsed into EventIDs using Drain3. To control annotation costs, the system samples at most K labeled lines per EventID (e.g., K=100). These samples are split into training and calibration subsets.
• LLM-Guided Semantic Partitioning: An LLM is invoked once offline. It receives statistics and representative examples for each EventID and proposes a partition of templates into semantically coherent failure domains (e.g., "DDR Memory Errors," "Network Link Failures").
• Certification: The LLM's proposal is validated deterministically against the K-shot labels:
  • Pure-Anomaly Domains: If all K samples in a domain are anomalous, the domain is certified as "pure-anomaly." No classifier is trained; the routing mechanism itself acts as the detector.
  • Mixed Domains: If samples contain both normal and anomalous lines, a dedicated expert classifier is trained.
  • Universal Normal: Unassigned or non-anomalous templates default to a universal normal pool.
• Model Training:
  • Router: A lightweight DistilBERT model is trained to route incoming lines to the correct failure domain (or the Universal Normal pool). It consists of a binary Gate (expert vs. normal) and a multiclass Selector (specific expert).
  • Experts: Mixed domains are trained using a Two-Phase approach:
    1. Domain-Adaptive MLM Pre-training: BERT is continued pre-training on a large corpus of normal logs to adapt to log syntax.
    2. Supervised Fine-tuning: Experts are fine-tuned with Focal Loss to handle severe class imbalance, using the K-shot labeled data and a pool of normal lines.

2. Online Inference Phase

• Routing: Incoming log lines are processed by the Gate. If the score is below the threshold, the line is routed to the Universal Normal expert. If above, the Selector routes it to a specific failure domain.
• Detection:
  • Lines routed to Pure-Anomaly domains are immediately flagged as anomalies.
  • Lines routed to Mixed domains are scored by the specific expert and compared against a calibrated threshold.
  • Lines routed to Universal Normal are scored by a fused model (combining the expert score and gate output) to catch misrouted anomalies.
• Output: The system outputs a binary anomaly decision and a failure-domain label, enabling direct fault localization without session reconstruction.

Key Contributions

1. Message-Level Detection with Failure-Domain Routing: FAME produces both a binary anomaly decision and a named failure-domain label for every log line, enabling direct fault localization.
2. Failure-Aware Asymmetric MoE: The framework formalizes an asymmetric confidence property under K-shot sampling: observing all-anomalous labels in a domain is statistically conclusive (allowing routing-only detection), while all-normal labels are not. This allows the system to skip training classifiers for pure-anomaly domains.
3. Fully On-Premise Inference with Minimal Labeling: After a single offline LLM invocation, all inference runs on local models, eliminating recurring API costs and data egress. The system achieves high performance with a 76× reduction in annotation effort compared to full line-level labeling.

Experimental Results

The framework was evaluated on two benchmarks: BGL (Blue Gene/L, heterogeneous and difficult) and Thunderbird (supercomputer logs, structurally separable).

• Performance on BGL: At K=100, FAME achieves an F1 score of 98.16 (Precision: 98.18, Recall: 98.14). This significantly outperforms single-model baselines (e.g., Single BERT F1 ≈ 29.65) and classical detectors. It detects 86.3% of anomalies from unseen EventIDs (EventIDs not seen during training), demonstrating strong generalization.
• Performance on Thunderbird: FAME reaches F1 = 99.95 with perfect recall, matching the performance of simpler baselines on this easier dataset.
• Cost Efficiency:
  • Annotation: At K=100, FAME requires only 53,287 labeled lines on BGL, a 76× reduction compared to labeling the full 4.7M-line dataset.
  • Inference Cost: FAME incurs a one-time setup cost of $10.23 (including training and a single LLM call). In contrast, direct LLM inference for the same task would cost between $6,698 and $10,047 per run.
  • Throughput: The system processes up to 1.20M lines/hour on a single 4-GPU node.
• Ablation Studies: Removing the MoE routing (using a single global detector) causes a drastic performance drop, confirming that decomposing the problem into specialized sub-problems is the primary driver of success. Removing the LLM-based grouping (using TF-IDF) still yields strong results (F1=93.01 on BGL), indicating that the expert decomposition architecture is the dominant factor, while LLMs serve as a semantic booster.

Significance and Claims

The paper claims that FAME demonstrates that strong message-level detection can be achieved without the prohibitive costs of continuous LLM inference or exhaustive manual labeling. By leveraging a routed expert architecture and the statistical properties of K-shot sampling, FAME shifts the burden of reasoning to a one-time offline setup.

The authors emphasize that the primary contribution is the architecture itself rather than the reliance on LLMs. The LLM is used merely to propose a semantic partition, which is then validated and fixed. The system's ability to generalize to unseen templates and provide failure-domain labels makes it a practical solution for real-world production environments where operators need precise, actionable alerts without incurring massive API costs or manual review overhead. The framework is presented as a foundation for future work in online adaptation and causal incident analysis, but the current work focuses strictly on the efficacy of the offline-setup, on-premise inference pipeline.