Technical Summary: CelerLog: Fast Log Parsing via Dynamic Routing

Problem Statement

Log parsing is a prerequisite for automated system analysis, transforming raw, semi-structured log messages into structured formats comprising static templates and dynamic parameters. While syntax-based parsers (e.g., Drain) offer high efficiency, they often lack the semantic reasoning required to handle complex or evolving log patterns, leading to lower accuracy. Conversely, emerging LLM-based semantic parsers achieve superior accuracy by leveraging natural language understanding but suffer from prohibitive latency and financial costs. These costs stem from the fundamental design flaw of applying heavy semantic inference to every log message, regardless of whether such complexity is necessary. Existing optimization strategies like caching or sampling fail to address the root cause: the inability to distinguish logs that require semantic reasoning from those that can be resolved via simple statistical analysis.

Methodology

The authors propose CelerLog, a hybrid log parser that dynamically routes logs to the most appropriate processing engine based on their statistical characteristics. The core insight is that the vast majority of logs exhibit repetitive patterns with varying parameters (dense groups) that can be parsed via statistical analysis, while only a minority of logs lack such patterns (sparse groups) and require LLM-based semantic inference.

CelerLog consists of three primary components:

1. Dynamic Router

The router acts as a traffic controller, classifying incoming logs into dense and sparse groups through a two-stage process:

• Two-Stage Grouping:
  • Masked Skeleton-based Grouping: Raw logs are preprocessed to mask variable tokens (numbers, strings, etc.) into designated placeholders (e.g., <NUM>, <UCL>), creating "masked skeletons." Logs sharing identical skeletons are grouped.
  • Length-based Grouping: These skeleton groups are further aggregated into "buckets" based on their token length, as logs from the same template typically share the same length.
• Anchor-based Merging: Within each bucket, the router identifies "anchor" groups (those with the highest count of unique logs). It then calculates the Position-aware Jaccard Similarity between the anchor and other groups.
  • A dynamic similarity threshold is derived from the "Singleton Ratio Curve" to determine the optimal cut-off for merging, preventing over-merging while capturing valid patterns.
  • A lightweight verb check ensures semantic consistency (e.g., ensuring the core action verb is shared).
  • Groups that successfully merge with an anchor are classified as dense log groups. Unmerged groups are classified as sparse log groups.

2. Statistical Processor (Fast Track)

Dense log groups are routed to this efficient module. Instead of invoking an LLM, it performs a straightforward statistical analysis:

• It aligns the logs in the group and analyzes the value distribution at each token position.
• Positions containing multiple distinct values are identified as parameters and masked with wildcards (e.g., <*>).
• Positions with single values are retained as static template parts.
• This process yields the final template and parameters with minimal computational overhead and zero risk of LLM hallucination.

3. LLM-based Processor (Slow Track)

Sparse log groups, which lack sufficient statistical signals, are routed to an LLM.

• Task Design: To minimize hallucination and cost, the LLM is not asked to generate the full template. Instead, it is prompted to perform variable identification, returning a list of exact variable strings found in the log.
• Verification: The system verifies that the identified variables exist in the original log before masking them to generate the final template. If the LLM fails or produces invalid output, the system reverts to the original log to prevent errors.
• Parallelism: To handle scalability, CelerLog employs data parallelism for the routing phase (processing buckets across CPU cores) and asynchronous I/O for the LLM phase (batching requests).

Key Contributions

1. Insight into Log Distribution: The authors identify that the bottleneck in existing LLM-based parsers is the uniform application of semantic inference. They demonstrate that a majority of logs possess repetitive statistical patterns sufficient for parsing without LLMs.
2. Dense vs. Sparse Categorization: The paper formally defines dense log groups (rich in statistical variation) and sparse log groups (static or isolated), proposing a hybrid strategy to exploit this distinction.
3. CelerLog Framework: The introduction of a dynamic routing mechanism that directs logs to either a fast statistical processor or an LLM-based processor, maximizing both efficiency and effectiveness.
4. Comprehensive Evaluation: Extensive experiments on 14 public datasets demonstrating that CelerLog outperforms state-of-the-art baselines in accuracy while drastically reducing time and cost.

Experimental Results

Evaluated on 14 public datasets (including Apache, HPC, OpenSSH, and Zookeeper), CelerLog demonstrates significant superiority over both syntax-based (Drain, AEL, Brain) and LLM-based (LILAC, LogBatcher, LUNAR) baselines:

• Effectiveness: CelerLog achieves an average Grouping Accuracy (GA) of 0.927 and Parsing Accuracy (PA) of 0.875, outperforming the best LLM-based baseline (LogBatcher) by a significant margin in PA (0.875 vs. 0.786).
• Efficiency:
  • CelerLog is 7.9× to 18.6× faster than LLM-based methods.
  • Notably, the parallel version of CelerLog is up to 1.5× faster than the widely adopted syntax-based parser Drain.
• Cost Reduction:
  • Token Consumption: Reduced by 80.2%–94.1% compared to LLM methods (e.g., from ~207k tokens for LUNAR to ~12k for CelerLog).
  • LLM Invocations: Reduced by 86.4%–90.9% (e.g., from ~370 invocations for LUNAR to ~34 for CelerLog).
• Robustness: Ablation studies confirm that removing the router or the statistical processor severely degrades performance or increases costs. Sensitivity analysis shows CelerLog is robust to hyperparameter settings (similarity thresholds and top-k selection). Furthermore, the system maintains stable performance across various backbone LLMs, from large models (Gemini-3-pro) to smaller ones (Qwen3-8b).

Significance

The paper claims that CelerLog establishes a new Pareto frontier for log parsing, offering a solution that is simultaneously highly accurate and cost-effective. By recognizing that not all logs require complex semantic understanding, CelerLog bridges the gap between the speed of syntax-based parsers and the accuracy of semantic parsers. The authors position this work as a practical enabler for real-world, large-scale log analysis, where the prohibitive costs of current LLM-based approaches have previously limited their deployment. The hybrid approach ensures that expensive LLM resources are reserved only for the logs that genuinely require them, making automated log analysis scalable and economically viable.