VarParser: Unleashing the Neglected Power of Variables for LLM-based Log Parsing

1. Problem Statement

Log parsing is a critical preprocessing step for automated log analysis (e.g., anomaly detection, root cause analysis). While Large Language Models (LLMs) have shown superior performance in parsing compared to traditional syntax-based or semantic-based methods, existing LLM-based parsers suffer from a "constant-centric" strategy. They focus primarily on the static text (constants) of logs while ignoring the dynamic parts (variables).

This approach leads to four key limitations:

1. Inefficient Sampling: Grouping logs based only on constants results in a sampling pool size roughly equal to the number of templates, limiting efficiency.
2. Low Parsing Efficiency: Constant-based caching fails to capture dynamic variable positions, leading to frequent matching failures and redundant LLM invocations.
3. High Costs: Prompts dominated by constant text consume excessive tokens, increasing LLM invocation costs.
4. Loss of Visibility: Existing methods replace variables with generic placeholders (e.g., <*>), discarding valuable dynamic information (e.g., specific IP addresses, error codes) needed for system visibility and downstream tasks.

2. Methodology: VarParser

The authors propose VarParser, a variable-centric log parsing framework. Instead of treating variables as noise to be masked, VarParser leverages them as the primary signal for grouping, matching, and learning. The framework consists of three core modules:

A. Variable Contribution Sampling

To select a diverse and information-rich set of historical logs for LLM prompting, VarParser introduces a new sampling strategy:

• Grouping by Uncommon Tokens: Instead of grouping by constants, logs are grouped based on "uncommon tokens" (variables) extracted using NLTK. Logs sharing the same variable types are grouped together, regardless of their template constants.
• Contribution Calculation: Variables are clustered based on Jaccard similarity of their character sets. The contribution of a log is calculated based on the frequency and diversity of the variable clusters it contains.
• Greedy Diversified Sampling: An algorithm iteratively selects logs with the highest contribution, resetting the contribution of their contained variable clusters to zero. This ensures the sampled set covers the maximum variety of variable types with the minimum number of logs.

B. Variable-Centric Parsing Cache

VarParser optimizes the parsing cache to reduce LLM calls and improve matching accuracy:

• Variable-Fuzzed Matching Table: Before querying the LLM, logs are pre-processed by masking common dynamic content (IPs, numbers) into a generic {Variable} token. A fuzzy matching table maps these fuzzed logs to existing templates.
• Mask Matching with Variable Units: If fuzzy matching fails, the system performs token-level matching against a prefix tree. Crucially, it uses Variable Units stored in the cache. If a log partially matches a template but fails due to a new variable instance, the system masks the known variable positions in the template and calculates similarity (Jaccard) on the remaining tokens. This prevents "over-parsing" and reduces redundant LLM queries.
• Cache Correction: To address LLM hallucinations (e.g., misidentifying constants as variables), the system validates variable labels. If a label is generic (e.g., "label"), identical to the content, or a punctuation mark, it is corrected back to a constant.

C. Adaptive Variable-Aware In-Context Learning (ICL)

When a log cannot be matched in the cache, it is sent to the LLM with a specialized prompt:

• Token-Level Example Selection: Instead of selecting full log-template pairs (which are heavy on constants), VarParser selects specific token-level variable examples from the cache. It matches the uncommon tokens in the new log to the most similar variable instances in the cache.
• Prompt Design: The prompt includes instructions, these targeted variable examples (formatted as variable -> {label}), and a format reference. This significantly reduces token consumption while providing the LLM with the specific domain knowledge needed to identify variable boundaries.

3. Key Contributions

1. Variable-Centric Strategy: The first log parsing framework that shifts the focus from constants to variables, demonstrating that variables are the key to efficient grouping and matching.
2. VarParser System: A complete implementation featuring:
   • Variable contribution sampling for efficient data selection.
   • A variable-fuzzed matching table and mask-matching mechanism to minimize LLM calls.
   • Adaptive ICL that uses token-level variable examples to reduce costs.
3. Enhanced Result Integrity: By caching "Variable Units," VarParser preserves rich variable information (types, specific instances, frequencies) rather than discarding them, improving system visibility for engineers.
4. Comprehensive Evaluation: Extensive experiments on the Loghub-2.0 benchmark (14 datasets, ~50M logs) validating the approach against state-of-the-art baselines.

4. Experimental Results

Evaluated on 14 datasets from Loghub-2.0 using metrics like Grouping Accuracy (GA), Parsing Accuracy (PA), and F1-scores (FGA, FTA):

• Accuracy: VarParser achieved the highest average accuracy across all metrics compared to 8 baselines (including syntax-based, semantic-based, and other LLM-based parsers).
  • Outperformed the best baseline by 3.9% in GA, 8.5% in PA, and 5.8% in FTA on average.
  • Demonstrated superior performance in the most rigorous metric, FTA (Template Accuracy).
• Efficiency:
  • Sampling Speed: Achieved a 2.3× speedup in log sampling compared to LILAC by reducing the sampling pool scale from templates to variables.
  • Parsing Time: Reduced parsing time by 51.2% compared to LILAC and 14.8% compared to LogBatcher, approaching the speed of traditional syntax-based parsers like Drain.
• Cost Reduction:
  • Reduced total token consumption per dataset by 56.1% compared to LILAC.
  • Reduced token consumption by 13.0% compared to LogBatcher, despite LogBatcher's batch processing advantages.
• Robustness: The method maintained high performance across different LLMs (GPT-3.5-Turbo, Llama3-70b, Qwen-Plus), proving its adaptability.

5. Significance

VarParser addresses a fundamental inefficiency in current LLM-based log parsing: the over-reliance on static text. By treating variables as the primary source of information, it solves the "token wastage" and "redundant invocation" problems inherent in constant-centric approaches.

The significance of this work lies in:

• Practicality: It makes LLM-based log parsing economically viable for large-scale industrial systems by drastically reducing API costs and latency.
• Operational Visibility: Unlike previous methods that strip away dynamic data, VarParser retains variable instances, providing engineers with actionable insights (e.g., specific failing IPs or file paths) directly from the parsing process.
• Paradigm Shift: It establishes a new direction for LLM applications in software engineering, suggesting that leveraging dynamic data structures (variables) is more effective than relying solely on semantic understanding of static text.