Technical Summary: Parser-Free Querying of Security Logs

Problem Statement

Security analysts routinely query system logs to detect threats and investigate incidents, but the diversity of log formats creates a significant bottleneck. While logs are inexpensive to produce, they are expensive to use because each source (firewalls, authentication services, endpoint agents) utilizes its own semi-structured format.

The industry standard approach involves building per-source parsers to normalize logs into structured schemas (e.g., OCSF) before analysis. While powerful, this requires continuous engineering effort to maintain parsers as formats evolve; for instance, Google SecOps maintains roughly 1,000 active parsers with over 200 updates monthly. Conversely, skipping parsing and searching raw files directly with tools like grep or ad-hoc scripts avoids maintenance costs but shifts the burden to the analyst. This manual approach requires deep knowledge of specific message variants and struggles to express complex, multi-line temporal queries (e.g., "find users who logged in from two different IPs within 60 seconds") which require state, grouping, and aggregation.

Both approaches fail for the same reason: they require upfront knowledge of the source log's syntax and semantics. Structured pipelines encode this in parsers, while raw-text workflows require the analyst to reconstruct it manually.

Methodology: The Sieve System

The authors present Sieve, a system that generates executable query code from natural-language security questions without requiring pre-built parsers or normalized schemas. Sieve operates by grounding a Large Language Model (LLM) with lightweight, automatically extracted log-format context.

Pipeline Overview

Sieve operates in four stages:

1. Prompt Construction: The system assembles a prompt containing the natural-language query, the log file path, and a "context strategy" (examples of log templates or raw lines). It also includes constraints (e.g., code must be Python or Bash, no network/file writes) and an output specification.
2. Code Generation: An LLM (primarily Gemini 3.1 Pro Preview or OpenAI GPT-5.4) is invoked once per query to emit a self-contained Python or Bash script. The model is instructed to return code in a JSON wrapper to ensure reliable extraction.
3. Sandboxed Execution: The generated code runs in a subprocess with no network access and restricted filesystem permissions. It is subject to safety checks (rejecting eval, exec, rm, etc.) and output-format validation.
4. Retry Mechanism: If execution fails, the script throws an exception, or the output format is violated, the error message is fed back to the LLM for a retry. Up to four retries are allowed.

Context Strategies

To ground the LLM in the specific log format, Sieve evaluates five strategies to provide context:

• Matryoshka Templates: Uses an offline LLM pipeline to generate high-quality semantic templates with named field placeholders.
• Drain3: A streaming parser that extracts templates with generic wildcards in a single pass.
• Frequency-based Templater: A lightweight, heuristic algorithm (inspired by IPLoM) that normalizes tokens, groups by length, and identifies variable positions to produce templates in under ten seconds without LLM calls.
• Random Sample: Provides 100 randomly sampled log lines without explicit templates.
• No Context: Provides only a single sample line, relying on the model's pre-trained knowledge.

The system treats the LLM as a compiler: it generates the extraction logic once, and the resulting script executes deterministically at normal program speed over the entire log.

Key Contributions

1. Sieve Pipeline: An open-source system for LLM-assisted querying of raw security logs that generates executable code from natural language, requiring only one LLM call per query followed by deterministic execution.
2. Benchmark: A new benchmark comprising 133 security queries across 5 log types (Linux Audit, SSH, cron, Puppet, DHCP). The queries range from simple filters (Class 1–2) to complex multi-line correlations involving temporal windows and stateful aggregation (Class 3–4). Ground truth answers are provided for all queries.
3. Design Space Exploration: An evaluation of context strategies, execution languages (Python vs. Bash), and model families to determine the optimal configuration for log querying.
4. Baseline Comparisons: A comparison against a human baseline (analysts writing naive grep/Python scripts in under 10 minutes) and an automated structured-parsing pipeline (Matryoshka).

Experimental Results

The evaluation was conducted on 133 queries across 5 log types.

• Performance vs. Human Baseline: Sieve matches human performance on simple filter queries but significantly outperforms manual scripting on complex queries. On complex workloads (brute-force detection, session anomaly identification, cross-event correlation), Sieve reduces the average error rate from 0.34 (human) to 0.10, a roughly 3.6× reduction. The largest gains were observed on tasks requiring timestamp parsing, joining events by shared identifiers, and maintaining rolling-window state.
• Performance vs. Structured Parsing: On simple filtering workloads, Sieve (using the lightweight Drain3 context) achieves a macro F1 score of 0.981, which is within noise of the Matryoshka structured-parsing pipeline (0.999). This demonstrates that LLM-generated code can approach the accuracy of fully parsed representations without the offline parser-generation step.
• Context Strategy: The choice of template source matters less than expected. A simple, frequency-based templater (running in seconds with no LLM calls) produced statistically indistinguishable results from the high-effort Matryoshka templates. The critical factor is coverage of message variants rather than the sophistication of the template representation.
• Language Choice: Python is the superior default for complex queries. While Bash performs comparably on simple filters, it fails significantly on stateful temporal queries (e.g., DHCP complex queries saw a drop from 0.815 F1 in Python to 0.446 in Bash) due to limitations in handling data structures and timestamp arithmetic.
• Model Robustness: The pipeline transfers across model families (Gemini 3.1 Pro and GPT-5.4), though GPT-5.4 showed lower performance on complex stateful tasks in this specific configuration.

Significance and Claims

The paper claims that Sieve bridges the gap between the expressiveness of structured log querying and the immediacy of working directly with raw files. Its primary significance lies in demonstrating that LLM-generated code can serve as a just-in-time query plan that eliminates the need for continuous parser maintenance while preserving the accuracy of structured analysis.

The authors argue that Sieve is particularly valuable for the "long tail" of analyst queries that fall outside a SIEM's normalized schema or for ad-hoc investigations where time pressure precludes building a parser. By shifting the cost from continuous engineering (maintaining parsers) to a bounded per-query cost (one LLM call + deterministic execution), Sieve offers a scalable alternative to the "normalize first, query later" architecture. The results suggest that for complex security investigations, automated code generation is not only feasible but superior to manual scripting, provided the system is grounded with sufficient log-format context.