🤖 machine learning

LogSieve: Task-Aware CI Log Reduction for Sustainable LLM-Based Analysis

LogSieve is a lightweight, task-aware log reduction technique that significantly decreases the volume and computational cost of CI logs for LLM-based analysis while preserving semantic fidelity, thereby enabling more sustainable and interpretable automated debugging.

Original authors: Marcus Emmanuel Barnes, Taher A. Ghaleb, Safwat Hassan

Published 2026-01-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcus Emmanuel Barnes, Taher A. Ghaleb, Safwat Hassan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a few clues, you are handed a stack of 10,000 pages of notes. Most of those pages are just boring lists of supplies bought, weather reports, or the detective walking into the room. Only a handful of pages actually contain the clues that solve the crime.

This is exactly the problem software developers face with CI Logs (the digital "diaries" of their software building process). When a software build fails, the logs are often massive, noisy, and full of repetitive, unimportant text.

LogSieve is a new tool designed to act like a super-efficient editor for these logs before a smart AI (called a Large Language Model, or LLM) tries to read them.

Here is how the paper explains it, broken down into simple concepts:

1. The Problem: Too Much Noise, Too Much Cost

When software fails to build, the logs are like a messy room. They contain:

The Clues: Error messages, crash reports, and specific reasons why it failed.
The Clutter: "Loading package," "Downloading tools," timestamps, and generic progress bars.

Currently, when developers ask an AI to "figure out why this failed," they feed the AI the entire messy room. This is expensive (it costs money to run the AI), slow (it takes time to read), and wasteful (it uses a lot of energy).

2. The Solution: LogSieve (The Smart Filter)

The researchers created LogSieve. Think of LogSieve as a sieve (a kitchen tool with holes) that you shake the logs through.

What it does: It looks at every single line of the log before the AI reads it.
The Rule: If a line is just "boring noise" (like "loading library"), it falls through the holes and is thrown away. If a line is a "clue" (like "Error: File not found"), it stays in the basket.
The Goal: It keeps the "Root Cause" (the real reason for the failure) but removes the rest.

3. The Results: Smaller, Faster, Just as Smart

The team tested LogSieve on 20 different open-source Android projects. Here is what happened:

Shrinking the Stack: On average, LogSieve cut the size of the logs by 42% (lines) and 40% (words/tokens).
- Analogy: It turned a 100-page novel into a 60-page version, but the plot and ending were exactly the same.
The AI Still Gets It: They asked a smart AI (GPT-4o) to explain the failure using the full logs and the reduced logs.
- The Result: The AI gave almost the exact same answer for both. The "meaning" was preserved. The AI didn't get confused by the missing "boring" lines.
- Accuracy: The AI's categorization of the error was correct 80% of the time on the reduced logs, compared to the full logs.
Beating the Competition: They compared LogSieve to two other methods:
1. LogZip: A tool that just compresses files (like zipping a folder) but keeps all the boring lines. Result: The AI got confused because the noise was still there.
2. Random Removal: Just deleting random lines. Result: The AI sometimes got lucky, but often missed the clues.
- LogSieve was the clear winner because it knew what to keep based on the meaning, not just the structure.

4. Making it Automatic: The "Robot Editor"

You might ask, "Who decides which lines are boring?"

The Human Touch: First, the researchers manually labeled thousands of lines to teach the computer what "important" looks like.
The Automation: They trained a smaller, faster computer program (a machine learning classifier) to do this labeling automatically.
The Score: This "Robot Editor" was 97% accurate at deciding what to keep. This means LogSieve can run automatically in the background without needing a human to read every log.

5. Why This Matters: Saving Energy and Money

The paper emphasizes that processing fewer words saves energy.

The Analogy: Imagine driving a truck. If you remove 40% of the cargo (the useless log lines), the truck uses less gas to get to the destination.
The Impact: By feeding the AI less data, companies save money on computing costs and reduce their carbon footprint (energy use).

Summary

LogSieve is a tool that cleans up the messy, noisy logs of software builds before an AI tries to read them. It throws away the boring stuff and keeps the important clues. This makes the AI faster, cheaper, and greener, while still allowing it to solve the mystery of why the software failed.

What the paper does NOT claim:

It does not claim to fix the software bugs itself; it only helps find the reason for the failure.
It does not claim to work on every single type of log in the world yet (it was tested on Android projects).
It does not claim to replace human developers, but rather to help them by making the AI's job easier.

Technical Summary: LogSieve

Problem Statement

Continuous Integration (CI) logs, particularly within mobile ecosystems like Android, are characterized by high volume, verbosity, and unstructured noise. While essential for diagnosing build failures and performance regressions, their sheer size makes manual inspection and automated analysis increasingly costly, time-consuming, and environmentally unsustainable. The adoption of Large Language Models (LLMs) for CI tasks (e.g., failure explanation, root cause analysis) exacerbates these issues, as inference costs and latency scale directly with input token length.

Existing log reduction techniques primarily focus on "structure-first" goals such as compression, deduplication, or template clustering (e.g., LogZip, Drain, Spell). While these methods improve storage efficiency and indexing, they often retain task-irrelevant content (e.g., timestamps, dependency downloads, environment setup) that inflates token budgets without aiding downstream reasoning. Conversely, LLM-based reduction methods often incur high inference costs during the reduction process itself. There is a critical gap in "task-aware" reduction that filters low-information lines based on semantic relevance to specific diagnostic tasks (like Root Cause Analysis) before LLM inference occurs.

Methodology

The paper introduces LogSieve, a lightweight, pre-inference log reduction technique designed to filter CI logs while preserving semantic fidelity for downstream LLM reasoning. The methodology is evaluated through a four-part study design involving 20 open-source Android projects using GitHub Actions.

1. Data Collection and Ground Truth

The authors collected 9,166 GitHub Actions workflow runs from 452 Android repositories, focusing on 1,806 failed runs. A stratified random sample of 20 failed workflows (one per repository) was selected for detailed analysis, comprising 14,646 log lines.

Manual Annotation: Two co-authors independently labeled each line as RCA-relevant (1) or RCA-irrelevant (0) based on guidelines prioritizing diagnostic content (error traces, exceptions) over boilerplate. Inter-rater reliability was high (Cohen's $\kappa = 0.80$ ).
Ground Truth: Disagreements were resolved via consensus, creating a validated dataset for training and evaluation.

2. LogSieve Mechanism

LogSieve operates as a pre-processing step before LLM inference:

Embedding Generation: Log lines are converted into feature vectors using three embedding families: TF-IDF (lexical), BERT (contextual), and LLaMA3 (instruction-tuned).
Classification: A suite of supervised machine learning classifiers (including Logistic Regression, SVM, Random Forest, XGBoost, and LightGBM) is trained to predict the relevance of each line.
Filtering: Lines classified as irrelevant are removed, reducing the input token count sent to the LLM.

3. Evaluation Framework

The study evaluates LogSieve against two baselines: LogZip (structure-based compression) and Random-line removal.

LLM Tasks: GPT-4o was used to perform two tasks on both full and reduced logs:
1. Failure Explanation: Generating a summary of why the workflow failed.
2. Failure Categorization: Assigning a category to the failure.
Metrics:
- Reduction: Percentage of lines and tokens removed.
- Semantic Fidelity: Cosine similarity, BERTScore, ROUGE, BLEU, and GPTScore comparing LLM responses from reduced logs vs. full logs.
- Categorical Accuracy: Exact-match accuracy for failure categorization.

Key Contributions

LogSieve Technique: A novel, RCA-aware log reduction method that filters low-information lines while retaining content essential for failure diagnosis, bridging the gap between structural compression and LLM reasoning.
Empirical Evaluation: A comprehensive evaluation on 20 Android projects demonstrating that LogSieve achieves significant size reduction with minimal semantic loss.
Automation Feasibility: Evidence that embedding-based classifiers can automate relevance detection with near-human accuracy, enabling scalable integration into CI workflows without expensive LLM inference during the filtering phase.
Sustainability Impact: A demonstration that pre-inference token reduction directly lowers computational costs and proportional energy consumption for LLM-based CI analysis.

Results

RQ1: Log Size Reduction

LogSieve achieved an average 42% reduction in log lines and 40% reduction in tokens across the 20 repositories. The reduction varied by project (ranging from 11% to 85% line reduction), indicating that CI logs contain substantial non-essential content.

RQ2: Semantic Fidelity

Despite the 40% token reduction, the reduced logs preserved high semantic fidelity for LLM reasoning:

Explanation Task: Cosine similarity averaged 0.93, and GPTScore averaged 0.93 when comparing responses from reduced vs. full logs.
Categorization Task: 80% of the reduced-log responses matched the full-log classification exactly.
These results indicate that the removed lines were largely noise that did not hinder the LLM's ability to diagnose failures.

RQ3: Automation Performance

Machine learning classifiers successfully automated the relevance labeling task:

Accuracy: Models achieved up to 97% accuracy (e.g., Logistic Regression with TF-IDF or BERT embeddings, SVM with LLaMA3 embeddings).
Efficiency: Classical ML models (linear, tree-based) performed comparably to more complex neural models, suggesting that lightweight classifiers are sufficient for this task, avoiding the overhead of running LLMs for filtering.

RQ4: Baseline Comparison

LogSieve significantly outperformed both baselines in preserving semantic and categorical fidelity:

vs. LogZip: LogSieve achieved a Cosine similarity of 0.93 (vs. 0.70 for LogZip) and 80% exact-match accuracy (vs. 20% for LogZip). LogZip's structure-first approach retained irrelevant lines and discarded contextually important ones.
vs. Random Removal: While random removal achieved decent surface similarity (Cosine 0.90), it resulted in lower categorical fidelity (70% exact match) compared to LogSieve's 80%. This confirms that semantic relevance, not just volume reduction, is critical for LLM reasoning.

Significance and Claims

The paper claims that LogSieve offers a practical path toward greener and more interpretable CI automation. By shifting from "structure-first" to "semantics-first" reduction, LogSieve addresses the growing environmental and economic costs of LLM inference in software engineering.

Sustainability: The 40% reduction in input tokens translates to proportional reductions in energy use and carbon emissions, as inference cost scales linearly with token count.
Interpretability: Unlike compression methods that obscure context, LogSieve preserves the specific diagnostic information required for root cause analysis.
Scalability: The ability to automate relevance detection with near-human accuracy using lightweight classifiers allows LogSieve to be integrated into CI pipelines without introducing significant latency or computational overhead.

The authors conclude that LogSieve effectively bridges log management and LLM reasoning, enabling scalable diagnostics without compromising the quality of automated analysis. The study acknowledges limitations regarding generalizability to other CI platforms, languages, and log types, and notes that future work will explore adaptive reduction strategies and broader LLM evaluations.