💻 computer science

LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding

LogFold is a novel log compression method that improves upon state-of-the-art baselines by identifying new redundancy patterns in structured tokens and employing a type-aware hybrid encoding strategy, achieving an average 11.11% improvement in compression ratio across 16 public datasets.

Original authors: Shiwen Shan, Yintong Huo, Hongzhan Zhong, Zhining Wang, Yuxin Su, Zibin Zheng

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiwen Shan, Yintong Huo, Hongzhan Zhong, Zhining Wang, Yuxin Su, Zibin Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the librarian of a massive, chaotic library. Every day, millions of books (log files) are delivered. These books contain the history of a computer system's life: what it did, when it did it, and if it made a mistake.

The problem? The library is running out of shelf space. The books are huge, and storing them all is getting incredibly expensive. You need a way to shrink these books down so they fit, but you can't throw away any pages because you might need to read them later to solve a mystery.

Enter LogFold. Think of LogFold not just as a "compressor," but as a super-smart bookbinder that understands the structure of the stories inside the books, rather than just squishing them together randomly.

Here is how LogFold works, explained through a simple story:

1. The Problem: The "Generic" Squeeze

Most standard compressors (like Gzip) are like a vacuum-sealer. They look at a pile of clothes (data) and suck out the air. They are good, but they don't understand what the clothes are. They treat a t-shirt the same as a pair of jeans.

Old log compressors tried to be smarter. They looked for repeated sentences (like "System started") and replaced them with a short code. But they missed a huge opportunity: they didn't look closely enough at the messy, complex parts of the sentences.

2. The Insight: The "Skeleton" and the "Flesh"

LogFold realizes that many log messages look like a human skeleton with different flesh attached to it.

The Skeleton (Delimiter Skeleton): Imagine a date written as 2015-07-29. The "skeleton" is the pattern Year-Month-Day (the dashes and the structure).
The Flesh (Sub-tokens): The actual numbers 2015, 07, and 29 are the flesh.

Old tools treated the whole thing as one messy blob. LogFold says, "Wait! The skeleton is the same for thousands of entries. Let's group them by skeleton first!"

3. The Four Steps of LogFold

LogFold uses a four-step assembly line to shrink the logs:

Step 1: The Sorter (Token Analyzer)

Imagine a robot sorting incoming mail. It looks at every word in a log message and asks:

Is this a Static Word? (e.g., "Error" or "Warning" – these never change).
Is this Unstructured? (Just random text).
Is this Structured? (Like a date or an IP address that has a pattern).

It separates the mail into three piles.

Step 2: The Pattern Detective (Structured Token Processor)

This is LogFold's secret sauce. It takes the "Structured" pile (the dates, IDs, etc.) and looks for patterns within the patterns.

The Skeleton Grouping: It groups all dates together because they all share the Year-Month-Day skeleton.
The "Critical Position" Hunt: It looks at the "flesh" (the numbers). It notices that in 90% of cases, the "Month" is always 07.
The Split: Instead of keeping 07 in every single entry, LogFold says, "Okay, let's make a special rule for 'July'." It splits the group into a "July" pile and a "Not-July" pile.
The Result: By breaking the data down into smaller, more uniform groups, it finds more repetition than anyone else ever could. It's like realizing that instead of writing "July 5th, July 6th, July 7th," you can just write "July" once and then list the days.

Step 3: The Translator (Hybrid Encoder)

Now that the data is organized, LogFold translates it into a secret code, but it uses different codes for different types of data:

For Numbers: It uses a "Delta Code." Instead of writing 100, 101, 102, it writes 100, +1, +1. It's much shorter!
For Mixed Data: If a line has both numbers and words, it turns the words into numbers first, then uses the number code.
For Repeated Words: It uses a "Dictionary Code." If the word "Error" appears 1,000 times, it replaces it with a tiny symbol like #1.

Step 4: The Packer (Packer)

Finally, it takes all these tiny, efficient codes and shoves them into a single suitcase (the archive file), then zips it up one last time with a standard tool (like Gzip) to make it even smaller.

4. The Result: A Super-Small Library

When LogFold is done, the library is 11% smaller than the next best method, and in some cases, 500% smaller!

Speed: It's fast enough to be useful (about 10 MB per second), though it's not the absolute fastest. It trades a little bit of speed for massive space savings.
Unpacking: When you need to read the logs again, LogFold reverses the process. It's like un-zipping a suitcase and re-attaching the "flesh" to the "skeleton" to rebuild the original story perfectly.

Why Does This Matter?

In the real world, companies like eBay or Google generate petabytes of logs every day. Storing this data costs millions of dollars. LogFold is like a magic shrinking potion that lets them keep more history for less money, without losing a single detail.

In short: LogFold is the first tool that doesn't just squish logs; it understands their grammar, finds the hidden patterns inside complex numbers and dates, and rewrites the story in the most efficient language possible.

1. Problem Statement

Software systems generate massive volumes of log data, necessitating long-term storage for maintenance, auditing, and retrospective analysis. While general-purpose compressors (e.g., Gzip, Bzip2) exist, they fail to exploit the specific structural redundancies inherent in log data. Existing log-specific compressors suffer from two primary limitations:

Overlooking Structured Token Redundancies: Many log tokens are complex and structured (e.g., timestamps like 2015-07-09 or process IDs like ftpd[4305]). Current methods often treat these as opaque strings or fail to recognize recurring patterns within the token's internal structure (delimiters and sub-tokens).
Lack of Type-Aware Encoding: Existing approaches apply coarse-grained encoding strategies. They typically use dictionary encoding for all strings and specific numeric encoding only for pure numbers, ignoring the unique structural characteristics of mixed-type tokens (combinations of strings and numbers).

2. Methodology: LogFold

LogFold is a novel log compression framework designed to address these limitations through a delimiter skeleton-aware approach and a hybrid encoding strategy. It consists of four main components:

A. Token Analyzer

Unlike traditional parsers that first extract log templates, LogFold treats log entries holistically. It tokenizes raw logs and classifies tokens into three types:

Static: Fixed text (e.g., "Error", "Info").
Unstructured: Dynamic strings without internal delimiters (e.g., IP addresses, random IDs).
Structured: Tokens containing internal delimiters (e.g., dates, paths, bracketed IDs).
Dynamic tokens are replaced with placeholders (<-> for structured, <*> for unstructured) to preserve the static token sequence.

B. Structured Token Processor

This component targets the internal redundancy of structured tokens through a two-phase process:

Delimiter Skeleton-Aware Grouping: Tokens are grouped by their "delimiter skeleton" (the sequence of non-alphanumeric characters, e.g., <>-<>-<>). This reduces redundancy in the structural frame.
Pattern Mining: Within each group, sub-tokens are arranged into a matrix. The system identifies critical positions (columns with dominant values) using a heuristic based on histogram frequency, dominance ratio, and Shannon entropy.
- Re-grouping: Based on critical positions, the matrix is partitioned into smaller, more homogeneous subgroups.
- Refinement: Constant columns identified after partitioning are embedded back into the skeleton pattern (e.g., refining 2015-<>-<> to 2015-07-<>), creating highly specific patterns.

C. Hybrid Encoder

This component applies tailored encoding strategies based on the token type:

Optimized Numeric Encoding: Uses Dynamic Delta Encoding (checking if delta encoding reduces size compared to raw values) and Length-based Optimization (grouping numbers by length before encoding). It also features Combined Column Optimization, concatenating numeric sub-tokens (e.g., [2015, 07, 28] $\to$ 20150728) to capture progression better than column-wise delta encoding.
Dictionary Encoding: Applied to unstructured strings, skeleton patterns, and static token sequences.
Mixed-Type Encoding: A novel strategy for columns containing both numbers and strings. Strings are mapped to unique integer IDs. To allow uniform numeric encoding, a parity-based transformation is applied: original numbers become even integers ( $2n$ ), and string IDs become odd integers ($2id + 1$). This allows the entire column to be compressed using efficient numeric schemes (like Elastic or Delta encoding) while remaining lossless.

D. Packer

Aggregates the dictionaries and binary data files, then applies a general-purpose compressor (e.g., LZMA) to produce the final archive.

3. Key Contributions

Identification of Limitations: The paper highlights the neglect of intra-token patterns in structured tokens and the lack of fine-grained encoding for mixed-type data in existing tools.
LogFold Framework: Proposes a four-component architecture that decomposes structured tokens into skeletons and sub-tokens, enabling deep pattern mining.
Hybrid Encoding Strategy: Introduces a type-aware approach, specifically the Mixed-Type Encoding (parity transformation) and Combined Column Optimization, which significantly enhances compression for complex log data.
Comprehensive Evaluation: Validated on 16 diverse public datasets ranging from supercomputers to mobile systems.

4. Experimental Results

The evaluation was conducted on 16 datasets (totaling >77 GB) against 9 baselines (4 general-purpose, 5 log-specific).

Compression Ratio (CR): LogFold achieved an average CR of 89.519, outperforming the best baseline (Denum, 80.568) by 11.11% and Gzip by 494.19%. It secured the highest compression ratio on 12 out of 16 datasets.
Compression Speed: LogFold achieved an average speed of 9.842 MB/s, which is competitive with other log-specific tools (e.g., LogBlock at 9.636 MB/s) and significantly faster than some high-ratio tools like LogZip.
Ablation Studies: Removing either the Structured Token Processor or the Hybrid Encoder resulted in significant performance drops, confirming that both components are essential.
Sensitivity Analysis: LogFold demonstrated high robustness across various parameter settings (thresholds for critical position identification and re-grouping) and different token selection strategies.
Generalizability: The tool maintained superior performance regardless of the backend general-purpose compressor (Gzip, Bzip2, LZMA) or compression level, effectively eliminating the typical speed-vs-ratio trade-off found in standard compressors.
Decompression: Decompression speed is limited primarily by the restoration of numeric tokens (Step 7 in the pipeline), a necessary trade-off for achieving high compression ratios.

5. Significance

LogFold represents a state-of-the-art advancement in log management. By moving beyond simple template extraction to structured token decomposition and hybrid type-aware encoding, it solves the inefficiencies of previous methods. Its ability to achieve significantly higher compression ratios while maintaining practical processing speeds makes it ideal for long-term archiving, centralized log aggregation, and cost-sensitive storage environments where minimizing storage footprint is critical. The paper also provides a robust framework for handling the diversity of modern log formats, including complex mixed-type tokens that were previously under-optimized.