How Far Does Entropy Go? Duplicates, Confounding, Drift and Padding in Entropy-Based Detection of Malicious PE Files
This paper demonstrates that entropy-based detection of malicious PE files is severely compromised by data duplicates, size confounding, and adversarial padding, revealing that such models largely memorize known sections and can be easily evaded by appending benign-looking bytes, thereby necessitating more robust detection strategies.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, security experts often rely on a simple mathematical idea called entropy to spot dangerous software. Imagine a file as a long string of numbers. If those numbers follow a predictable pattern, like a story written in plain English or a standard program, the string has low entropy; it is orderly and easy to describe. But if the numbers are scrambled, like a message locked in a safe or a file compressed to save space, the string has high entropy; it looks random and chaotic. Because malicious programs often hide their true nature by scrambling their code, security tools frequently scan for this high level of randomness. If a file looks too chaotic, the tool flags it as suspicious. This method is fast, cheap, and widely used to sort through millions of files, deciding which ones need a closer look and which ones can be ignored.
However, a new study by researchers at the University of Florence and the University of Pisa asks a critical question: how much can we really trust this measure of chaos? The team investigated whether the randomness of a specific part of a Windows executable file—the first section of its code—is actually a reliable sign of danger, or if it is a trick that can be easily fooled. They analyzed nearly 43,000 file headers and almost 800,000 labeled samples from a major security dataset. Their work reveals that the current reliance on this single measure is far more fragile than previously thought, often mistaking repetition for danger and failing completely when faced with new, unseen threats.
The researchers began by looking at a hidden flaw in how these security tools are tested. They discovered that in both of their datasets, about 64 percent of the malicious files shared their first section of code with at least one other file. This means that many different malicious programs were built from the exact same scrambled template. When security tools are tested using random shuffling, the same template often appears in both the training group and the test group. The tool isn't actually learning to recognize danger; it is simply memorizing that specific template. When the researchers corrected for this by grouping identical files together, the tool's ability to detect new threats dropped significantly. On a test designed to mimic real-world conditions where the tool sees files it has never encountered before, the entropy-based detector became almost useless, catching less than one percent of the malware at a safe error rate.
The study also uncovered that the tools were often relying on the size of the file rather than its chaos. In one dataset, malicious files happened to be larger than safe files, so the detector learned to flag large files as dangerous. But when the researchers tested this on a different, larger dataset, this size difference disappeared, and the tool's performance collapsed. This proved that the detector was not finding a universal rule about malware but was instead exploiting a coincidence specific to one collection of data. Furthermore, they found that files with empty first sections—a common sign of packing software used to hide code—were frequently found in safe programs as well, meaning this feature alone cannot distinguish between a virus and a legitimate application.
Perhaps the most striking finding concerns how easily these detectors can be bypassed. The researchers calculated the mathematical limits of what happens when an attacker adds extra, harmless bytes to the end of a file's first section. They found that an attacker could almost always add enough padding to change the file's appearance just enough to trick the detector, without actually breaking the program. If the attacker is allowed to add a large amount of padding, they can guarantee that the detector will miss the threat in over 90 percent of cases. Even if the attacker tries to be subtle and only adds a small amount of padding, they can still evade detection in a significant number of cases, especially if they choose the padding carefully to look like normal, safe data.
The researchers also explored whether they could build a detector that is immune to these tricks. They found that a model based on a strict mathematical rule—that the total amount of information in a file cannot decrease when bytes are added—would indeed be impossible to fool by adding padding. However, this "unhackable" model was so cautious that it failed to detect almost any malware at all. It was a perfect defense against evasion, but a terrible tool for finding threats. The study concludes that while entropy is a useful clue, relying on it alone is a mistake. The signal it provides is often a reflection of repeated code or file size rather than true malice, and it can be removed with simple, predictable changes. For security to be effective, experts must look beyond this single measure and combine it with many other factors to see the full picture.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.