Evaluation Pitfalls and Multimodal Baselines for the \dataset{} IoT Malware Dataset

In the digital world, security researchers act like immune system doctors for computers. They build tools to spot malicious software, or malware, before it can cause harm. To train these tools, they need vast libraries of data: recordings of how computers behave when they are healthy, and recordings of how they behave when they are sick. The quality of the training data determines how well the resulting security tools work. If the data is flawed, the tools will fail in the real world, often missing dangerous threats or raising false alarms. A major challenge in this field is ensuring that the way researchers split their data into training and testing groups does not accidentally introduce bias. If a computer program sees a specific piece of malware during training and then sees the exact same piece during testing, it is not really learning to detect new threats; it is simply memorizing the answer. Furthermore, researchers must decide whether to judge a security tool by how well it catches individual moments of suspicious activity, or by how well it identifies entire infected programs, which can be a very different task.

A team of researchers from Tarim University recently turned their attention to a new and unusually detailed library of data called CIC-YNU-IoTMal2026. This dataset is a treasure trove for security experts because it records three different types of information simultaneously for thousands of computer programs. It captures the network traffic flowing in and out, the system activity like CPU usage, and the specific list of commands the program gives to the operating system. These programs were run on four different types of computer chips to see if the security tools could work across different hardware. The researchers did not build a new security tool; instead, they acted as auditors. They examined the dataset itself to see if the standard ways of testing security tools on this data were hiding serious problems. They wanted to know if the high scores reported by other researchers were real or if they were the result of subtle mistakes in how the data was handled.

The audit began by looking at how the data was divided. The most common method is to take all the recorded moments of activity and shuffle them randomly, putting some into a training pile and some into a testing pile. The researchers found that this approach was fundamentally flawed for this specific dataset. Because the data records entire programs, shuffling the moments meant that the testing pile contained moments from programs that were already in the training pile. In fact, every single test moment shared its parent program with the training set, and a small percentage of the test moments were exact duplicates of training moments. This meant that any security tool tested this way was essentially being graded on questions it had already seen the answers to. While this did not artificially inflate the scores on this particular dataset, it meant the results could not be trusted to predict how well a tool would work on truly new, unseen programs. The researchers concluded that the only fair way to test these tools is to keep entire programs together, ensuring that no program appears in both the training and testing groups.

Even when the researchers fixed the data splitting method, they discovered a second, more deceptive problem. The dataset contains thousands of tiny snapshots of activity, and many security tools are judged by how well they catch these individual snapshots. The researchers found that this method hides a dangerous failure mode. About sixteen percent of the malicious programs in the dataset were completely missed by the security tools. These missed programs were not sophisticated; they were simply dormant. They had been executed in the testing environment but had failed to wake up and start their malicious activities, so they looked exactly like harmless programs. Because the malicious programs that did wake up were very loud and generated hundreds of activity snapshots, they dominated the statistics. The quiet, dormant programs were drowned out, making the security tool look much better than it actually was. The researchers emphasized that to get a true picture of safety, one must judge the tool on whether it catches the entire program, not just the noisy moments.

The study also investigated how the tools performed when faced with different types of computer chips. The dataset included programs running on four different architectures, and the researchers tested whether a tool trained on three of them could detect threats on the fourth. The results were stark. When the tool was tested on a specific type of chip called ARM, which is common in many small devices, it collapsed. The tool became so unreliable that it flagged harmless programs as dangerous nearly eighty percent of the time, while still catching almost all the actual malware. This failure was not because the tool could not recognize the malware; it was because the harmless programs on the ARM chip behaved differently than harmless programs on the other chips. The tool had learned the wrong patterns for what "normal" looked like on that specific hardware. The researchers found that this problem could be fixed very cheaply. By showing the tool just a tiny fraction of the new chip's harmless programs—less than one percent of the total data—the tool's performance instantly recovered to near perfection. This suggested that the solution to cross-chip security is not complex new algorithms, but simply exposing the tool to a few examples of the new environment.

Finally, the researchers looked at the different types of data available: network traffic, system activity, and command traces. They found that in a controlled environment, combining all three types of data made the security tools nearly perfect. However, this perfection was fragile. When the tools were forced to rely only on network traffic, they missed the dormant programs. When they relied only on the command traces, they failed completely when the chip architecture changed. The most robust combination turned out to be a mix of network traffic and system activity, which avoided the specific weaknesses of the other methods. The researchers also noted that the specific type of mathematical model used to build the tool mattered very little. Whether they used a simple linear model or a complex neural network, the results were almost identical. This proved that the biggest factor in success was not the sophistication of the tool, but the choices made about how to split the data, how to define a success, and how to handle the different types of computer chips.

The paper concludes with a set of clear guidelines for anyone using this dataset. They recommend always keeping entire programs together when splitting data, reporting results based on whether whole programs are caught rather than just individual moments, and being transparent about how unknown or unclassified programs are treated. They also urge researchers to test their tools on different computer chips and to report how much the results vary if the experiment is run multiple times. The most important takeaway is that the path to better security does not lie in building more complex models, but in asking better questions and using cleaner data. By fixing the evaluation methods, the community can ensure that the tools they build will actually work when they are needed most.