A Multi-Seed Evaluation of Explanation Faithfulness and Stability under Adversarial Evasion in IoT/IIoT Intrusion Detection

In the hidden layers of the modern world, billions of smart devices—from factory sensors to home thermostats—constantly talk to one another. This vast network, known as the Internet of Things, is the nervous system of our digital infrastructure. But like any nervous system, it is vulnerable to interference. Hackers can send subtle, calculated signals designed to trick the security software watching these devices, making a dangerous attack look like harmless background noise. For years, security experts have relied on artificial intelligence to spot these intruders. However, a new problem has emerged: when these AI systems flag a threat, they often provide an explanation of why they made that call. The critical question is whether that explanation remains honest when the AI is being actively tricked. If a hacker can fool the system into ignoring an attack, does the system's explanation also become a lie, or does it still point to the truth?

A team of researchers from Daffodil International University set out to test the honesty and reliability of these AI explanations under fire. They focused on two popular methods used to explain AI decisions, known as SHAP and LIME. These tools act like a spotlight, highlighting which parts of a network signal—such as the size of a data packet or the duration of a connection—caused the AI to sound an alarm. The researchers wanted to know if these spotlights would stay steady and accurate when the input data was subtly altered by a hacker, or if the explanations would wobble and point in the wrong direction. To get a truly reliable answer, they avoided the common mistake of testing just one version of an AI model. Instead, they trained twenty different versions of the same security system, each starting with a slightly different random setup, to ensure their findings were not just a lucky accident of one specific training session.

The researchers put these systems through a rigorous series of tests using real-world data from two major collections of network traffic. They simulated three types of attacks: a broad, unrestricted attack that changes data in ways a real network could never produce; a more realistic attack that only changes numbers the attacker can control; and a highly constrained attack that respects all the physical rules of how data packets are built. They found that the unrestricted attacks, which often produce impossible data values like a packet with a fractional number of bytes, were mostly a false alarm. When the researchers forced these impossible records to look like real, valid data, the success rate of the attacks against the neural network models dropped dramatically. However, the tree-based models, which work differently, were more vulnerable to these valid-looking tricks.

The most striking discovery concerned the explanations themselves. When the AI models were attacked, the SHAP method continued to provide faithful and stable explanations, consistently reflecting the model's behavior even when the decision changed. In contrast, the LIME method proved far less reliable. Under attack, LIME's explanations often shifted as much as they did when the system was simply running normally, suggesting that its changes were mostly random noise rather than a true reflection of the attack. In fact, in every single comparison across twenty different training sessions, SHAP outperformed LIME in accuracy and stability. The researchers also found that the choice of a security system's "alarm threshold"—the point at which it decides to sound the alert—matters more than the model itself. A system tuned to be very sensitive might catch almost every attack but generate so many false alarms that security teams are overwhelmed, while a stricter setting might miss attacks but keep the noise down.

Perhaps the most sobering finding was that no single security system is invincible. On one of the datasets, even the most advanced neural networks could be almost entirely fooled by a hacker who simply added extra data padding to a packet, a technique that costs very little but bypasses the defense entirely, with the models detecting at most 3% of such attacks. The tree-based models were more robust, but a clever, targeted attack designed specifically for their structure could still evade them in a significant portion of cases. The study concludes that while we can build AI systems that explain their own reasoning, those explanations are only as trustworthy as the system's ability to resist manipulation. The research provides a clear roadmap for how to test these systems in the future, emphasizing that we must look beyond simple accuracy scores and examine how explanations hold up when the system is under genuine pressure. The tools we use to understand our digital defenses must be as resilient as the defenses themselves.