Do Machine Learning Security Detectors Survive New Attacks? A Leakage Free Evaluation of Hidden Prompt Injection Detection in AI Coding Agents, Malicious npm and PyPI Package Detection, and Network Intrusion Detection
This paper demonstrates that machine learning security detectors, which typically achieve near-perfect performance on standard random splits, suffer catastrophic failure rates when evaluated against unseen attack techniques and temporal shifts using a rigorous, leakage-free protocol across AI prompt injection, malicious package, and network intrusion detection scenarios.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital world, security systems often rely on machine learning, a type of computer program that learns to spot trouble by studying thousands of examples of past attacks. These programs act like digital guards, scanning network traffic, checking software files before they are installed, and even reading the instructions that artificial intelligence coding assistants use to help programmers. For years, the standard way to test these guards has been to shuffle a pile of known attacks and split them into two groups: one for training and one for testing. This method assumes that the attacks the guard will face in the future will look very much like the ones it has already studied. However, real-world attackers do not stay still; they constantly invent new tricks, hide their tools in different ways, and strike at times and places their defenders have never seen. The critical question is whether these digital guards can actually recognize a new kind of threat, or if they are simply memorizing the old ones and failing when the game changes.
A team of researchers from the University of Port Harcourt set out to answer this question by testing security detectors against three very different types of digital threats using a single, rigorous method designed to prevent data leakage. They examined how well detectors could spot hidden instructions planted in files for AI coding agents, how they fared against malicious software packages in online libraries, and how they performed when identifying network intrusions. Instead of shuffling data randomly, the researchers forced the detectors to face situations they had never encountered: attacks using new hiding techniques, attacks appearing in a later year, or attacks from entirely different datasets. They also checked for "shortcuts," where a detector might be relying on a simple, obvious clue—like the size of a file—rather than actually understanding the danger.
The first part of their study focused on AI coding agents, which are tools that help programmers write code by reading files like configuration guides. Attackers have begun hiding secret instructions inside these files using invisible characters or strange symbols that a human eye would miss but a computer reads as a command. The researchers created a collection of nearly five thousand real files from public repositories and then planted harmless hidden instructions into them using nine different hiding techniques. They tested five different detectors to see if they could find these traps. When the detectors were tested on the same hiding techniques they had learned during training, they performed almost perfectly, catching nearly every single hidden instruction. However, when the researchers tested them on a hiding technique the detector had never seen before, the results collapsed for some models. A detector that relied on counting specific features, like the number of invisible characters, missed ninety-six percent of the new attacks. In contrast, a detector that read the actual text of the file, after first converting the hidden characters into plain, readable text, maintained its high performance. The study also found that small, open-source AI models often followed these hidden instructions, obeying the secret commands in up to two-thirds of the cases, while simply deleting the invisible characters sometimes made the attack even easier for the AI to understand.
The second study looked at malicious software packages in the npm and PyPI libraries, which are vast online stores where developers download code. The researchers gathered over thirteen thousand packages, including thousands of known malicious ones from 2024, 2025, and 2026. They trained detectors on older packages and then tested them on the new waves of attacks from 2026. One detector, which looked at the code itself, remained strong, catching over ninety percent of the new malicious packages. Another detector, which relied on static features like the total size of the package and the number of files inside it, performed well at first but failed dramatically when the researchers removed the size-based clues. Without relying on the fact that malicious packages were often small, this detector's ability to spot new attacks dropped significantly, revealing that it had been relying on a shortcut rather than truly identifying the malware. The study also showed that detectors struggled to find malicious code that had been slipped into legitimate, popular libraries, a technique that grew more common in 2026.
The third study revisited network intrusion detection, which involves scanning data flowing through a computer network to spot hackers. The researchers used three widely used datasets that have served as the standard for testing these systems for years. They found that these datasets contained a high number of duplicate entries and errors, which had likely inflated previous success rates. When they tested the detectors by training them on data from earlier days and testing them on data from later days, the performance of the systems plummeted. A detector that looked perfect on a random shuffle of the data performed no better than a coin flip when faced with the traffic from the following week. It failed to recognize entire families of attacks that it had never seen before, such as specific types of botnet traffic or infiltration attempts, proving that the system had learned to recognize the specific patterns of the past rather than the nature of the attack itself.
Across all three areas of security, a clear pattern emerged. Detectors that relied on summary features or simple counts tended to fail when the attackers changed their methods, while detectors that read the actual content of the threat held up much better. The researchers found that a simple step of converting hidden or confusing text into a standard, readable format before scanning it allowed the detectors to recognize the underlying message regardless of how it was disguised. They also demonstrated that removing duplicates from the data was necessary but not enough to fix the problem; the real issue was the assumption that future attacks would look like past ones. The study concludes that security systems must be tested against new, unseen techniques before they are trusted, and that relying on standard, random testing methods gives a false sense of security. The work suggests that to stay safe, we must design systems that understand the content of an attack rather than just memorizing its appearance.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.