`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMs

In the last few years, artificial intelligence has learned to speak. These systems, known as large language models, have moved beyond reading and writing text to understanding and generating human speech. They power the voice assistants in our cars, the customer service bots on our phones, and the new conversational agents that can listen to a story and answer questions about it. To make these machines safe, developers build digital guardrails designed to refuse harmful requests, such as instructions on how to build a weapon or how to harass someone. For text-based systems, researchers have long known that cleverly worded questions can trick these guardrails into breaking their rules. However, as these models gain the ability to hear and speak, a new question arises: can a voice, rather than just words, bypass these same safety measures?

A team of researchers from the University of Sydney, the University of Chicago, and the University of Texas at San Antonio set out to answer this question by testing the security of modern voice-enabled AI. They focused on two different ways these systems are built. The first method, called a cascaded pipeline, works like a relay race: a computer first listens to the voice and writes it down as text, then a separate text-based brain reads that text and decides what to say, and finally, a voice synthesizer speaks the answer. The second method, known as an end-to-end model, skips the middle step entirely. It listens to the raw sound waves and processes the meaning and the voice together in a single, unified system. The researchers wanted to know if both of these very different architectures could be tricked into ignoring their safety rules when spoken to, and if so, how to do it automatically without human help.

The researchers developed a new testing framework that acts like an automated explorer. Instead of manually crafting hundreds of different voice commands, they built a system that learns from its own mistakes. The process begins with a list of harmful questions that the AI should never answer. The system first wraps these dangerous questions inside a harmless story or a role-playing scenario, a technique they call a "flanking attack." By hiding the harmful request inside a long list of innocent questions about daily life or fictional movie plots, the system hopes to lower the AI's defenses. Once the text is prepared, the system converts it into speech using a standard voice synthesizer.

This is where the real experimentation begins. The system does not just play the speech once. It takes the audio and deliberately distorts it in specific ways to see how the AI reacts. It adds background noise, like the hum of a busy street or the sound of wind. It changes the speed of the voice, making it speak slightly faster or slower. It even adds echoes, simulating the sound of a voice bouncing off the walls of a large room. The system then plays these distorted audio clips to the target AI. If the AI refuses to answer, the system notes the failure. If the AI accidentally agrees to the harmful request, the system records the success.

Crucially, the system uses these results to improve its next attempt. If adding noise helped the attack succeed, the system learns to use more noise. If changing the speed worked better, it focuses on that. It also rewrites the text of the question, trying different ways to say the same thing without changing the meaning. This cycle of testing, learning, and refining happens automatically and repeatedly. The researchers tested this approach on six different voice-enabled AI systems, ranging from well-known commercial products to open-source models. They ran thousands of these automated tests, covering twenty-one different categories of harmful requests, from fraud to illegal activities.

The results were striking. The researchers found that both types of systems—the relay-race style and the unified end-to-end style—were highly vulnerable to these voice-based attacks. The automated framework was able to trick the AI into violating its safety rules in the vast majority of cases. For the most vulnerable system tested, the attack succeeded more than ninety-six percent of the time. Even the systems that were considered more robust still failed to block the attacks in over seventy-five percent of the trials. The study showed that the combination of hiding the harmful intent in a story and then distorting the voice was far more effective than trying to trick the AI with just the words or just the sound alone.

The researchers also tested what happens when the attack is not played through a computer file, but spoken out loud in a real room and recorded by a microphone. This is known as an over-the-air attack, and it introduces real-world problems like background chatter and the quality of the microphone. Despite these extra hurdles, the automated framework remained effective. The systems still fell for the tricks, with success rates remaining high even when the audio had to travel through the air and be captured by a device. This suggests that the vulnerability is not just a quirk of digital files but a fundamental weakness in how these models process spoken language.

The study also looked at which specific tricks worked best. They found that the most successful attacks often involved changing the wording of the question to be more complex or adding realistic background noise. Surprisingly, simply speeding up or slowing down the voice was less effective on its own, but it helped when combined with other changes. The system learned that the best way to break the guardrails was to keep the harmful meaning intact while making the sound and the words look different enough to confuse the safety filters.

This work highlights a significant gap in the current safety of voice AI. While developers have spent years hardening text-based systems against clever prompts, the addition of voice has opened a new door that remains largely unguarded. The researchers demonstrated that an automated system can systematically find and exploit these weaknesses without needing to know the internal code of the AI. They did not find a single magic bullet that breaks every system, but rather showed that a flexible, learning approach can consistently bypass the protections in place today. The findings suggest that as voice interfaces become more common, the safety measures for these systems need to evolve to handle not just what is said, but how it is said and how it sounds. The study concludes that without these improvements, the promise of safe, conversational AI may remain out of reach.