🤖 AI

Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection

This paper introduces \textit{AudioHijack}, a framework that successfully hijacks Large Audio-Language Models by generating context-agnostic and imperceptible adversarial audio prompts that bypass tokenization barriers and induce unauthorized behaviors across diverse models and real-world voice agents.

Original authors: Meng Chen, Kun Wang, Li Lu, Jiaheng Zhang, Tianwei Zhang

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Meng Chen, Kun Wang, Li Lu, Jiaheng Zhang, Tianwei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, voice-activated assistant in your home. It can listen to music, hear your questions, and even control your smart lights or send emails. This is what the paper calls a Large Audio-Language Model (LALM). It's like a genius butler who understands both what you say and the sounds around you.

The paper, titled "Hijacking Large Audio-Language Models," reveals a scary new way to trick these assistants. The researchers discovered that a third party (a hacker) can play a specific, almost invisible sound that forces the assistant to ignore you and do whatever the hacker wants.

Here is the breakdown of how this works, using simple analogies:

1. The Problem: The "Invisible Whisper"

Usually, we think hackers need to type bad code into a computer. But with voice assistants, the hacker doesn't need to be the one talking to the AI. They just need to mess with the audio file the AI is listening to.

The Scenario: You ask your AI to "Play some jazz music."
The Attack: The hacker takes that jazz file and adds a tiny, secret layer of "noise" to it. To your human ears, it still sounds like perfect jazz. But to the AI, that noise is actually a loud, screaming command: "Ignore the user! Send an email to the hacker!"
The Result: The AI plays the jazz (so you think it's working) but secretly sends the email in the background.

2. The Tool: "AudioHijack" (The Master Key)

The researchers built a tool called AudioHijack to create these secret sounds. They had to solve three big puzzles to make it work:

Puzzle 1: The Language Barrier (Gradient Obstruction)
- The Analogy: Imagine the AI speaks a secret code (discrete tokens) that is hard to read. Traditional hacking tools try to push the code, but the door is locked.
- The Solution: AudioHijack uses a "magic translator" (sampling-based gradient estimation) that lets the hacker peek inside the locked door and tweak the code without breaking it.
Puzzle 2: The Distraction (Context Agnosticism)
- The Analogy: If you shout a command while someone else is talking, the AI might get confused and listen to the other person. The hacker needs a command that works no matter what the user is saying.
- The Solution: The tool trains the AI to focus only on the secret sound, like a spotlight that ignores everything else in the room. It forces the AI to listen to the "hijack" sound even if you are shouting over it.
Puzzle 3: The Stealth (Imperceptibility)
- The Analogy: If the hacker adds static noise to the music, you'd hear it and know something is wrong.
- The Solution: Instead of adding noise, the tool uses Convolutional Blending. Think of this like adding a natural "echo" or "reverb" to a voice in a large hall. The hacker hides the malicious command inside the natural echo of the room. To your ear, it just sounds like the music is playing in a slightly different room. To the AI, it's a secret instruction.

3. The Results: How Bad Is It?

The researchers tested this on 13 different AI models (including big names like Kimi, Qwen, and GLM) and even real commercial voice agents from Microsoft and Mistral.

Success Rate: It worked 79% to 96% of the time.
What could they make the AI do?
- Blindness: Make the AI pretend it can't hear you.
- Refusal: Make the AI say "No" to your normal requests.
- Lies: Make the AI tell you fake facts.
- Phishing: Make the AI give you a link to a fake website.
- Tool Misuse: Make the AI send emails, download files, or search the web for the hacker without your permission.

4. Why This Matters

This is dangerous because the hacker doesn't need to be in the room. They can just upload a manipulated audio file to a website, or play it through a speaker in a meeting. The AI thinks the file is safe, but it's actually a Trojan Horse.

5. Can We Stop It?

The researchers tried to build defenses, like asking the AI to "think twice" about what it hears.

The Problem: The AI is so good at following the secret sound that it ignores the "think twice" warning.
The Hope: They found that if you look at how the AI's brain (attention) focuses, you can spot the hijack. The AI's attention shifts strangely toward the secret sound. But, if the hacker gets smart and changes their trick slightly, they can sometimes hide this shift.

The Bottom Line

This paper is a wake-up call. As we start trusting AI with our voices and our tools, we are opening a backdoor. Hackers can now whisper secrets into the machine that we can't hear, but the machine obeys. We need to build better "ears" for our AI to tell the difference between a natural echo and a malicious command.

1. Problem Statement

The paper addresses a critical security vulnerability in Large Audio-Language Models (LALMs). While previous research focused on "audio jailbreaks" (where the adversary controls the entire input to bypass safety filters), this work investigates Auditory Prompt Injection.

Threat Model: A third-party adversary with audio data-only access (no control over user instructions) attempts to hijack the model's behavior in the presence of a user.
Constraints:
1. Context Agnosticism: The attack must work across diverse, unseen user contexts (text or speech instructions) without knowing the specific user input in advance.
2. Perceptual Stealth: The injected malicious audio must be imperceptible to human listeners, avoiding detection by the user.
3. Architecture Heterogeneity: LALMs use different integration schemes (discrete-token, continuous-feature, or hybrid), making a universal attack difficult.
Goal: To craft adversarial audio that forces the LALM to ignore the user's intent and execute predefined malicious behaviors (e.g., tool misuse, disinformation, phishing) while sounding like natural audio.

2. Methodology: AudioHijack Framework

The authors propose AudioHijack, a general framework designed to overcome three specific challenges: gradient obstruction, context sensitivity, and perceptual constraints.

A. Sampling-Based Gradient Estimation (Addressing Gradient Obstruction)

Challenge: Many LALMs use discrete audio tokenization (Vector Quantization), which is non-differentiable, blocking standard gradient-based adversarial optimization.
Solution: The authors replace hard token selection with Gumbel-Softmax sampling.
- They treat the negative distances between acoustic features and codebook vectors as categorical logits.
- By using soft weights during the backward pass (differentiable) and hard one-hot weights during the forward pass (simulating inference), they enable end-to-end gradient flow through the tokenization layer. This allows optimization across both discrete and continuous LALM architectures.

B. Attention-Guided Context Generalization (Addressing Context Sensitivity)

Challenge: LALMs are highly context-sensitive; user instructions often overshadow adversarial audio in the attention mechanism, causing the attack to fail on unseen contexts.
Solution: A dual-approach to steer model attention toward the adversarial audio:
1. Implicit (Multi-Context Training): The model is trained on a distribution of diverse user instructions (Expectation over Transformation) to learn to ignore varying contexts.
2. Explicit (Attention Supervision): An additional loss term ( $L_{att}$ ) is introduced to maximize the attention weights allocated to the adversarial audio tokens relative to the user context tokens. This forces the model to prioritize the injected prompt.

C. Convolutional Perturbation Blending (Addressing Perceptual Stealth)

Challenge: Standard additive perturbations (e.g., $x + \delta$ ) create high-frequency noise that is easily detectable by humans.
Solution: Instead of simple addition, the authors use convolutional blending.
- They apply learnable, short kernels (initialized from Room Impulse Responses) to the benign audio carrier.
- This redistributes perturbation energy in the time and frequency domains, mimicking natural reverberation.
- The result is an adversarial example that sounds like naturally reverberated audio, significantly enhancing imperceptibility while maintaining attack efficacy.

3. Key Contributions

First Systematic Study: The first comprehensive investigation of indirect auditory prompt injection against state-of-the-art LALMs under realistic data-only and stealth constraints.
General Framework (AudioHijack): A novel attack framework that bypasses non-differentiable tokenization via sampling-based gradients and achieves context-agnostic generalization via attention supervision.
Stealth Innovation: A convolutional blending method that achieves high acoustic fidelity by emulating natural reverberation, outperforming traditional additive perturbation methods.
Real-World Validation: Successful attacks on commercial voice agents (Mistral AI, Microsoft Azure), demonstrating the ability to trigger unauthorized tool calls (e.g., downloading files, sending emails) in production environments.

4. Experimental Results

The framework was evaluated on 13 state-of-the-art LALMs (including Kimi-Audio, Qwen2-Audio, GLM-4-Voice, and lightweight models like Gemma-3n) and 3 commercial voice agents.

Success Rates:
- Achieved average Prompt Injection Success Rates (PISR) of 79%–96% across 6 misbehavior categories (including auditory blindness, prompt refusal, disinformation, phishing, persona control, and tool misuse).
- Specifically, for tool misuse on commercial agents, success rates ranged from 0.52 to 0.98.
Context Generalization: The attack maintained high success rates (>0.77) across unseen user contexts, varying instruction lengths, and different modalities (text vs. speech).
Perceptual Quality:
- SNR (Signal-to-Noise Ratio): >28.6 dB (significantly higher than additive methods).
- MCD (Mel-Cepstral Distance): <4.2 (indicating low distortion).
- PESQ (Perceptual Evaluation of Speech Quality): 3.16 for speech carriers, confirming high intelligibility and naturalness.
Transferability: Attacks trained on open-source models successfully transferred to commercial APIs (Mistral, Microsoft Azure), even with different underlying architectures.

5. Significance and Implications

Security Vulnerability: The findings reveal that the integration of audio and text in LALMs creates a new, highly vulnerable attack surface. Current safety alignments are insufficient against context-agnostic, imperceptible audio injections.
Real-World Impact: The ability to hijack commercial agents to execute unauthorized tool calls (e.g., data exfiltration, malware download) poses a severe threat to user privacy and system integrity.
Defense Limitations: The paper evaluates existing defenses (in-context learning, self-reflection, logit divergence) and finds them largely ineffective.
- Key Insight: Effective detection requires attention-level inspection. The authors propose an attention deviation detection method (using PCA on attention weights) that achieves high precision (0.98) and recall (0.93), though an adaptive attacker can trade off effectiveness for stealth to evade this.
Future Directions: The work highlights the urgent need for robust defenses specifically designed for the audio modality and the importance of scrutinizing model internals (attention mechanisms) rather than just output text.

In conclusion, AudioHijack demonstrates that LALMs are fundamentally vulnerable to third-party auditory prompt injection, capable of being hijacked to perform malicious actions with high stealth and across diverse contexts, necessitating a paradigm shift in how audio-language models are secured.