🤖 AI

Exploiting Neural Audio Codec Latents for Adversarial Audio Attacks

This paper proposes a generative adversarial attack framework that operates in the continuous latent space of a neural audio codec to synthesize targeted perturbations in a single forward pass, achieving up to 99% success rates with sub-7 ms inference latency while significantly outperforming existing methods in both speed and efficiency.

Original authors: Sameek Bhattacharya, Bharath Krishnamurthy, Ajita Rattani

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sameek Bhattacharya, Bharath Krishnamurthy, Ajita Rattani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, high-tech security guard at a door. This guard listens to your voice to decide if you are allowed inside. Usually, this guard is excellent at recognizing who you are. However, researchers have discovered that you can trick this guard with a "magic whisper" that sounds almost exactly like you, but contains a tiny, hidden signal that makes the guard think you are someone else entirely.

This paper introduces a new, super-fast way to create those "magic whispers" to test how vulnerable these voice systems are.

Here is the breakdown of their discovery using simple analogies:

The Problem: The Slow and Clunky Attack

Previously, to trick these voice guards, hackers had to use two main methods, both of which had big flaws:

The "Sculptor" Method (Iterative Attacks): Imagine trying to carve a perfect statue out of a block of marble by chipping away tiny bits, checking your work, chipping again, and repeating this thousands of times. This is how older methods worked. They tweaked the sound wave over and over until it fooled the system. It was very effective, but it took a long time—like trying to sculpt a statue in real-time while the guard is watching. It was too slow for live situations.
The "Instant Artist" Method (Generative Attacks): Imagine a robot that can instantly paint a picture. Newer methods tried to do this: generate the trick sound in one go. However, these robots often produced "bad art"—sounds that were full of static, glitches, or robotic noises that humans could easily hear. Also, the robots were often too heavy and slow to run on a phone or a smart speaker.

The Solution: The "Secret Code" Shortcut

The authors of this paper found a clever shortcut. Instead of trying to hack the raw sound wave (the marble), they decided to hack the secret code the sound is translated into first.

Think of it like this:

Raw Audio: A long, detailed letter written in a complex language.
Neural Audio Codec (The Translator): A device that instantly reads that long letter and summarizes it into a short, 3-word secret code (a "latent" representation) that captures the essence of the message without all the fluff.
The Attack: Instead of rewriting the whole long letter, the researchers built a machine that takes the 3-word secret code, adds a tiny, invisible tweak to it, and then translates it back into a long letter.

Because they are only tweaking the short "secret code" instead of the whole long letter, the process is incredibly fast.

How Their Machine Works

The Translator: They use a pre-trained tool (called a "Neural Audio Codec") that turns sound into these compressed secret codes. This tool is frozen, meaning they don't change it; they just use it as a bridge.
The Tweak Machine: They built a special generator (a type of AI) that looks at the secret code and the "target" they want to trick the system into (e.g., "Make the guard think this is Bob, not Alice").
One-Step Magic: In a single split-second step, this machine adds a tiny perturbation to the secret code.
The Result: The code is turned back into sound. To a human ear, it sounds perfectly normal. To the security guard, it sounds exactly like the target person.

Why This is a Big Deal

The paper claims their method is a game-changer for three reasons:

Speed: It takes less than 7 milliseconds to create a trick sound. That is faster than a camera shutter click. It is roughly 24 times faster than other "instant" methods and nearly 19,000 times faster than the slow "sculptor" methods.
Stealth: Because they are working with the "essence" of the sound (the secret code) rather than the raw noise, the resulting audio is high-quality and doesn't sound glitchy or robotic.
Effectiveness: They tested this on systems that recognize speech commands (like "Hey Siri") and verify speakers (like unlocking a phone with your voice).
- On speech commands, they fooled the system 96% to 99% of the time.
- On speaker verification, they achieved a 100% success rate in tricking the system.

The Bottom Line

The researchers didn't just find a way to break these systems; they found a way to break them instantly and silently.

They are warning that as we put more voice-activated security in our homes and devices, we need to realize that these systems can be fooled in real-time. Their work proves that the "secret code" layer of audio processing is a weak spot that needs to be defended, because right now, a hacker could potentially generate a fake voice command or identity in the blink of an eye.

Note: The authors explicitly state they used AI tools only to polish the English and formatting of the paper. The ideas, experiments, and results were created entirely by the human researchers.

Technical Summary: Exploiting Neural Audio Codec Latents for Adversarial Audio Attacks

Problem Statement
Deep learning-based audio classification systems, including automatic speaker verification (ASV) and environmental sound detection, are vulnerable to adversarial attacks. While traditional optimization-based methods like Projected Gradient Descent (PGD) and Carlini–Wagner (C&W) are effective, they operate in the high-dimensional waveform domain and require costly iterative updates. This computational overhead renders them impractical for real-time or streaming scenarios, which are critical attack vectors for live services. Conversely, existing generative approaches (e.g., VAEs, diffusion models, autoregressive methods) often rely on heavy, task-specific architectures that incur high inference latency or introduce perceptible artifacts, failing to balance efficiency with stealth.

Methodology
The authors propose a white-box, generative adversarial framework that operates entirely within the continuous latent space of a pre-trained neural audio codec, specifically the Descript Audio Codec (DAC). The approach is designed as a fully differentiable, end-to-end pipeline consisting of four key components:

Latent Projection: A frozen DAC encoder maps raw audio waveforms ( $x$ ) into a compressed, continuous latent representation ( $z$ ). This structured manifold preserves acoustic fidelity while reducing dimensionality, allowing for minimal displacements to induce significant semantic shifts.
Conditional Generation: A trainable feed-forward generator ( $G_\theta$ ) synthesizes a targeted perturbation ( $\delta_z$ ) in a single forward pass. The generator is conditioned on the pristine latent $z$ and a target class embedding ( $y_t$ or $v_{tgt}$ ). It outputs a residual perturbation rather than an absolute latent, utilizing a learnable scaling parameter for stability and zero-initialization to ensure negligible initial noise.
Differentiable Decoding: The perturbed latent ( $z_{adv}$ ) is reconstructed into an adversarial waveform ( $x_{adv}$ ) via a frozen DAC decoder. To bridge the gap between the waveform and the victim classifier, a differentiable preprocessor performs Short-Time Fourier Transform (STFT), Mel-filterbank mapping, and logarithmic compression.
Optimization Objective: The framework employs a unified composite loss function combining an adversarial loss (Cross-Entropy for classification, cosine similarity maximization for speaker verification) with a margin loss to widen decision boundaries and an $L_2$ regularization term to bound perturbations. To stabilize the non-convex optimization landscape, the generator weights are maintained using an Exponential Moving Average (EMA).

Key Contributions

Latent-Space Generative Attack: The paper introduces a trainable conditional generator that manipulates high-level compressed acoustic representations within the DAC latent space, avoiding direct raw waveform perturbation.
Fully Differentiable Codec Pipeline: By operating in the codec's pre-quantization space, the method overcomes the non-differentiability of discrete codebooks, enabling direct gradient-based optimization of latent perturbations.
Real-Time Generation: The framework generates adversarial examples in a single forward pass with sub-7 ms inference time per sample. This represents a speedup of up to 24× over generative baselines and 18,900× over iterative methods like C&W.
High and Consistent Attack Success: The method achieves targeted Attack Success Rates (ASR) up to 99% across CNN- and transformer-based audio models, demonstrating strong architectural generality.

Results
The proposed method was evaluated on four benchmark datasets: Google Speech Commands, TAU Urban Acoustic Scenes 2019, UrbanSound8k, and LibriSpeech (for speaker verification).

Speech Commands: The method achieved an untargeted ASR of 96.58% and a targeted ASR of 77.65%, outperforming gradient-based methods (FGSM, PGD, C&W) and generative baselines (FAPG, CGAN) in both efficiency and effectiveness.
Environmental Sound Classification: On UrbanSound8k and DCASE2019, the approach achieved untargeted ASRs of 99.11% and 100%, respectively. In targeted settings, it achieved ASRs of 97.17% and 94.07%, significantly outperforming iterative baselines while maintaining the lowest latency (0.0039–0.0056 s).
Speaker Verification: On LibriSpeech, the method achieved 100% untargeted ASR and 99.80% targeted ASR against an ECAPA-TDNN model. It was 23.8× faster than the FAPG baseline and orders of magnitude faster than iterative methods.

Significance and Claims
The paper claims to address a critical gap in evaluating real-time, highly efficient generative attacks against audio classification and biometric systems. By leveraging the continuous latent manifold of a general-purpose neural audio codec, the proposed framework combines the stealth and structural fidelity of latent-space manipulation with ultra-low latency. The authors assert that compressed semantic latent manifolds represent a powerful yet previously unexplored adversarial surface, enabling high attack success rates without the prohibitive computational costs of iterative optimization. This viability suggests that real-time streaming audio systems are more susceptible to such attacks than previously demonstrated by existing methods. Future work is noted to investigate black-box transferability and latent-space defenses.