🤖 AI

Feature-Aligned Speech Watermarking for Robustness to Reconstruction Distortions

This paper proposes a feature-aligned speech watermarking method that leverages a pretrained speech codec to embed high-energy watermarks within voiced regions, thereby overcoming the traditional fidelity-robustness trade-off to achieve imperceptible audio that remains robust against both seen and unseen speech reconstruction models.

Original authors: Haiyun Li, Shuhai Peng, Zhisheng Zhang, Jingran Xie, Xiaofeng Xie, Hanyang Peng, Zhiyong Wu

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haiyun Li, Shuhai Peng, Zhisheng Zhang, Jingran Xie, Xiaofeng Xie, Hanyang Peng, Zhiyong Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to hide a secret message inside a song so that only you can find it later, but you don't want anyone listening to the song to notice that the message is there. This is called audio watermarking.

For a long time, the rule for hiding these messages has been: "Keep the message tiny and quiet." If the message is too loud, it sounds like static or noise, ruining the song. But here's the problem: modern technology (like AI tools that clean up bad audio or compress it for phone calls) acts like a powerful vacuum cleaner. It sucks out all the "quiet" stuff, including your tiny secret message, leaving you with nothing.

The paper you shared introduces a clever new way to hide messages that solves this problem. Here is how it works, explained simply:

The Old Way: The "Whisper in a Storm"

Traditional methods try to hide the message by whispering it very quietly into the audio.

The Problem: If you whisper too loudly, people hear it (bad quality). If you whisper too quietly, the "vacuum cleaners" (AI reconstruction tools) wipe it out completely.
The Result: You have to choose between a message that is safe but invisible, or a message that is loud but gets deleted.

The New Way: The "Ghost in the Machine"

The authors of this paper decided to stop trying to hide the message under the music and instead make the message part of the music's natural structure.

1. The "Pseudo-Speech" Trick
Instead of just adding random noise, the system uses a smart AI (a "speech codec") to generate a fake voice that sounds exactly like the original singer. Think of this as a ghostly twin of the original audio.

This twin carries the secret message.
Because the twin is made of the same "stuff" as the original voice, it fits perfectly into the song's natural rhythm and tone. It doesn't sound like an intruder; it sounds like part of the band.

2. Hiding in the "Voice" Zones
The system is very smart about where it hides the message. It knows that human voices have "voiced" parts (like singing or talking) and "silent" parts (breaths or pauses).

The Analogy: Imagine trying to hide a sticker on a moving car. If you put it on the spinning wheels, it might fly off. If you put it on the solid door, it stays.
This method only sticks the secret message onto the "solid door" parts of the audio (the voiced regions where the human voice is active). It avoids the quiet, empty spaces where the AI vacuum cleaners are most likely to sweep it away.

3. The "Feature-Aligned" Blend
The system mixes this "ghost twin" with the original song using a special recipe. Because the ghost twin is built to match the original voice's features, the mixture sounds natural to human ears, even though it carries a stronger, more robust message.

Why This Matters (The Results)

The paper tested this new method against the old ones:

Against AI Cleaners: When they ran the watermarked audio through AI tools that clean noise or compress files, the old methods lost their messages almost entirely. The new method kept the message safe, even against tools it had never seen before.
Human Ears: Despite carrying a stronger message, human listeners couldn't tell the difference between the original song and the watermarked one. In fact, it sounded just as good as the best existing methods.

The Bottom Line

The authors solved the "Robustness vs. Quality" trade-off.

Old Logic: To be safe, you must be quiet. To be loud, you must be noisy.
New Logic: If you make the message look and sound exactly like the voice itself, you can make it louder (safer) without anyone noticing it's there.

It's like hiding a message in a letter by writing it in the same ink and handwriting as the rest of the letter, rather than trying to hide a tiny note in the corner. The "vacuum cleaner" can't suck it out because it looks like part of the letter itself.

Technical Summary: Feature-Aligned Speech Watermarking for Robustness to Reconstruction Distortions

1. Problem Statement

Audio watermarking is essential for content attribution and copyright protection, yet existing methods face a critical limitation in modern speech applications. Current approaches prioritize imperceptibility by employing high-fidelity, low-energy designs. While effective against traditional signal processing, these watermarks are highly vulnerable to speech reconstruction models (e.g., denoisers, neural codecs, and vocoders). These models, which reconstruct clean speech based on prior feature distributions, inadvertently suppress the low-energy watermarks, leading to significant decoding failures.

The core challenge lies in the inherent robustness–fidelity trade-off in existing designs. Increasing watermark energy to improve robustness typically degrades perceptual quality. Conversely, embedding-based methods (e.g., WavMark, AudioSeal) constrain energy to maintain fidelity, rendering them fragile against reconstruction. Generative methods (e.g., VoiceMark) offer better robustness but often suffer from lower fidelity and higher training complexity due to the need to jointly optimize generation and watermarking tasks.

2. Methodology

The authors propose AlignMark, a feature-aligned watermarking framework designed to break the traditional trade-off by aligning the watermark's feature distribution with the original speech rather than merely constraining its energy.

Architecture Overview:
The system consists of a Watermark Embedder and a Watermark Decoder, trained jointly with a frozen pretrained speech codec (SpeechTokenizer).

Pseudo-Speech Watermark Generation:
Instead of adding noise directly to the waveform, the embedder injects watermark bits into the latent space of a pretrained codec. A 6-layer 2D CNN adapter transforms the original latent features combined with watermark embeddings into modified latents. These are decoded by the frozen codec to generate a "pseudo-speech" watermark ( $x'$ ). This ensures the watermark inherently follows the statistical distribution of natural speech.
Spectrogram Fusion:
The pseudo-speech watermark is fused with the original audio ( $x$ ) in the spectrogram domain. Both signals are converted to complex spectrograms via STFT. A learnable 2D CNN integrator predicts frame-frequency fusion weights ( $\alpha$ ) to compute the final spectrogram: $s_w = s_{x'} \cdot \alpha + s_x \cdot (1 - \alpha)$ . The inverse STFT yields the watermarked audio ( $\hat{x}$ ).
Perceptual Guidance (VAD and Losses):
To ensure imperceptibility while allowing higher energy, the method concentrates the watermark within voiced regions of the spectrogram.
- VAD Loss: A voice activity detection loss guides the embedder and decoder to focus on voiced frames, avoiding silent regions where artifacts are more noticeable.
- Perceptual Losses: The training objective includes auditory masking loss, speaker similarity loss, and latent similarity loss to preserve speaker identity and psychoacoustic naturalness.
- Fidelity Losses: Standard losses (L1, Mel-spectrogram, Adversarial, SI-SNR) are used but with relaxed weights compared to traditional methods, relying on the feature alignment to maintain quality.
Watermark Decoder:
The decoder processes the watermarked audio through a differentiable distortion pipeline (including reconstruction models) and a feature encoder. It employs:
- Temporal Weight Prediction: A 1D CNN predicts frame-wise likelihoods of voiced segments to weight the decoding process.
- Feature Pyramid Extraction: A multi-scale 2D CNN extractor captures fine-grained details from the speech latents to recover the watermark, addressing the challenge of extracting data fused into complex speech features.

3. Key Contributions

Feature-Aligned Design: The paper introduces a paradigm shift from energy-constrained embedding to feature-aligned generation. By generating a pseudo-speech watermark that matches the original speech distribution, the method allows for higher watermark energy without sacrificing imperceptibility.
Hybrid Framework: The authors develop a framework that embeds watermarks in pretrained codec latents and fuses them in the spectrogram domain, guided by VAD and perceptual losses to target voiced regions specifically.
Robustness to Unseen Models: The approach demonstrates significant robustness against both seen and unseen speech reconstruction models (denoisers, codecs, vocoders) without requiring the reconstruction models to be part of the training objective.

4. Experimental Results

The method was evaluated against state-of-the-art baselines (WavMark, AudioSeal, TimbreWM, VoiceMark, WMCodec) across 21 distortions, including six speech reconstruction models.

Robustness: Under speech reconstruction models (e.g., ClearerVoice, EnCodec, HiFiGAN), the proposed method achieved significantly higher bit accuracy (ACC) and lower false attribution rates (FAR) compared to embedding-based methods. For instance, against ClearerVoice, the method achieved 0.92 ACC compared to 0.51–0.78 for others. It also maintained strong performance against traditional distortions (resampling, noise, pitch shift).
Imperceptibility:
- Subjective (ABX): The method achieved an ABX listening score comparable to embedding-based methods (approx. 50-51%, indicating imperceptibility) and significantly better than generative methods like VoiceMark.
- Objective (VISQOL MOS): The method scored 4.44, ranking second only to AudioSeal (4.71) and outperforming generative approaches.
- Fidelity Metrics: While not optimized for maximum fidelity, the method outperformed generative baselines on NISQA, PESQ, and SI-SNR.
Ablation Study: Removing the spectrogram fusion or VAD loss resulted in substantial drops in robustness, confirming the necessity of feature alignment and voiced-region targeting. Removing the feature pyramid caused the largest drop in decoding accuracy, highlighting the need for multi-scale feature extraction.

5. Significance and Claims

The paper claims that by aligning the watermark with the original speech feature distribution, it is possible to mitigate the suppression effects of speech reconstruction models while maintaining imperceptibility comparable to existing embedding-based methods.

The authors position this work as a solution to the applicability limits of current audio watermarking in modern speech pipelines (voice calls, online meetings, social media) where reconstruction models are ubiquitous. By decoupling robustness from the strict low-energy constraints of traditional designs, the method offers a more flexible trade-off, making audio watermarking more reliable in real-world scenarios involving speech reconstruction. The work does not claim to solve all fidelity issues but asserts that it achieves a superior balance between robustness and perceptual quality compared to the current state of the art.