💻 computer science

Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecs

This paper proposes a self-embedding audio watermarking framework that utilizes compact neural codec representations instead of cryptographic hashes to enable training-free detection, precise frame-level localization, and full recovery of manipulated speech segments across various ultra-low-bitrate neural codecs.

Original authors: Yigitcan Özer, Xin Wang, Zhe Zhang, Junichi Yamagishi

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yigitcan Özer, Xin Wang, Zhe Zhang, Junichi Yamagishi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, a voice recording is often treated as a permanent record of truth. Yet, technology has advanced to the point where a single sentence in a speech can be swapped, deleted, or rewritten without leaving a trace that the human ear can detect. This ability to surgically alter audio creates a profound problem: how can we verify that a recording is genuine, and if it has been tampered with, can we pinpoint exactly where the lie began? Traditional methods for checking audio integrity rely on digital "seals," such as cryptographic hashes. These are like unique fingerprints calculated from the audio; if even a single bit of the recording changes, the fingerprint breaks, signaling that tampering has occurred. However, these traditional seals have a fatal flaw: once the seal is broken, the original content is lost forever. The system can tell you that a segment was altered, but it cannot tell you what the original words were, leaving the listener with a gap in the story that cannot be filled.

Researchers at the National Institute of Informatics in Tokyo have proposed a different approach, one that moves beyond simple detection to active recovery. Their work explores a method called self-embedding, where the audio file carries a hidden, compressed copy of itself inside its own digital structure. Imagine a document that contains a tiny, encrypted blueprint of its own text hidden within its margins; if a page is torn out or rewritten, the reader can use that blueprint to reconstruct the missing or altered text. The researchers tested this concept using a new generation of ultra-low-bitrate neural codecs. These are advanced computer programs capable of compressing speech into incredibly small digital footprints—so small that they can fit many copies of the original speech's "blueprint" inside the recording itself without changing how the audio sounds to a human listener. By embedding these compressed representations rather than just a simple hash, the system can not only detect that a segment has been manipulated but also reconstruct the authentic voice that was there before the attack.

The team tested this framework under ideal conditions, meaning they simulated perfect recording environments without background noise or signal degradation, to see how well the system could handle four specific types of digital tampering. They tested scenarios where words were replaced with new audio from the same speaker, where words were replaced with synthetic speech generated by artificial intelligence, where words were simply deleted, and where new words were inserted into the flow of speech. In every case, the system successfully recovered the hidden blueprint. Because the blueprint was embedded multiple times throughout the recording, the system could ignore the corrupted parts and piece together the original message using the intact copies, much like solving a puzzle where most of the pieces are still available. The result was a perfect recovery of the hidden data, allowing the system to generate a reconstruction of the original, unaltered speech for any manipulated section.

Once the system reconstructed what the audio should have been, it compared this reconstruction against the actual received audio to find the discrepancies. The researchers used a mathematical technique to align the two versions of the audio, allowing them to spot even slight timing differences or structural changes. They found that the system was highly effective at identifying that a recording had been tampered with and could locate the specific boundaries of the altered segments. The performance varied depending on the type of manipulation; replacing words with new audio was the easiest to detect, while deleting words proved the most difficult, as removing content compresses the timeline and makes the distortion harder to isolate. However, the most significant finding was that the choice of the compression tool, or codec, mattered far more than the method used to hide the data. The researchers tested three different neural codecs, and the one that produced the most faithful reconstruction of the original speech also provided the most accurate detection and localization of the tampering.

The study also revealed a surprising nuance in how these systems work. The quality of the reconstructed speech did not always correlate with how well the system could detect the tampering. One of the codecs produced a reconstruction that sounded very natural and smooth but was less accurate in matching the exact waveform of the original, which actually made it harder for the system to spot the differences. Conversely, another codec produced a reconstruction that was less natural-sounding but matched the original data more precisely, leading to better detection results. This suggests that for the purpose of verifying integrity, the goal is not necessarily to create a perfect-sounding copy, but to create a copy that is mathematically consistent with the original source. The researchers confirmed that the hidden data could be recovered without any errors, meaning the original spoken content could always be restored, a feat that traditional hash-based systems cannot achieve.

Ultimately, this work demonstrates that proactive defense against audio manipulation is possible without needing to train the system on examples of fake audio. The system operates by comparing the received signal to its own self-reconstruction, requiring no prior knowledge of what a "fake" looks like. While the current tests were conducted in a controlled environment without real-world noise or compression, the results show that embedding a compressed version of the audio within itself is a viable path forward. It offers a way to not only flag a recording as compromised but to heal the damage, restoring the truth of the original utterance. The research indicates that as neural audio codecs continue to improve, becoming smaller and more efficient, this self-recovery method could become a powerful tool for preserving the integrity of spoken communication in an era where digital forgeries are increasingly common.

Technical Summary: Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecs

Problem Statement
Partial manipulation of speech recordings—where only localized segments are altered while the rest remains intact—poses a significant challenge for content integrity verification. Traditional robust watermarks are designed for ownership tracing and survive signal processing but fail to localize edits. Conversely, classical fragile watermarking schemes using cryptographic hashes can precisely detect and localize tampering under ideal conditions but cannot recover the original content once a segment is overwritten. Furthermore, recent learned semi-fragile approaches often require training on spoofed examples and lack self-recovery capabilities. The core challenge addressed is developing a proactive defense mechanism that simultaneously enables detection, frame-level localization, and content recovery of manipulated speech without requiring training on forged data.

Methodology
The authors propose a self-embedding audio watermarking framework that embeds a compact, ultra-low-bitrate neural codec representation of the carrier signal directly into itself. The system operates through the following pipeline:

Embedding and Redundancy:
- A neural codec encoder compresses the input speech signal $x$ into a binary representation $f(x)$ at a bitrate $\rho$ (typically $<1$ kbps).
- To ensure robustness against partial manipulation, this compressed representation is repeated $R$ times to form the payload message $m$ .
- The payload is embedded into the host signal using Least Significant Bit (LSB) modification. The study evaluates three bit-depth variants ( $k \in \{1, 2, 4\}$ bits per sample).
- Each repetition is preceded by a 64-bit synchronization preamble. The embedding capacity $C$ is strictly limited by perceptual imperceptibility, but the ultra-low bitrate of modern neural codecs allows the compressed payload to fit multiple times within the available budget ( $R \geq 1$ ).
Verification and Recovery:
- At the receiver, the embedded bitstream is extracted. Due to potential corruption in manipulated regions, the system employs majority-vote decoding across the $R$ repetitions to recover the original codec representation $\hat{m}$ .
- A codec decoder reconstructs the authentic speech waveform $R(y) = g(\hat{m})$ from the recovered payload.
- Detection and Localization: The system computes a mismatch score between the received signal $y$ $y$ and the self-reconstructed signal $R(y)$ $R (y)$ . This is achieved using Dynamic Time Warping (DTW) on log-mel spectrogram features.
  - Detection: The mean DTW cost serves as the utterance-level decision metric.
  - Localization: The per-frame DTW cost profile is projected onto the time axis to identify specific tampered segments.
Experimental Setup:
- Codecs: Three ultra-low-bitrate neural codecs were evaluated: SNAC (0.98 kbps), SemantiCodec (0.65 kbps), and TAAE (0.4 kbps).
- Manipulations: Four controlled manipulation types were tested under ideal channel conditions (no noise or compression): Direct Replacement, TTS Replacement, Deletion, and Insertion.
- Dataset: Experiments utilized 1,480 utterances from the AV-Deepfake1M validation split.

Key Contributions
This work extends a prior neural codec self-embedding framework in three specific directions:

Frame-Level Localization: Extending evaluation from utterance-level detection to precise frame-level localization of manipulated segments.
Multi-Bit LSB Variants: Generalizing the embedding strategy from single-bit LSB to multi-bit variants ( $k=1, 2, 4$ ) to analyze the trade-off between capacity and distortion.
Codec Evaluation: Systematically evaluating the framework across multiple ultra-low-bitrate neural codecs to determine how codec reconstruction fidelity influences detection and recovery performance.

Results

Payload Recovery: Under ideal channel conditions, the embedded payload was recovered without bit errors across all configurations, enabling the exact reconstruction of the authentic content within manipulated regions.
Detection Performance: The choice of neural codec was the dominant factor in performance.
- SemantiCodec achieved the best detection performance (Mean EER: 14.47%–14.78%).
- SNAC followed closely (Mean EER: 15.00%–15.15%).
- TAAE yielded significantly higher error rates (Mean EER: ~32.1%), attributed to its generative nature which prioritizes natural acoustic variations over sample-faithful alignment, thereby increasing baseline mismatch scores.
Localization: Localization performance (AUC) mirrored detection trends, with SemantiCodec and SNAC achieving mean AUCs of ~0.84, while TAAE scored ~0.785.
Manipulation Difficulty: Direct and TTS replacements were the most detectable. Deletion was consistently the most difficult manipulation to detect and localize, as removing content compresses the DTW path rather than creating a high-cost region.
Recovery Quality: While the reconstructed speech had moderate PESQ scores (1.63–1.75) due to the inherent limitations of ultra-low-bitrate compression, the recovered speech remained fully intelligible, natural, and speaker-consistent.
Decoupling: The study observed that reconstruction quality (PESQ) and detection performance are decoupled; SemantiCodec provided the best detection despite having the lowest reconstruction PESQ.

Significance and Claims
The paper positions this work as an initial exploration of proactive defense using neural codec self-embedding. The primary significance lies in demonstrating that self-recovery is achievable for partial speech manipulations without training on spoofed examples, a capability absent in both classical hash-based schemes and recent learned methods.

The authors modestly claim that while the framework successfully recovers content and provides detection/localization, the performance is not perfect even under ideal conditions. They emphasize that the reconstruction fidelity of the chosen neural codec is the critical determinant of system efficacy. The work establishes that ultra-low-bitrate neural codecs provide a practical payload size for self-embedding, enabling a trade-off where the exactness of hash-based verification is sacrificed for the ability to reconstruct the original spoken content. The paper concludes that future work should focus on semi-fragile embedding to handle channel distortions and further payload compression to increase redundancy.

Technical Summary: Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecs

More like this