💬 NLP

Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing

The paper introduces SIEDD, a multi-codec discrete diffusion model that leverages hierarchical token generation and text-guided conditioning to achieve state-of-the-art performance in speech inpainting and editing while preserving speaker identity and prosody.

Original authors: Iftach Shoham, Tali Dror, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani

Published 2026-08-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Iftach Shoham, Tali Dror, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are listening to a favorite song, but suddenly a few seconds of the melody are missing, or perhaps someone whispered a word you didn't want to hear. In the world of audio science, fixing these holes without ruining the rest of the track is a bit like trying to repair a torn tapestry. You can't just glue a new piece of fabric on top; the new thread has to match the color, the weave, and the tension of the surrounding fabric perfectly. For decades, computers struggled with this. They could fix tiny scratches, but if a whole sentence was missing, they often sounded robotic or out of sync with the speaker's voice.

Recently, scientists have started using a clever trick called "diffusion" to solve this. Think of diffusion like a game of "telephone" played in reverse. Imagine a clear picture that gets slowly covered in static noise until it's just a blur of gray. A diffusion model is a smart AI that learns how to take that blurry gray mess and slowly peel away the noise, step-by-step, until the original picture reappears. While this works great for images, doing it for speech is tricky because speech isn't just one smooth line; it's built in layers, like a cake with different flavors stacked on top of each other. This paper dives into how to use this "reverse noise" game to fix speech, but with a special twist that respects those layers.

The researchers behind this study, Iftach Shoham and his team, introduce a new system called SIEDD (Speech Inpainting and Editing via Discrete Diffusion). Their main goal was to create a tool that can either fill in missing parts of a recording (inpainting) or swap out words for new ones (editing) while keeping the speaker's voice, rhythm, and emotion exactly the same. They found that previous methods, which tried to build speech one word after another like a train on a track, often made mistakes that piled up. If the AI got the first word slightly wrong, the rest of the sentence would sound off.

Instead, SIEDD uses a "discrete diffusion" approach. Imagine the speech isn't a train, but a sculpture being carved. The AI starts with a rough block of stone (the basic shape of the sound) and then slowly chips away to reveal the fine details. The paper's big discovery is that you have to respect the order of these layers. The AI first figures out the "coarse" structure—the big, rough shape of the sound—and locks that in as a solid foundation. Only then does it move to the next layer to add the "fine" details, like the specific texture of the voice. By treating the lower layers as a clean, committed context and only "denoising" the current layer, the model avoids the confusion of trying to guess the fine details before the rough shape is settled.

The team tested this on a benchmark called RealEdit, which involves realistic scenarios like fixing damaged recordings or changing what someone said. The results suggest that SIEDD is quite good at its job. On the speech-editing tests, it achieved the best overall performance among the methods they compared, with the lowest error rates in understanding the words (WER) and the highest similarity to the original speaker. For filling in missing gaps, it outperformed other popular methods, especially when there were multiple gaps to fill at once. The authors suggest that this success comes from explicitly modeling the "hierarchy" of the sound—acknowledging that some parts of the audio code are more important for the big picture, while others are just for the tiny details.

One of the clever tricks in their system is a "localized guide." Imagine you are editing a sentence, and you want to change the word "cat" to "dog." The AI needs to know exactly where to change the sound and where to leave the rest alone. The researchers built a mechanism that focuses its attention only on the specific span of words being changed, ensuring the new words fit perfectly with the old ones without messing up the surrounding conversation. They also added a way to predict how long the new words should be, so the edited speech doesn't sound rushed or dragged out.

In short, this paper suggests that by treating speech like a multi-layered cake and fixing it layer by layer—starting from the bottom and working up—computers can repair and edit audio much more naturally than before. While the system isn't perfect (it still struggles a bit as the gaps get larger), it represents a significant step forward in making AI sound less like a robot and more like a human who can seamlessly fix a broken sentence or swap a word without anyone noticing the edit. The authors conclude that this "hierarchy-aware" approach is a promising alternative to the older, step-by-step methods, offering a way to preserve the soul of a speaker's voice even when the words change.

Technical Summary: Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing (SIEDD)

Problem Statement

Speech recordings frequently contain missing, corrupted, or incorrect segments that require reconstruction or modification without re-synthesizing the entire utterance. Two primary tasks address this: speech inpainting (restoring missing segments) and speech editing (replacing spoken content based on an edited transcript). Both tasks demand that the generated speech not only express the intended words but also remain acoustically consistent with the surrounding speaker identity, prosody, timing, and recording conditions.

Traditional signal-processing methods (e.g., autoregressive prediction, sparse representations) often fail when missing regions span multiple phonemes or words, as they lack the capacity to model long-range linguistic and acoustic context. While recent generative approaches using neural codec tokens have improved quality, autoregressive (AR) models suffer from a fundamental limitation in editing: they generate tokens in a fixed causal order. This prevents the revision of earlier decisions, causing errors in linguistic content, duration, or prosody to propagate through the edited span. Furthermore, standard diffusion models applied to neural codecs often ignore the hierarchical nature of Residual Vector Quantization (RVQ), treating all codebooks (CBs) symmetrically rather than respecting their coarse-to-fine dependencies.

Methodology

The authors propose SIEDD (Speech Inpainting and Editing via Discrete Diffusion), a framework that leverages discrete diffusion over hierarchical codec tokens to address these limitations. The core innovation is HiCoDD (Hierarchical Codec Discrete Diffusion), which explicitly models the RVQ structure.

1. Hierarchical Multi-Codebook Discrete Diffusion (HiCoDD)
Neural audio codecs (e.g., EnCodec) represent speech using a hierarchy of RVQ codebooks, where lower levels capture coarse acoustic structure and higher levels encode residual refinements.

Architecture: HiCoDD generates codebooks in their inherent coarse-to-fine order. For a target codebook $k$ , the model treats the previously generated, clean lower codebooks ( $<k$ ) as a fixed, committed acoustic context. Diffusion is applied only to the current refinement codebook $k$ .
Training: The model uses a "clean-context encoder" to process the committed lower CBs and a "denoising decoder" for the noised current CB. This separation ensures that the decoder receives the exact clean context it will have during inference, preventing information leakage and enabling joint training without autoregressive constraints across time.
Factorization: The generative process is factorized hierarchically: $p(X|P) = p(x^{(1)}|P) \prod_{k=2}^K p(x^{(k)} | x^{(<k)}, P)$ . This allows the model to refine the acoustic representation step-by-step rather than reconstructing all levels independently.

2. Score Network and Conditioning
The score network is a Diffusion Transformer (DiT) adapted for absorbing-state discrete diffusion.

Text Conditioning: The target phoneme sequence is encoded via a pretrained XPhoneBERT and injected via cross-attention.
Context Conditioning: The committed lower CBs are encoded and injected via a second cross-attention stream.
Attention Masking: A block-triangular attention mask enforces the coarse-to-fine dependency, ensuring the current CB cannot attend to its own future clean states or higher CBs.

3. Localized Classifier-Free Guidance (CFG)
To balance adherence to the target text with preservation of the surrounding context, the authors adapt CFG for categorical score prediction.

Log-Space Guidance: Guidance is applied directly to the log concrete scores: $\log s_{cfg} = \alpha \log s_{cond} + (1-\alpha) \log s_{neg}$ .
Span Localization: For speech editing, the negative branch is constructed by randomizing only the phonemes within the edited span while keeping the surrounding phonemes and lower CB representations identical to the conditional branch. This isolates the guidance signal to the modification target, suppressing unwanted content changes without disrupting coarticulation cues at the boundaries. For inpainting, where the transcript is unchanged, the full phoneme sequence is randomized for the negative branch.

4. Duration Prediction
For text-guided editing (insertion/substitution), the duration of the generated segment is unknown. SIEDD employs a duration predictor (inspired by FastSpeech) to estimate the number of acoustic tokens required based on the target phoneme sequence, enabling variable-duration generation.

Key Contributions

Hierarchy-Aware Codec Diffusion: Introduction of HiCoDD, which denoises temporal positions within each codebook while generating codebooks in their inherent coarse-to-fine order. This preserves RVQ dependencies without imposing autoregressive generation across time.
Contrastive Guidance for Localized Editing: Adaptation of classifier-free guidance for categorical diffusion, specifically designed to localize the contrast to the edited span. This strengthens adherence to target phonemes while preserving the surrounding acoustic and linguistic context.
Performance: Demonstration of state-of-the-art performance on the RealEdit benchmark for speech editing and superior performance over autoregressive baselines across various speech inpainting settings.

Experimental Results

The model was evaluated on the RealEdit benchmark, which includes diverse speakers, accents, and recording conditions.

Speech Editing: SIEDD achieved the best overall performance among evaluated methods (including VoiceCraft and SSR-Speech). It recorded the lowest Word Error Rate (WER: 0.121), highest speaker similarity (0.98), and lowest Mel-Cepstral Distortion (MCD: 270.0). While a TTS baseline achieved a higher UTMOS score, it did so by synthesizing speech independently rather than adapting to the original recording's context; SIEDD maintained perceptual quality comparable to the original audio (UTMOS ~3.44).
Speech Inpainting: SIEDD outperformed autoregressive baselines (VoiceCraft, SSR-Speech) across all gap configurations (single and multiple gaps). Notably, it showed significant gains in short and medium gaps (e.g., reducing MCD to 21.1 for a 250ms gap vs. 91.3 for SSR-Speech). The authors attribute the stability of SIEDD in multi-gap scenarios to its ability to iteratively refine masked regions without the error accumulation inherent in sequential AR generation.
Ablation Studies: Removing the hierarchical factorization (joint modeling) degraded WER and SIM. Similarly, removing the duration predictor or localized CFG resulted in performance drops, confirming the necessity of these components for accurate duration estimation and content adherence.

Significance and Claims

The paper claims that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. By treating previously generated codebooks as clean, committed context, SIEDD enables leakage-free joint training that aligns with sequential coarse-to-fine inference. The framework demonstrates that discrete diffusion, when combined with hierarchical modeling and localized guidance, offers a robust alternative to autoregressive generation for tasks requiring simultaneous preservation of speaker identity, prosody, and linguistic accuracy. The authors position SIEDD as a promising direction for context-preserving speech modification, with future work suggested for faster sampling, improved duration modeling, and multilingual applications.