Technical Summary: PASE - Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement

1. Problem Statement

Generative models have recently emerged as a compelling alternative to traditional discriminative approaches in Speech Enhancement (SE), offering superior perceptual quality. However, these models suffer from a critical flaw: hallucination. Under severe noise conditions, generative models often produce outputs with inconsistent linguistic content (linguistic hallucinations) or altered speaker characteristics (acoustic hallucinations).

The authors argue that existing evaluations often overlook these issues because standard non-intrusive metrics (e.g., DNSMOS, UTMOS) cannot detect content distortion. While recent studies have begun incorporating Word Error Rate (WER) and Speaker Similarity (SpkSim), these evaluations are frequently conducted under high Signal-to-Noise Ratio (SNR) conditions where models perform reliably. In low-SNR scenarios, hallucinations become severe, compromising the practical utility of these systems.

The paper identifies two root causes for these failures in current generative approaches:

1. Linguistic Hallucinations: Stemming from the model's failure to constrain valid phonological structures. Existing Language Model (LM)-based approaches attempt to learn these priors from noise-corrupted input representations, leading to "contaminated" knowledge.
2. Acoustic Hallucinations: Resulting from the loss of fine-grained acoustic details (pitch, timbre) often caused by discretizing speech into tokens or insufficient acoustic conditioning.

2. Methodology: The PASE Framework

The authors propose the Phonologically Anchored Speech Enhancer (PASE), a generative framework designed to mitigate both types of hallucinations by leveraging the robust, pre-existing phonological prior embedded in the self-supervised model WavLM.

Core Components

The framework consists of two primary stages:

A. Denoising WavLM (DeWavLM) via Representation Distillation

Instead of learning a phonological prior from scratch or from corrupted inputs, PASE adapts a pre-trained WavLM into a denoising expert.

• Denoising Representation Distillation (DRD): The authors employ a feature-level knowledge distillation strategy. A "teacher" WavLM (frozen) processes clean speech, while a "student" WavLM (trainable) processes noisy speech. The student is trained to minimize the Mean Squared Error (MSE) between its final-layer output and the teacher's output.
• Preservation of Prior: This process is guided by WavLM's intrinsic phonological prior. Crucially, the authors found that using a simple distillation objective (KD) alone was superior to joint objectives involving masked prediction loss, as it effectively prevents catastrophic forgetting of the pre-trained prior while refining the model for denoising.
• Continuous Space: Unlike Discrete Language Modeling (DLM) approaches that discretize speech into tokens, PASE operates entirely in the continuous representation space, avoiding information loss associated with discretization.

B. Dual-Stream Acoustic-Conditioned Reconstruction

To address acoustic hallucinations and ensure speaker identity preservation, PASE utilizes a dual-stream conditioning mechanism for the neural vocoder (Vocos).

• Phonetic Stream: The output of the final transformer layer (Layer 24) of DeWavLM is used. This layer encodes high-level, abstract phonetic content, ensuring linguistic integrity.
• Acoustic Stream: The output of the first transformer layer (Layer 1) is used. This layer retains fine-grained acoustic details and speaker characteristics.
• Fusion: These two streams are aligned via a linear projection and fused via element-wise addition before being fed into the vocoder. This design ensures the synthesized waveform is both linguistically accurate and faithful to the original speaker's identity.

3. Key Contributions

The paper outlines three primary contributions:

1. New Conceptualization of Hallucination: The authors formally categorize hallucinations in SE into linguistic (content distortion) and acoustic (speaker characteristic distortion) types, tracing them to distinct root causes (failure to constrain phonological structures vs. loss of acoustic cues).
2. A Novel Prior-Leveraging Paradigm: PASE shifts the paradigm from learning potentially flawed priors from noisy data to leveraging robust, pre-existing priors from self-supervised models. It introduces a dual-stream design that simultaneously mitigates linguistic hallucinations (via phonological anchoring) and acoustic hallucinations (via low-level acoustic conditioning).
3. State-of-the-Art Performance: Extensive experiments demonstrate that PASE achieves new SOTA results, significantly outperforming existing generative and discriminative models in reducing both types of hallucinations while maintaining superior perceptual quality and lower computational complexity.

4. Experimental Results

The authors evaluated PASE on simulated LibriTTS test sets and the DNS1 test set (including reverberant conditions), comparing against discriminative models (TF-GridNet), diffusion/flow-matching models (StoRM, FlowSE), LM-based models (LLaSE-G1), and commercial solutions (Adobe Enhance Speech V2).

• Linguistic Integrity: PASE achieved the lowest Word Error Rate (WER) among all generative models (7.49% vs. 36.58% for LLaSE-G1 and 21.32% for AES-V2), demonstrating superior resistance to linguistic hallucinations.
• Speaker Fidelity: PASE achieved a Speaker Similarity (SpkSim) of 0.80, matching the performance of the discriminative baseline TF-GridNet (0.80) and significantly outperforming other generative models like LLaSE-G1 (0.42) and AES-V2 (0.60). While this score is lower than the 'Clean' reference baseline (1.00), PASE demonstrates the highest fidelity among the tested enhancement methods, effectively mitigating acoustic hallucinations.
• Perceptual Quality: PASE maintained high perceptual quality scores (UTMOS: 3.09), comparable to or exceeding other generative models, while significantly outperforming the discriminative TF-GridNet (UTMOS: 2.62).
• Efficiency: PASE demonstrated lower computational complexity (MACs) compared to other generative baselines like StoRM and FlowSE.

Ablation Insights:

• Prior Source: Experiments revealed that the phonological prior is not solely a function of data scale but is fundamentally rooted in the masked prediction objective of pre-training. Models initialized from scratch failed to learn the necessary contextual inference capabilities, regardless of data size.
• Distillation Strategy: Using only the Knowledge Distillation (KD) loss yielded the best performance, preserving the teacher's representation fidelity (RFS) and contextual inference ability (MRS) better than joint objectives.
• Dual-Stream Design: Conditioning the vocoder on the first-layer acoustic representation significantly improved SpkSim (from 0.57 to 0.80) with minimal impact on linguistic metrics, validating the necessity of low-level acoustic cues.

5. Significance and Claims

The paper claims that PASE represents a significant step forward in making generative speech enhancement reliable for real-world applications. By anchoring the enhancement process to the robust phonological prior of WavLM, PASE bridges the gap between high perceptual quality and content accuracy.

The authors emphasize that their approach addresses the fundamental flaw of learning from corrupted inputs. Instead of attempting to "unlearn" noise or learn new priors from noisy data, PASE utilizes the "clean" knowledge already encoded in self-supervised models. This results in a system that is not only perceptually pleasing but also faithful to the original speech content and speaker identity, even under severe noise and reverberation. The work suggests that for generative SE to be practical, it must move beyond simple perceptual optimization and explicitly constrain the generation process using robust, pre-existing linguistic and acoustic priors.