Prosody-Guided Harmonic Attention for Phase-Coherent Neural Vocoding in the Complex Spectrum

Technical Summary: Prosody-Guided Harmonic Attention for Phase-Coherent Neural Vocoder

Problem Statement
Despite the success of neural vocoders in speech synthesis, current state-of-the-art models face two persistent limitations: limited prosody modeling and inaccurate phase reconstruction. Conventional approaches, such as HiFi-GAN and WaveGlow, rely on mel-spectrograms as conditioning features. While efficient, mel-spectrograms are lossy and non-invertible, discarding fine-grained harmonic and phase information. This leads to pitch artifacts, blurred harmonics, and temporal discontinuities, particularly in expressive or prosodically rich utterances. Furthermore, while some recent models (e.g., Vocos, BigVGAN) have moved toward spectral domain prediction or pitch-aware conditioning, they often fail to explicitly leverage prosody-aware mechanisms like harmonic attention or to jointly model magnitude and phase in a unified, phase-coherent framework.

Methodology
The authors propose a neural vocoder architecture designed to capture both prosody and phase information for high-fidelity waveform generation. The system operates directly in the complex spectral domain rather than the mel-spectrogram domain. The architecture consists of three primary components:

1. Prosody-Guided Harmonic Attention:
   The model extracts the fundamental frequency (F0) from the reference waveform using the Harvest algorithm. This F0 contour is embedded and injected into the encoder-decoder pathway via a harmonic attention mechanism. Unlike models that treat pitch as a simple auxiliary feature, this module computes attention weights based on the interaction between encoded spectral features ($H$) and F0 embeddings ($F$). This mechanism explicitly aligns the encoded representation with the harmonic structure indicated by F0, emphasizing voiced regions and harmonic structures while leaving unvoiced frames unaffected.

2. Direct Complex-Spectrum Prediction:
   The decoder operates directly in the complex Short-Time Fourier Transform (STFT) domain. Instead of predicting magnitude-only features and reconstructing phase heuristically, the model predicts the real and imaginary components of the spectrum simultaneously. The output is split into real ($\hat{R}$) and imaginary ($\hat{I}$) parts, forming a complex spectrum $\hat{S} = \hat{R} + j\hat{I}$. This spectrum is converted to a waveform via the Inverse STFT (ISTFT), ensuring phase coherence by construction and eliminating the need for post-hoc phase estimation.

3. Multi-Objective Perceptual Optimization:
   Training utilizes a multi-objective loss function combining three terms:

   • Multi-resolution STFT (MR-STFT) Loss: Provides stable spectral supervision across multiple analysis windows to capture harmonic structure and spectral envelopes.
   • Adversarial Loss: A lightweight setup using multi-period and multi-resolution discriminators operating on waveforms reconstructed from the predicted complex spectrum to enhance perceptual realism.
   • Phase-Aware Loss: A novel term computed between the predicted and reference complex spectra. By normalizing to unit magnitude, this loss focuses exclusively on phase alignment, enforcing phase coherence independent of amplitude scaling.

Key Contributions
The paper identifies three main innovations:

1. Prosody-Guided Harmonic Attention: A mechanism that leverages F0 to condition spectral modeling, improving prosodic fidelity in the encoding stage by emphasizing voiced regions.
2. Direct Complex-Spectrum Prediction: A unified framework that predicts both magnitude and phase (real and imaginary components), enabling phase-coherent waveform reconstruction via inverse STFT without decoupling.
3. Multi-Objective Perceptual Training: An optimization strategy integrating adversarial, spectral, and phase-aware losses to better align the model with perceptual quality and phase coherence.

Experimental Results
Experiments were conducted on the LJSpeech and VCTK corpora, comparing the proposed system against HiFi-GAN, AutoVocoder, Vocos, and a Griffin-Lim anchor. The results demonstrate consistent gains across objective and subjective metrics:

• Pitch Accuracy: The F0 Root Mean Squared Error (RMSE) was reduced by 22% compared to HiFi-GAN and 18% compared to AutoVocoder.
• Prosodic Reliability: The Voiced/Unvoiced (V/UV) error rate was lowered by 18% relative to HiFi-GAN.
• Perceptual Quality: The Mean Opinion Score (MOS) improved to 4.45, surpassing HiFi-GAN (4.2), AutoVocoder (4.3), and Vocos (4.1). In preference tests, listeners favored the proposed system in 88% of cases.
• Spectral Fidelity: Analysis of residual mel energy showed the proposed model maintained lower residuals and tracked reference harmonics more closely than baselines, indicating reduced temporal smearing.

Significance and Claims
The authors claim that this work establishes a unified vocoder architecture that addresses long-standing gaps in prosody and phase modeling. By combining prosody-guided attention with direct complex-spectrum prediction, the system yields more natural, pitch-accurate, and robust synthetic speech. The paper posits that explicitly modeling prosody and phase coherence simultaneously is critical for advanced applications such as voice cloning, emotional speech synthesis, and brain-to-speech decoding, where faithful prosody reproduction is essential. The work sets a foundation for expressive neural vocoding by demonstrating that joint magnitude-phase modeling, guided by prosodic cues, outperforms traditional mel-spectrogram-based approaches.