BemaGANv2: Discriminator Combination Strategies for GAN-based Vocoders in Long-Term Audio Generation

BemaGANv2 is an advanced GAN-based vocoder that enhances long-term audio generation for Text-to-Music and Text-to-Audio applications by integrating Anti-aliased Multi-Periodicity composition modules in the generator and systematically evaluating novel discriminator combination strategies, including the Multi-Envelope Discriminator, to achieve high-fidelity and temporally coherent results.

The Gist

Imagine you are trying to teach a robot to sing a song or narrate a story for an hour straight. You give the robot a sheet of music (the text or melody), and it needs to turn that into actual sound waves. This is the job of a Vocoder.

For a long time, these robots were good at singing short phrases but would start sounding robotic, glitchy, or even lose their rhythm when asked to sing for a long time. This paper introduces a new, upgraded robot called BemaGANv2 that fixes these problems, especially for long audio generation like full songs or audiobooks.

Here is how it works, explained through simple analogies:

1. The Problem: The "Short-Term Memory" Robot

Previous models (like HiFi-GAN) were like a musician who is great at playing a single note perfectly but gets confused if asked to play a whole symphony. They would sometimes lose the beat, sound flat, or even accidentally double the length of the song (a glitch the authors found in older models).

2. The Solution: BemaGANv2

The authors built a new system with two main upgrades: a better "singer" (Generator) and a smarter "critic" (Discriminator).

The Singer: The "Snake" with a Rhythm

The part of the AI that actually creates the sound has been upgraded.

• Old Way: Imagine a singer who just shouts notes. It's loud, but it doesn't capture the natural "wobble" or vibration of a real voice or instrument.
• New Way (AMP & Snake): The new singer uses a special tool called the Snake activation function. Think of this as giving the singer a built-in metronome that understands the natural "wiggles" and vibrations of sound. Instead of just shouting, the singer now knows how to naturally oscillate (wiggle) like a real guitar string or vocal cord. This helps the robot keep a steady rhythm even during a 90-minute song.

The Critics: The "Envelope" and the "Spectrogram"

In AI, the "Discriminator" is the teacher that grades the singer. If the singer sounds fake, the teacher gives a bad grade. BemaGANv2 doesn't just have one teacher; it has a team of two specialized critics working together.

1. The "Envelope" Critic (MED):

   • What it does: Imagine looking at a wave in the ocean. The "envelope" is the overall shape of the wave—how high the peaks are and how deep the troughs are.
   • The Analogy: This critic doesn't care about the tiny details of the water; it cares about the rhythm and energy. Is the song getting louder and softer naturally? Does the breath sound right? It checks if the "shape" of the sound feels human and emotional.
   • Why it matters: This prevents the robot from sounding flat or robotic over long periods.

2. The "Spectrogram" Critic (MRD):

   • What it does: This critic looks at the sound like a detailed map of colors (a spectrogram). It checks if the high notes are sharp and the low notes are deep.
   • The Analogy: If the Envelope critic is checking the rhythm, this critic is checking the instrument quality. Is the violin sounding like a violin, or like a cheap toy?
   • Why it matters: This ensures the sound is crisp, clear, and high-fidelity.

The Magic Combination:
Previous models used different combinations of critics. The authors tested many pairs and found that combining the "Envelope" critic and the "Spectrogram" critic was the winning team. It's like having a conductor who checks the tempo and a sound engineer who checks the audio quality simultaneously. They cover each other's blind spots.

3. The Results: A Marathon Runner

The authors tested this new system on two types of tasks:

• Sprints (Short audio): 20-second clips of speech or drum beats.
• Marathons (Long audio): 90-second full music tracks.

The Findings:

• Old Models: They stumbled in the marathon. They would get tired, lose the rhythm, or the audio would stretch out weirdly (like a tape getting pulled).
• BemaGANv2: It ran the marathon with ease. It kept the rhythm steady, the voice natural, and the music clear from start to finish.
• The "Snake" Effect: The authors discovered that the "Snake" activation function in the singer was the secret sauce. It allowed the AI to "extrapolate"—meaning it could guess how to keep the rhythm going for a long time, even though it was mostly trained on short clips.

4. Why This Matters

This isn't just about making better robot voices. This technology is crucial for:

• Text-to-Music: Generating full songs from a text description.
• Text-to-Audio: Creating sound effects for movies or games that last a long time without glitching.
• Real-time Streaming: Because it's fast (it can generate audio 100x faster than real-time), it could eventually be used for live, AI-generated radio or music.

Summary

Think of BemaGANv2 as a new kind of AI musician. It has a singer that understands natural vibrations (thanks to the "Snake" function) and a judging panel that checks both the emotional shape of the song and the technical quality of the instruments. Together, they allow the AI to create long, high-quality audio that sounds natural and stays in rhythm, solving a problem that plagued previous models for years.

Technical Summary

Here is a detailed technical summary of the paper "BemaGANv2: Discriminator Combination Strategies for GAN-based Vocoders in Long-Term Audio Generation."

1. Problem Statement

The paper addresses the critical challenge of long-term audio generation in Text-to-Music (TTM) and Text-to-Audio (TTA) systems. While existing Generative Adversarial Network (GAN) vocoders (e.g., HiFi-GAN, BigVGAN) excel at short-term speech synthesis, they often struggle with:

• Temporal Coherence: Maintaining consistent prosody and harmonic structure over extended durations (e.g., full music tracks).
• Periodicity Modeling: Standard activation functions (like ReLU) and residual blocks fail to effectively model the oscillatory nature of periodic signals (pitch) over long sequences.
• Discriminator Limitations: Existing discriminator combinations often focus on either time-domain features or frequency-domain features but lack a synergistic approach to capture both temporal energy envelopes and spectral sharpness simultaneously.
• Inference Instability: The authors observed a specific anomaly in HiFi-GAN where long-term audio generation resulted in waveform duration doubling, suggesting a failure in temporal extrapolation.

2. Methodology

The authors propose BemaGANv2, an advanced GAN-based vocoder that integrates architectural innovations in both the generator and the discriminator.

A. Generator Architecture

• AMP Blocks: Replaces traditional ResBlocks with Anti-aliased Multi-Periodicity (AMP) blocks (originally from BigVGAN).
• Snake Activation: Embeds the Snake activation function ($f_\alpha(x) = x + \frac{1}{\alpha}\sin^2(\alpha x)$) within the AMP blocks. This provides a learnable periodic inductive bias, allowing the network to better model harmonic structures and rhythmic patterns.
• Anti-Aliasing: Utilizes low-pass filtering (LPF) during upsampling and downsampling to prevent high-frequency artifacts caused by non-linear operations.

B. Discriminator Framework (Key Innovation)

The core contribution is the systematic evaluation and combination of discriminators to optimize long-term generation. BemaGANv2 employs a dual-discriminator strategy:

1. Multi-Envelope Discriminator (MED): A novel architecture proposed by the authors.
   • Mechanism: Extracts time-domain envelopes from the audio using Hilbert transforms (for upper/lower bounds) and Butterworth low-pass filters (at 300Hz and 500Hz) to capture multi-scale amplitude variations.
   • Goal: To detect temporal energy patterns, prosodic variations, and phrasing, which are crucial for perceptual naturalness.
2. Multi-Resolution Discriminator (MRD):
   • Mechanism: Operates on STFT-based log-magnitude spectrograms with varying FFT sizes and hop lengths.
   • Goal: To enforce spectral consistency, ensuring harmonic sharpness and timbral accuracy across different frequency resolutions.

Configuration Strategy: The paper systematically evaluates various combinations (e.g., MSD+MED, MPD+MRD, MED+MRD) to determine the optimal pairing. BemaGANv2 specifically adopts the MED + MRD combination.

C. Training Objectives

The model uses the Least-Squares GAN (LSGAN) formulation with:

• Adversarial Loss (LSGAN).
• Mel-Spectrogram Loss (L1).
• Feature Matching Loss (L1 between intermediate discriminator features).

3. Key Contributions

1. BemaGANv2 Architecture: A unified framework combining AMP/Snake generators with a complementary MED+MRD discriminator setup.
2. Systematic Discriminator Analysis: A comprehensive study demonstrating that the combination of discriminators is as critical as the individual components. The study proves that MED (temporal envelope) + MRD (spectral structure) yields the most balanced performance, outperforming traditional MPD+MSD or MPD+MRD setups.
3. Ablation Studies:
   • Confirmed that the Snake activation is critical for long-term stability, resolving the "duration doubling" anomaly seen in standard HiFi-GAN.
   • Showed that while MED alone improves temporal fidelity, the addition of MRD is necessary for spectral sharpness.
4. Reproducibility: Provided full implementation details, training configurations, and code (available on GitHub), including a detailed analysis of the HiFi-GAN inference anomaly.

4. Experimental Results

Experiments were conducted on the LJSpeech dataset (training) and Freesound.org (evaluation for out-of-distribution generalization).

Objective Metrics

BemaGANv2 (MED+MRD) achieved state-of-the-art results across both short-term (20s) and long-term (90s) audio:

• FAD (Fréchet Audio Distance): Lowest scores (0.911 short, 2.681 long), indicating high distributional similarity to ground truth.
• SSIM & PCC: Highest structural similarity and correlation coefficients.
• Periodicity Error: Lowest error rates, confirming superior pitch preservation.
• Comparison: Significantly outperformed HiFi-GAN (which suffered from duration doubling and poor long-term scores) and BigVGAN in long-term tasks.

Subjective Metrics

• MOS (Mean Opinion Score) & SMOS (Similarity MOS): BemaGANv2 received the highest scores from human listeners for both quality and similarity to ground truth in long-form audio.
• Observation on Over-discriminator setups: The combination of MED + MPD + MRD showed competitive objective metrics but lower subjective scores, attributed to mode collapse due to adversarial instability, highlighting that quality of discriminator pairing matters more than quantity.

Anomaly Resolution

The study identified that HiFi-GAN's failure in long-term generation (duration doubling) was caused by the generator architecture (specifically the lack of Snake activation and anti-aliasing), not the discriminator. Replacing HiFi-GAN's generator with the AMP+Snake architecture resolved the issue.

5. Significance and Impact

• Long-Term Generation: BemaGANv2 establishes a new benchmark for generating high-fidelity, temporally coherent audio over extended durations, a prerequisite for realistic TTM and TTA systems.
• Design Philosophy: The paper shifts the focus in vocoder design from simply stacking discriminators to strategically combining complementary discriminators (temporal envelope + spectral resolution).
• Efficiency: Despite high fidelity, the model remains lightweight (13.95M parameters for the generator) and achieves ~103x real-time speed on an NVIDIA A100, making it suitable for real-time streaming applications where diffusion-based models are too slow.
• Theoretical Insight: It provides empirical evidence linking specific activation functions (Snake) to the stability of periodic signal extrapolation in generative models.

In conclusion, BemaGANv2 demonstrates that a carefully engineered combination of periodicity-aware generators and multi-faceted discriminators can overcome the limitations of current GAN vocoders, enabling high-quality, long-duration audio synthesis.