⚡ electrical engineering

SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios

This paper introduces SE-AGCNet, an end-to-end framework that jointly optimizes speech enhancement and automatic gain control to overcome the limitations of conventional discrete pipelines, thereby achieving target loudness while improving speech quality and ASR accuracy in meeting scenarios.

Original authors: Jinming Zhang, Wei Rao, Xionghu Zhong, Eng Siong Chng

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinming Zhang, Wei Rao, Xionghu Zhong, Eng Siong Chng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hosting a meeting in a room where the audio quality is a bit of a mess. Some people are whispering from the back of the room, others are shouting from the front, and there's a constant hum of air conditioners and typing in the background.

Traditionally, fixing this audio has been like hiring two separate workers who don't talk to each other:

The Noise Cleaner (Speech Enhancement): Their job is to scrub away the background noise. But because they are so focused on removing noise, they sometimes accidentally "clean" the quiet whispers too, making them disappear completely.
The Volume Controller (Automatic Gain Control): Their job is to make sure everyone is heard at the same volume. But if they do their job before the Noise Cleaner, they might turn up the volume on the background noise, making the mess worse. If they do it after, they might turn up the volume on the parts the Noise Cleaner accidentally deleted.

The paper introduces a new solution called SE-AGCNet. Think of this not as two separate workers, but as a single, highly trained orchestra conductor who manages both tasks simultaneously.

How It Works: The "Joint Training" Approach

Instead of treating noise removal and volume control as separate steps, SE-AGCNet teaches the computer to do them at the same time.

The Synergy: The system learns a special trick: it tells the "Noise Cleaner" to be gentle with the quiet voices, knowing that the "Volume Controller" will boost them up later anyway. This prevents the quiet speakers from being erased.
The Result: The system removes the background noise while keeping the quiet speech intact, and then perfectly balances the volume so everyone sounds like they are sitting right next to the microphone.

The Training Data: "The Simulation Factory"

One of the biggest hurdles in building this system was that there was no existing library of audio data that showed both noisy speech and wildly varying volumes (which is common in real meetings).

To solve this, the authors built a data simulation pipeline called SE-AGC-DataGen.

The Analogy: Imagine a sound engineer in a lab who takes clean recordings of people speaking, mixes them with realistic meeting noises (like fans and keyboards), and then artificially makes some people sound very quiet and others very loud. They do this thousands of times to create a "practice gym" for the AI.
The Goal: This allows the AI to practice handling the exact kind of chaos found in real-world meetings without needing to record thousands of hours of actual messy meetings first.

How They Measured Success: The "Loudness Ruler"

The authors didn't just listen to see if it sounded good; they used a new, standardized way to measure "loudness" that is more like how human ears actually work.

The Old Way: Measuring volume like a simple thermometer (RMS), which can be misleading.
The New Way: Using LUFS (Loudness Units relative to Full Scale). Think of this as a "perception ruler." It measures how loud the audio feels to a human, not just the raw electrical signal. They aimed for a specific "comfort zone" of -23 LUFS, which is the standard for clear, balanced speech.

The Results: What Happened?

When they tested SE-AGCNet against the old "separate worker" methods:

Better Clarity: The speech sounded clearer, and the background noise was reduced more effectively.
Perfect Volume: The system successfully brought the volume of quiet speakers up and loud speakers down to the target "comfort zone" without distorting the sound.
Smarter Computers: When they fed the cleaned audio into a speech-to-text computer (ASR), the computer made fewer mistakes. This is crucial because if the volume is too low or the noise is too high, the computer can't understand what was said.

In Summary

The paper presents SE-AGCNet, a new framework that unifies noise removal and volume control into one smart system. By training them together, the system avoids the mistakes of doing them separately. It uses a custom-made "practice gym" of simulated meeting data to learn, and it proves that this joint approach results in clearer speech, better volume balance, and more accurate speech-to-text recognition in meeting scenarios.

Technical Summary: SE-AGCNet

1. Problem Statement

Conventional audio pipelines for meeting scenarios typically treat Speech Enhancement (SE) and Automatic Gain Control (AGC) as discrete, cascaded modules. This separation introduces significant performance limitations:

Cascading Order Issues: Applying AGC before SE indiscriminately amplifies both speech and background noise, degrading the signal-to-noise ratio (SNR) and complicating subsequent denoising. Conversely, applying AGC after SE makes its performance highly dependent on the quality of the SE output; residual noise is often unintentionally amplified.
Over-Suppression: SE models tend to over-suppress low-volume speech (e.g., far-field or quiet speakers), mistakenly classifying them as noise.
Lack of Joint Optimization: Existing unified frameworks often treat AGC as a detached post-processing step or rely on proprietary, non-reproducible target generation, limiting the model's ability to learn the synergy between noise suppression and volume normalization.

2. Methodology: SE-AGCNet

The authors propose SE-AGCNet, an end-to-end deep learning framework that jointly optimizes SE and AGC within a single training process. The system operates in the time-frequency domain to transform a noisy, volume-unbalanced input waveform into a clean, volume-balanced output.

Architecture

The framework employs a two-stage architecture:

Speech Enhancement (SE) Module:
- Backbone: Uses MP-SENet as the base architecture.
- Input/Output: Processes the magnitude and phase spectra of the noisy input.
- Training Strategy: The target is clean but volume-unbalanced speech. This forces the SE module to learn noise suppression while preserving loudness variations for the subsequent AGC module.
- Asymmetric Reweighting: To prevent the model from over-suppressing quiet speech, an asymmetric loss strategy is applied. If the predicted magnitude is lower than the target magnitude, the loss contribution is multiplied by a factor of $\alpha = 10.0$ .
Automatic Gain Control (AGC) Module:
- Input: Takes the RMS-normalized enhanced magnitude spectrum from the SE module.
- Architecture: Consists of three components:
  - Frequency-domain processing: Two 2D convolutional layers to extract frequency-aware features.
  - Temporal processing: A 2-layer Bidirectional LSTM (BiLSTM) to handle temporal sequences.
  - Spectral reconstruction: Transposed convolutions to reconstruct the magnitude spectrum.
- Normalization: The module applies peak normalization to 0.4 to achieve a target loudness of -23 LUFS. It uses independent RMS normalization for input and target to learn relative amplitude control.
- Loss Strategy: A conditional weighting mechanism penalizes the model 10x if it predicts positive energy in silent target regions, preventing noise amplification.

Joint Training Objective

The total loss function combines the SE loss and the AGC loss:
$L_{total} = L_{MP-SENet} + \lambda_{AGC} \times L_{AGC}$
where $\lambda_{AGC} = 0.9$ . A curriculum learning strategy is employed, pre-training the SE module for 5 epochs before jointly optimizing the entire framework.

3. Key Contributions

A. SE-AGCNet Framework

A flexible, end-to-end framework that enables joint training of SE and AGC. It supports integration with diverse existing SE architectures and addresses the specific challenges of meeting-room acoustics, including reverberation, background noise, and significant volume discrepancies caused by speaker distance and movement.

B. SE-AGC-DataGen Pipeline

Recognizing the lack of public AGC datasets, the authors developed a comprehensive data simulation pipeline to generate the LibriAGC dataset.

Source: Based on LibriTTS.
Simulation Stages:
1. Concatenation of 2–5 speakers.
2. Volume processing involving basic adjustment and four augmentation modes (sudden spikes, gradual changes, fluctuations) to simulate real-world meeting dynamics.
3. Noise mixing using 35 clips from the DNS Challenge at 5–25 dB SNR to mimic meeting-room acoustics.
Targets: Generates distinct targets for SE (clean, volume-unbalanced) and AGC (clean, volume-balanced) training.

C. Standardized Loudness Evaluation

The paper introduces standardized loudness metrics based on ITU-R BS.1770 and EBU R128 to provide objective and perceptually meaningful evaluation:

Integrated Loudness (LUFS): Set to a target of -23 LUFS.
Short-term Loudness (St LUFS): Uses a 3-second sliding window to avoid gating biases common in meeting scenarios where near-field and far-field speech coexist.
Loudness Range (LRA): Quantifies dynamic range, with 3–6 LU identified as optimal for balanced speech.

4. Experimental Results

Datasets

Experiments were conducted on:

LibriAGC: The simulated dataset (9,487 training utterances, 1,406 test utterances).
Real-world Datasets: MMCSG (CHiME-8 challenge) and AliMeeting-far (far-field portion).

Performance Metrics

Speech Quality: PESQ, SIGMOS (including loudness MOS), and DNSMOS.
AGC Performance: LUFS, St LUFS, and LRA.
Downstream ASR: Word Error Rate (WER) and Character Error Rate (CER).

Key Findings

Loudness Control: SE-AGCNet consistently achieves the target loudness of -23 LUFS with an St LUFS close to the target and an LRA within the optimal 3–6 LU range. In contrast, conventional cascaded approaches (SE + pyagc) or standalone SE models fail to maintain consistent loudness or exhibit excessive LRA.
Speech Quality & ASR:
- On the LibriAGC test set, SE-AGCNet outperforms the original MP-SENet and cascaded baselines in PESQ, SIGMOS, and DNSMOS scores.
- It achieves the lowest WER/CER among all tested systems on real-world datasets.
- The improvement is particularly pronounced for ASR models trained on limited data (e.g., stt_en_conformer_ctc_large), suggesting the framework is highly beneficial for systems sensitive to audio quality variations.
Comparison: SE-AGCNet demonstrates that joint optimization is superior to treating AGC as a separate post-processing step, effectively mitigating the risk of SE misclassifying quiet speech as noise and preventing AGC from amplifying residual noise.

5. Significance

The paper claims that SE-AGCNet provides a principled solution to the limitations of cascaded audio pipelines. By leveraging the synergy between SE and AGC, the framework ensures that the SE module preserves low-volume speech content while the AGC module handles volume normalization. This results in:

Improved Audio Quality: Better noise suppression without over-silencing quiet speakers.
Optimal Loudness: Consistent achievement of target loudness levels suitable for meeting scenarios.
Enhanced Downstream Utility: Significant reductions in recognition errors for Automatic Speech Recognition (ASR) systems.

The authors emphasize the reproducibility of their work through the open-source code and the novel SE-AGC-DataGen pipeline, which addresses the critical gap of missing public AGC datasets. Future work is noted to focus on extending the framework to more complex acoustic conditions and exploring lightweight real-time models.