⚡ electrical engineering

Fast-ULCNet: A fast and ultra low complexity network for single-channel speech enhancement

This paper introduces Fast-ULCNet, an optimized single-channel speech enhancement model that replaces ULCNet's GRU layers with FastGRNNs and employs a trainable complementary filter to mitigate state drifting, achieving comparable performance to the state-of-the-art while reducing model size by over half and latency by 34%.

Original authors: Nicolás Arrieta Larraza, Niels de Koeijer

Published 2026-04-29

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Nicolás Arrieta Larraza, Niels de Koeijer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to listen to a friend talking in a very noisy room. Your brain is doing a fantastic job of filtering out the background clatter to focus on their voice. This paper is about teaching a computer to do the same thing, but with a specific goal: making it fast and light enough to run on small, battery-powered devices like hearing aids or smart speakers, without needing a massive supercomputer.

Here is the story of their solution, Fast-ULCNet, broken down into simple concepts:

1. The Starting Point: The "ULCNet"

The researchers started with a model called ULCNet. Think of ULCNet as a very efficient, compact "noise-canceling robot" that was already the best in its class. It was good at cleaning up speech, but the researchers wanted to make it even faster and smaller so it could run on even tinier chips.

2. The Problem: The "Tired Worker"

To make the robot faster, they swapped out one of its main brain components (called a GRU) for a newer, lighter version called a FastGRNN.

The Analogy: Imagine a worker who is incredibly fast at sorting packages. However, if this worker has to sort packages for 10 seconds, they are great. But if they have to sort packages for 90 seconds straight, they start to get "drifty." They lose track of where they are, their internal notes get messy, and they start making mistakes.
The Discovery: The researchers found that while the new "FastGRNN" worker was super fast, it suffered from state drift. When listening to long audio clips (like a long conversation), the model's internal memory would slowly wander off, causing the audio quality to get worse the longer the clip played.

3. The Fix: The "Complementary Filter" (Comfi-FastGRNN)

To fix the "tired worker," the team invented a new tool called Comfi-FastGRNN.

The Analogy: Think of this like a gyroscope in a smartphone or a navigational compass. If you walk in a circle, a compass might slowly drift and point the wrong way. To fix this, you use a second sensor (like a gyroscope) to constantly check and correct the compass.
The Solution: They added a "trainable complementary filter" to the model. This acts like a constant supervisor that checks the model's internal memory. If the memory starts to drift or get messy over a long conversation, the supervisor gently nudges it back on track. This keeps the model stable, whether the audio is 10 seconds or 90 seconds long.

4. The Result: A Leaner, Faster Machine

By swapping the old brain part for the new "FastGRNN" and adding the "Comfi" supervisor, they created Fast-ULCNet.

Here is what they achieved, according to their tests:

Same Quality: It cleans up noise just as well as the original, top-tier model (ULCNet).
Half the Size: The model is now less than half the size (in terms of memory and parameters) of the original.
Much Faster: It processes audio 34% faster on average.
Stable: Unlike the uncorrected version, it doesn't get "tired" or make mistakes during long conversations.

Summary

The paper is essentially about taking a high-performance noise-canceling algorithm, making it lighter and faster by using a new type of "brain cell," and then adding a smart "safety net" to ensure it doesn't lose its focus during long tasks. The result is a tool that is perfect for small, resource-limited devices that need to work instantly and reliably.

1. Problem Statement

Single-channel speech enhancement is critical for speech recognition and assistive hearing systems, particularly when deployed on resource-constrained embedded devices. These environments demand:

Low Latency: Real-time processing capabilities.
Low Complexity: Minimal memory footprint and computational cost.

While deep learning models (like the state-of-the-art ULLCNet) offer superior audio quality compared to traditional signal processing, they often struggle to meet strict latency and memory constraints. Furthermore, existing lightweight Recurrent Neural Network (RNN) architectures, such as FastGRNN, were observed to suffer from performance degradation over long audio sequences during inference due to internal state drift, a phenomenon not fully addressed in prior literature.

2. Methodology

The authors propose Fast-ULCNet, an adaptation of the ULCNet architecture designed to reduce computational overhead while maintaining performance. The methodology involves three core components:

A. Architecture Replacement (GRU $\to$ FastGRNN)

The original ULCNet uses Gated Recurrent Units (GRUs) for its recurrent layers. Fast-ULCNet replaces these with FastGRNN layers.

FastGRNN: A lightweight RNN design that uses a weighted residual connection and reuses weight matrices for both hidden-state updates and gating. This reduces the parameter count by 2–4 times compared to standard RNNs while maintaining high accuracy.
Integration: The FastGRNN layers are integrated into the ULCNet pipeline, which includes power-law compression, channel-wise feature reorientation, depthwise separable convolutions, and complex ratio masking (CRM) for phase refinement.

B. Addressing State Drift: Comfi-FastGRNN

The authors identified a critical flaw: while FastGRNN is stable during training, its internal state drifts during long inference sessions (e.g., >60 seconds), leading to performance decay.

Cause: The state update equation lacks a contraction guarantee, allowing the hidden state magnitude to accumulate over time.
Solution: The authors introduce Comfi-FastGRNN (Complementary filter FastGRNN). This extends the FastGRNN state update equation by adding a trainable complementary filter.
- It incorporates two trainable scalar parameters ( $\gamma$ and $\lambda$ ).
- $\lambda$ acts as a modulation factor to compensate for drift.
- $\gamma$ controls the balance between the raw hidden state and the drift correction term.
- This mechanism stabilizes the mean RNN state over long sequences without significantly increasing complexity.

C. Model Architecture

The network follows a two-stage design:

Magnitude Estimation: Input features undergo power-law compression and reorientation. A series of depthwise separable convolutions and bidirectional FastGRNN (or Comfi-FastGRNN) layers process the data along frequency and sub-band temporal axes. A fully connected stack predicts a real-valued magnitude mask.
Phase Refinement: A CNN refines the phase using the estimated magnitude mask and the noisy phase, outputting a complex mask via Complex Ratio Masking (CRM) to reconstruct the enhanced spectrogram.

3. Key Contributions

Fast-ULCNet: The first application of FastGRNN layers to single-channel speech enhancement, successfully adapting the ULCNet architecture for ultra-low complexity.
Empirical Discovery of State Drift: The paper provides empirical evidence that FastGRNN performance decays over long audio sequences due to internal state drift, a limitation previously unreported in the context of speech enhancement.
Comfi-FastGRNN: The proposal of a novel trainable complementary filter to mitigate RNN state drift. This is the first implementation of such a filter specifically designed to handle RNN state accumulation in long-sequence inference.
Open Source: The authors released the code and an online demo, facilitating reproducibility and further research.

4. Experimental Results

Experiments were conducted using the Interspeech 2020 Deep Noise Suppression (DNS) Challenge dataset.

Objective Performance (Quality)

Short Sequences (10s): Fast-ULCNet (both standard and Comfi versions) achieved performance on par with the original ULCNet across metrics like DNSMOS (OVRLMOS, SIGMOS, BAKMOS), PESQ, and SI-SDR.
Long Sequences (90s):
- Standard Fast-ULCNet showed significant performance degradation (e.g., OVRLMOS dropped from 3.09 to 2.93; SI-SDR dropped from 16.89 to 13.58).
- Fast-ULCNet with Comfi-FastGRNN successfully mitigated this drift, restoring performance to match the original ULCNet (OVRLMOS: 3.10; SI-SDR: 16.48).

Computational Efficiency

The model was tested on embedded platforms (Raspberry Pi 3 B+ and Arm Cortex-A53):

Parameters: Reduced by >50% (from 0.685M to 0.338M).
MACs (Multiply-Accumulate Operations): Reduced by ~18% (from 2.057M to 1.691M).
Latency (RTF): Achieved a 34% average reduction in processing time.
- Raspberry Pi 3 B+: RTF improved from 0.976 to 0.657.
- Arm Cortex-A53: RTF improved from 0.927 to 0.604.

5. Significance

This work bridges the gap between high-performance deep learning speech enhancement and the strict constraints of embedded hardware. By demonstrating that FastGRNN can be effectively used for speech enhancement if state drift is managed, the paper offers a viable path for deploying high-quality, real-time speech processing on low-power devices. The introduction of the Comfi-FastGRNN mechanism provides a generalizable solution for stabilizing lightweight RNNs in long-duration inference tasks, extending beyond just speech enhancement.