🤖 AI

BASENet: Band-Adapted Speech Enhancement Network with Cross-Band Attention

BASENet is a highly efficient, frequency-adapted speech enhancement network that leverages Bark-scale band partitioning and cross-band attention to achieve state-of-the-art performance with minimal parameters, making it ideal for real-time deployment on resource-constrained devices.

Original authors: Damien Martins Gomes, François Capman

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Damien Martins Gomes, François Capman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to clean up a muddy photo of a city skyline. Most computer programs treat every part of the photo the same way: they apply the same amount of "cleaning power" to the sky, the buildings, and the tiny details on the windows. But human eyes don't work that way. We notice tiny details in the center of our vision but are less sensitive to the edges.

BASENet is a new computer program designed to clean up noisy speech (like a voice on a bad phone call), but instead of treating all sound frequencies equally, it mimics how human ears actually work.

Here is how it works, broken down into simple concepts:

1. The "Human Ear" Rulebook

Our ears are not uniform.

Low sounds (like a deep drum or a man's voice) are very important. Our ears can hear tiny differences in these sounds, so we need to pay close attention to them.
High sounds (like a whistle or a hiss) cover a wider range, but our ears are less sensitive to the tiny details there.

Most old speech programs use a "one-size-fits-all" approach. They give the same amount of computing power to low sounds and high sounds. This is like hiring the same number of detectives to investigate a massive bank robbery and a lost set of keys. It's a waste of resources.

2. The "Band-Adapted" Solution

The authors created BASENet, which acts like a smart manager who assigns workers based on the difficulty of the job.

The Low Frequencies (The Busy District): Because our ears are very sensitive here, BASENet assigns a deep, heavy team of workers to this part of the sound. It digs deep to fix the complex harmonics and pitch.
The High Frequencies (The Quiet District): Because our ears are less sensitive here, it assigns a lighter, faster team. They do the job quickly without wasting energy.

This is done automatically using a mathematical rule based on the "Bark scale" (a way scientists measure how humans hear). The computer calculates exactly how much "attention" each slice of sound needs and builds a custom team for it.

3. The "Group Chat" (Cross-Band Attention)

Even though the teams work on different parts of the sound separately, they need to talk to each other. Speech is like a choir; the low notes and high notes are connected.

Imagine the low-frequency team is the bass singer, and the high-frequency team is the soprano. If the bass singer changes their note, the soprano needs to know to stay in tune.
BASENet has a special "Cross-Band Attention" module. Instead of making every single worker talk to every other worker (which would be slow and chaotic), they send a quick summary to a central "group chat."
This allows the different teams to share information about the "big picture" (like the rhythm or the shape of the voice) very quickly, without slowing down the computer.

4. The Result: Fast and Clear

The paper tested this system on a standard dataset called VoiceBank+DEMAND.

Quality: It produced very clear speech (a score of 3.55 out of 5 on a quality scale called PESQ).
Efficiency: It did this using very few resources (only 0.83 million parameters). To put that in perspective, it is much smaller and faster than other top-tier models that achieve similar quality.
Real-Time: Because it is so efficient, it can run in real-time. The authors even made a "causal" version (which only looks at the past, not the future) that works almost as well as the non-real-time versions. This means it could be used on devices like hearing aids or live phone calls without lag.

Summary

Think of BASENet as a smart audio janitor. Instead of scrubbing the whole floor with the same intensity, it knows exactly where the dirtiest spots are (the low frequencies) and scrubs those hard, while giving the cleaner spots (the high frequencies) a quick wipe. It also has a walkie-talkie system so all the janitors stay coordinated. The result is a clean voice that sounds natural, achieved with a fraction of the computing power required by older methods.

Technical Summary of BASENet: Band-Adapted Speech Enhancement Network with Cross-Band Attention

1. Problem Statement

Speech enhancement models traditionally apply uniform computational capacity across the entire frequency spectrum. This approach disregards the non-uniform spectral resolution of the human auditory system, where low-frequency critical bands are narrow and perceptually dense (encoding fundamental frequency and harmonics), while high-frequency bands are wider with lower perceptual acuity. Existing state-of-the-art (SOTA) models often rely on expensive full-band self-attention or bidirectional operations that hinder real-time deployment. Furthermore, many sub-band approaches either apply uniform capacity across all bands or rely on shared recurrent layers without modulating encoder depth based on perceptual importance. There remains a need for an architecture that systematically derives capacity from auditory principles while maintaining cross-band coherence and supporting causal, real-time inference.

2. Methodology

The proposed BASENet (Band-Adapted Speech Enhancement Network) is a frequency-adapted architecture designed to align encoder capacity with human auditory perception.

Perceptually Motivated Frequency Decomposition:
The input spectrogram is partitioned into $B$ non-overlapping frequency bands based on the Bark scale. The boundaries are determined by transitions between perceptually distinct regions (fundamental/harmonic, formant, fricative). For each band $b$ , a critical-band density ( $\rho_b$ ) is calculated, representing the number of Bark units spanned per Hertz. This metric quantifies the perceptual density of the band.
Density-Driven Capacity Allocation:
Unlike uniform architectures, BASENet assigns encoder depth ( $L_b$ ) as a closed-form function of the critical-band density:
$L_b = \left\lfloor L_{\max} \cdot \frac{\rho_b}{\max_{b'} \rho_{b'}} \right\rceil$
This rule ensures that perceptually dense low-frequency bands receive deeper, wider branches to model fine-grained harmonic structures, while broader high-frequency bands are processed by shallower, lighter branches. This eliminates the need for per-band hyperparameter tuning.
Scaled-Capacity Band Processing:
Each band is processed by a dedicated encoder branch of depth $L_b$ . The architecture utilizes inverted residual blocks with dense connectivity and exponentially dilated convolutions along the frequency axis. Deeper branches develop larger receptive fields to capture pitch and harmonic details in low frequencies, while shallower branches handle spectrally smoother high-frequency content efficiently.
Cross-Band Attention:
To restore coherence between independently processed bands, a Cross-Band Attention module is introduced. Instead of attending to full time-frequency grids (which incurs $O(N^2F^2)$ complexity), the module computes compact global contexts by averaging keys and values across frequency bins within each band. These band-level summaries are concatenated, and each band attends to this shared context. This mechanism operates at linear complexity $O(NF_bB)$ and supports frame-by-frame causal inference.
Temporal Modelling and Decoders:
After cross-band fusion, features are processed by a Convolutional Recurrent Network (CRN) using a unidirectional or bidirectional GRU for temporal modelling. The network follows the mask-and-phase estimation paradigm, predicting a time-frequency magnitude mask and an enhanced phase spectrogram to reconstruct the clean signal.

3. Key Contributions

Perceptually Scaled Encoder: A novel design rule that derives per-band capacity directly from Bark-scale critical-band density via a single closed-form equation, removing the need for manual per-band tuning.
Efficient Cross-Band Attention: A module that models harmonic and formant dependencies across bands using compact frequency-pooled representations, achieving linear complexity and enabling real-time causal processing.
Lightweight, Streaming-Ready Architecture: A model with only 0.83 M parameters and 7.3 G MACs that achieves high-quality enhancement. It natively supports causal inference by simply replacing the bidirectional GRU with a unidirectional variant, requiring no architectural redesign.

4. Experimental Results

The model was evaluated on the VoiceBank+DEMAND dataset.

Performance: BASENet-3 (using 3 bands) achieved a PESQ of 3.55 and STOI of 96%.
Efficiency Comparison: Among all methods achieving PESQ $\ge$ 3.50, BASENet has the fewest parameters (0.83 M). It matches the performance of SE-Mamba (PESQ 3.55) with 4.5 $\times$ fewer MACs and 2.7 $\times$ fewer parameters. It also surpasses MP-SENet (PESQ 3.50) with significantly lower computational cost.
Causal Variant: The causal version of BASENet (0.81 M parameters) achieved a PESQ of 3.44, outperforming several non-causal baselines (e.g., CMGAN at 3.41, DPT-FSNet at 3.33), demonstrating its suitability for real-time streaming on resource-constrained devices.
Ablation Studies: Removing cross-band attention caused the largest performance drop (-0.13 PESQ), confirming its critical role in modelling harmonic relationships. Replacing the density-driven depth allocation with uniform depth also resulted in a significant drop (-0.11 PESQ). Finer band partitions (8 or 12 bands) degraded performance, suggesting that 3 bands provide the optimal balance of spectral context.

5. Significance

The paper claims that BASENet demonstrates that perceptually scaled capacity is a highly effective inductive bias for speech enhancement. By systematically aligning model depth with auditory resolution, the architecture achieves a superior quality-efficiency trade-off. The work addresses the tension between high performance and computational constraints, offering a solution that is both lightweight and capable of native causal streaming, making it suitable for applications such as hearing aids and live voice communication where real-time inference is critical. Future work is suggested to explore lower-triangular attention masks to further exploit the bottom-up harmonic structure of speech.