⚡ electrical engineering

Audio-Visual Speech Enhancement: Architectural Design and Deployment Strategies

This paper presents the design and evaluation of a real-time cloud-edge audio-visual speech enhancement system deployed on a public 5G network, demonstrating that while edge compute placement and aggressive compression are critical for meeting latency constraints, uplink capacity remains a dominant bottleneck and a fundamental trade-off exists between processing speed and enhancement quality.

Original authors: Anis Hamadouche, Haifeng Luo, Mathini Sellathurai, Amir Hussain, Tharm Ratnarajah

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anis Hamadouche, Haifeng Luo, Mathini Sellathurai, Amir Hussain, Tharm Ratnarajah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to have a conversation with a friend at a very loud, chaotic rock concert. You can barely hear them, but if you watch their lips move, you can understand what they are saying much better. This is the magic of Audio-Visual Speech Enhancement (AVSE). It's like giving a superpower to your ears by adding your eyes to the mix.

However, doing this in real-time on a mobile device (like a smart hearing aid or glasses) is tricky. Your device is small and has a weak battery, but the math required to "read lips" and "clean up sound" is heavy. So, the solution is to send the video and audio to a powerful computer in the cloud to do the heavy lifting, and then send the clean sound back to you.

The Problem: The journey back and forth takes time. If the computer is too far away (like in a different country) or the road is too crowded (slow internet), the sound comes back too late. You'd hear your friend's mouth move, but their voice would arrive a second later. That's annoying and breaks the illusion of a real conversation.

The Solution in This Paper:
The authors built a "test drive" for this technology using 5G networks and Edge Computing. Think of "Edge Computing" not as a giant warehouse of computers far away, but as a local coffee shop right next to the stadium. Instead of sending your data to a server in another city, you send it to the server right next to the 5G tower.

Here is a breakdown of their findings using simple analogies:

1. The "Road Trip" Analogy (Network Latency)

Imagine you are sending a package (your voice and video) to a warehouse (the server) to get it wrapped (enhanced) and sent back.

The Old Way (Wi-Fi/4G/Internet Cloud): You are stuck in heavy traffic on a highway. The package takes a long time to get there and back. By the time it returns, the conversation has moved on.
The New Way (5G Edge): You are on a private, empty track right next to the warehouse. The package arrives almost instantly.
The Finding: The paper found that 5G and wired Ethernet are the only roads fast enough to keep the conversation flowing naturally. Older networks (like 4G or standard Wi-Fi) were too slow and "bumpy," causing the audio to stutter or arrive out of sync.

2. The "Heavy Suitcase" Analogy (Data Compression)

Sending raw video and audio is like trying to mail a suitcase filled with rocks. It's heavy, takes up a lot of space on the truck (bandwidth), and moves slowly.

The Fix: The researchers figured out how to pack that suitcase into a tiny, lightweight box without breaking the contents. They used compression to shrink the data size by 80 times (like turning a suitcase into a small envelope).
The Result: Even though the "box" was smaller, the quality of the sound and video remained almost perfect. This allowed the system to work even when the "road" was narrow or crowded.

3. The "Speed vs. Quality" Trade-off (Processing Latency)

This is the most critical part of the study. Imagine you are a chef (the AI) trying to cook a perfect meal (clean speech).

The Slow Chef: If you give the chef a huge pile of ingredients (a long chunk of video/audio) and a fancy, complex recipe (a big AI model), the meal is delicious and perfect. But it takes a long time to cook.
The Fast Chef: If you give the chef a tiny pile of ingredients and a simple recipe, the food comes out in seconds. But it might taste a bit bland or miss some flavors, especially if the ingredients were already a bit spoiled (very noisy).
The Finding: The researchers found a "sweet spot." If they made the AI too simple to make it faster, the sound quality dropped significantly in noisy environments. If they made it too complex, the delay became too long for real-time chat. They had to find a balance where the "chef" was fast enough to keep up with the conversation but smart enough to still clean up the noise.

4. The "Stress Test" (Real-World Conditions)

They didn't just test this in a perfect lab; they tested it in the real world with Vodafone's 5G network in the UK. They simulated crowded conditions (like a busy concert) to see if the system would crash.

The Verdict: The system held up well! As long as the "Edge" server was close (in the same city) and the data was compressed, the system could handle the stress. However, if they pushed it too hard (trying to send huge, uncompressed video), the system started to lag.

The Big Takeaway

This paper proves that we can finally have real-time, "super-hearing" conversations on our phones and wearables, but only if we use 5G Edge networks.

Don't use the old internet: It's too slow and unpredictable.
Do use the "local" cloud: Processing data right next to the cell tower is the key.
Pack light: Compressing the data is essential to keep things moving fast.
Balance is key: You can't have the fastest speed and the absolute best quality at the same time; you have to find a happy medium.

In short, this technology is the bridge that will allow future smart glasses and hearing aids to help us hear clearly in noisy rooms, provided we build the right "roads" (5G Edge) to get the data there and back in time.

1. Problem Statement

Real-world speech signals are often degraded by background noise, reverberation, and competing speakers, rendering traditional audio-only enhancement techniques ineffective, particularly in low Signal-to-Noise Ratio (SNR) scenarios. While humans naturally utilize visual cues (e.g., lip movements) to disambiguate speech, Audio-Visual Speech Enhancement (AVSE) systems face significant hurdles in real-time deployment:

Resource Constraints: Wearable and mobile devices (e.g., hearing aids, smart glasses) lack the computational power to run complex deep learning models locally.
Latency and Synchronization: Offloading processing to the cloud introduces network latency, jitter, and packet loss. For AVSE to be effective, the system must maintain strict temporal coherence between audio and video streams. If the round-trip communication delay exceeds the duration of the data chunk being processed, the user experience degrades (e.g., "blank" periods in audio or desynchronized lip-sync).
Network Variability: Public 5G networks, while promising, introduce variability in throughput and latency that can disrupt real-time multimedia services.

2. Methodology and System Architecture

The authors propose a cloud–edge–assisted AVSE system deployed on a public 5G edge network (Vodafone-compatible AWS Wavelength).

A. Model Architecture

The system employs a multimodal deep learning pipeline:

Audio Stream: A Convolutional Neural Network (CNN) extracts acoustic features (e.g., log-mel or STFT magnitude) from the noisy input.
Visual Stream: OpenCV-based facial feature extraction (using Haar cascade classifiers and CNNs) isolates lip-region motion embeddings.
Fusion & Temporal Modeling: The audio and visual embeddings are concatenated and fed into a Long Short-Term Memory (LSTM) network to model temporal dependencies and ensure coherent speech reconstruction over time.
Decoding: Fully Connected (FC) layers decode the enhanced speech features.

B. Deployment Strategy

Client-Server Model: Client devices (wearables/IoT) capture raw audio/video and offload all processing to the edge server. This allows low-power devices to function without local compute requirements.
Buffering Mechanism: To handle processing delays, the system uses input/output buffers. The server processes data in chunks ( $t_{chunk}$ ) and feeds back enhanced audio only when sufficient data is available, ensuring the user receives a coherent stream despite algorithmic latency.
Edge Location: The system is hosted on AWS Wavelength, which embeds compute resources within the 5G radio access network (RAN) to minimize transport distance.

C. Experimental Setup

Networks Tested: Ethernet (direct), Wi-Fi (802.11n), 4G, Private 5G, Public 5G (Vodafone UK), and AWS over the public Internet.
Stress Profiles: The study defined 11 multimedia profiles (Table I) varying in video resolution, frame rate, JPEG compression, and audio configuration to simulate loads from "Audio-only" to "Extreme" stress.
Compression: Aggressive data compression (up to 80x) was tested to reduce payload size (from ~0.3 MB to ~70 KB) while maintaining perceptual quality.

3. Key Contributions

Real-World 5G Edge Deployment: The first end-to-end evaluation of an AVSE system on a public 5G edge cloud (AWS Wavelength), moving beyond theoretical models to practical system implementation.
Stress-Adaptive Profiles: Introduction of a systematic framework to evaluate throughput, latency, and quality trade-offs across heterogeneous network conditions.
Quantitative Performance Analysis: Comprehensive measurement of Round-Trip Time (RTT), uplink throughput, packet loss, and CPU utilization under varying loads.
Design Guidelines: Derivation of practical insights for latency-sensitive multimedia services, highlighting the critical roles of edge proximity, adaptive compression, and multi-modal synchronization.

4. Key Results

A. Network Latency and Coherence

Critical Threshold: Real-time coherence requires communication delay ( $t_{comm}$ ) to be less than the chunk duration ( $t_{chunk} = 40$ ms).
Performance by Network:
- Ethernet & Private 5G: Consistently met the $<40$ ms requirement (Latency: 5–35 ms).
- Public 5G (AWS Wavelength): Achieved moderate latency (54–73 ms). While acceptable for many applications, it operates with reduced margins compared to private networks.
- Wi-Fi 4 & 4G: Failed to meet requirements consistently (Latency: 50–150 ms), often causing desynchronization.
- Public Internet (AWS over WebRTC): Highest latency (80–200 ms), rendering it unsuitable for real-time AVSE.
Conclusion: Edge proximity is the dominant factor. Public 5G with edge cloud is a viable middle ground but requires careful orchestration.

B. Compression and Uplink Capacity

Bottleneck: Uplink capacity is often the limiting factor for interactive AVSE.
Compression Impact: Aggressive compression (factor of 80) reduced payload size by 80x (to ~70 KB) with negligible perceptual degradation. This enabled robust operation under constrained uplink conditions where uncompressed data failed.

C. Latency vs. Quality Trade-off

Processing Latency ( $t_{proc}$ ): The study analyzed the trade-off between model complexity/chunk size and enhancement quality.
- Longer Chunks (e.g., 10s): Provided rich temporal context for lip-tracking, yielding high-quality noise suppression but high latency (~1.2s).
- Shorter Chunks/Lightweight Models: Reduced latency significantly (down to ~0.35s) but degraded reconstruction quality, especially in low-SNR scenarios where temporal context is vital for speaker discrimination.
Finding: There is a fundamental trade-off; reducing model complexity lowers delay but sacrifices enhancement fidelity.

5. Significance and Conclusion

This paper demonstrates that public 5G edge environments can sustain real-time, interactive AVSE workloads, provided that:

Compute is placed at the edge (MEC) to minimize transport latency.
Adaptive profiles are used to balance bandwidth, latency, and quality (e.g., switching to audio-only or compressed video during network congestion).
Uplink capacity is prioritized, as it is often the primary bottleneck.

The study provides a blueprint for deploying delay-sensitive, multimodal services (such as augmented reality, assistive hearing, and interactive conferencing) on emerging 5G infrastructure. It highlights that while public 5G offers a practical solution, performance margins are tighter than in dedicated infrastructures, necessitating dynamic adaptation strategies to ensure a consistent Quality of Experience (QoE).