⚡ electrical engineering

ML-Enabled Deformable Matched Filters for Band-Limit Compensation in Free-Space Optics

This paper proposes a neural-network-assisted deformable matched filtering framework that learns residual filter deformations based on compact signal features to effectively compensate for bandwidth-induced pulse distortion in free-space optical CAP modulation systems, significantly outperforming conventional fixed filters without increasing latency.

Original authors: Paul Anthony Haigh

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paul Anthony Haigh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Fixing a Blurry Message

Imagine you are trying to send a clear, crisp message across a room using a flashlight. In a perfect world, the light beam stays tight and focused, hitting exactly where you aim. But in the real world, the air might be hazy, or your flashlight lens might be a bit old and blurry. By the time the light hits the wall, the beam has spread out, overlapping with its neighbors. If you are trying to send a series of dots and dashes (like Morse code), those blurred spots start to mash together, making it hard to tell where one dot ends and the next begins.

In the world of Free-Space Optics (sending data via light through the air), this "blurring" is caused by bandwidth limitations. It's like trying to squeeze a wide river into a narrow pipe; the water (the data signal) gets distorted, splashed, and delayed.

The Old Way: The Rigid Template

Traditionally, engineers use a "Matched Filter" to catch these signals. Think of this filter as a cookie cutter.

The Problem: The cookie cutter is designed for a perfect, round cookie. But if the dough (the signal) gets squished and stretched by the narrow pipe (the bandwidth limit), the round cookie cutter no longer fits. It cuts off the edges or misses the center, resulting in a messy cookie (bad data).
The Limitation: You can't just swap the cutter for a different shape every time the wind blows, because that would be too slow and complicated. So, engineers usually just accept the messy cookie or try to fix it after it's been cut, which is often too late.

The New Solution: The "Smart, Stretchy" Cookie Cutter

This paper proposes a new system called a Deformable Matched Filter (DMF) powered by Machine Learning.

Instead of using a rigid, unchangeable cookie cutter, imagine a cookie cutter made of smart, stretchy rubber.

The "Senses" (Features): Before the cutter touches the dough, a small team of sensors (the neural network) looks at the dough. They don't look at the whole messy pile; they check 16 specific clues, like "how bumpy is the surface?" or "how wide is the spread?" These are called signal features.
The "Brain" (Neural Network): A tiny, fast computer brain looks at those 16 clues and instantly says, "Okay, the dough is squished to the left, so I need to stretch the cutter's left side and shrink the right side."
The "Action" (Residual Correction): The system doesn't throw away the original perfect cookie cutter. Instead, it calculates a tiny adjustment (a "residual") to the shape of the cutter. It's like taking the perfect cutter and gently bending it just enough to fit the current mess.
The Result: The cutter snaps onto the distorted signal perfectly, extracting the data cleanly without the blurring.

How They Tested It

The researchers built a real-life test lab on a table.

The Setup: They used a laser to send light across a 20-centimeter gap (about 8 inches).
The Challenge: They artificially made the connection "bad" by using digital filters to simulate a very narrow pipe (severe bandwidth limits).
The Comparison: They compared their "Smart Stretchy Cutter" against the old "Rigid Cutter."

What They Found

When the signal is very distorted: The old rigid cutter failed miserably, producing a "messy cookie" (high error). The smart cutter, however, bent itself to fit the distortion and kept the data clean. It improved performance by more than 40% in the worst conditions.
When the signal is good: If the connection was clear and the bandwidth wasn't a problem, the smart cutter realized, "Hey, I don't need to bend." It stayed almost exactly like the original rigid cutter. This is important because it means the new system doesn't break things when they are already working well.
Speed: The system learns and adjusts so fast that it doesn't slow down the data transmission.

Why This Matters

The paper shows that you don't need to replace the entire communication system with a complex, black-box computer program. Instead, you can keep the reliable, proven math of the old system and just add a "smart helper" that makes tiny, necessary adjustments on the fly.

In short: They taught a computer to gently bend a standard data filter so it fits perfectly, even when the light signal is distorted by a narrow channel, resulting in much clearer communication without needing to slow down or change the whole system.

Technical Summary: ML-Enabled Deformable Matched Filters for Band-Limit Compensation in Free-Space Optics

Problem Statement
Carrier-less amplitude and phase (CAP) modulation is a spectral-efficient solution for bandwidth-constrained optical communication systems. However, its performance is highly sensitive to pulse-shape distortions caused by bandwidth limitations, atmospheric turbulence, and hardware impairments (e.g., photodetector saturation). Traditional CAP receivers rely on fixed, analytically derived matched filters (CMF) designed for ideal channels. In practice, bandwidth constraints cause rapid variations in signal statistics, leading to a mismatch between the nominal filter and the optimal receiver structure. While adaptive equalization techniques exist, they often require bandwidth-consuming training sequences, frequent retraining, or operate at the symbol level, where they cannot directly compensate for pulse-shape distortions that occur prior to sampling. Furthermore, existing machine learning (ML) approaches for optical receivers typically operate on high-dimensional waveforms or symbol sequences, often lacking physical interpretability and requiring substantial computational resources.

Methodology
The paper proposes a Neural-Network-Assisted Deformable Matched Filtering (DMF) framework. Rather than replacing the analytically derived CAP matched filter with a "black-box" model, the proposed receiver learns a residual deformation of the nominal filter.

Feature Extraction: The system extracts a compact set of 16 physically motivated signal features from the received waveform. These features are averaged across four temporal segments to ensure robustness. They are categorized into:
1. Time-domain statistics: RMS amplitude, variance, skewness, excess kurtosis, and mean envelope.
2. Frequency-domain characteristics: Spectral spread, roll-off, flatness, entropy, and 3 dB bandwidth.
3. Signal quality metrics: Peak-to-average power ratio (PAPR), peak power, autocorrelation at specific lags, and envelope crest factor.
Neural Network Architecture: A lightweight, fully connected neural network (two layers: 16 → 256 → 384 neurons) maps the 16-dimensional feature vector to complex-valued filter correction terms ( $\Delta h$ ). The network predicts residual corrections to the baseline CMF coefficients rather than learning filters from scratch.
Filter Adaptation: The final deformable filters are obtained by adding the predicted corrections to the analytically derived time-reversed filters. The resulting filters are energy-normalized before application.
Training: The network is trained end-to-end using a hardware-in-the-loop (HITL) setup. The loss function is a weighted combination of Error Vector Magnitude (EVM) and smoothness regularization penalties (first and second-order) to ensure physically realizable, non-oscillatory filter coefficients. Training utilizes online data generation with systematic bandwidth variation, avoiding fixed training sets to prevent overfitting to specific symbol patterns.

Experimental Setup
The validation was performed using a CAP transmission system operating at 850 nm over a 200 mm free-space optical link. The system utilized a Xilinx Zynq ZCU102 board with an Analog Devices FMC-DAQ3 for DAC/ADC conversion. Bandwidth limitation was emulated in the digital domain via a 101-tap FIR filter with normalized cut-off frequencies ( $\omega_n$ ) ranging from 0.5 to 0.9. The system tested 4-QAM and 16-QAM modulation formats across received optical powers from -25 dBm to +5 dBm.

Key Results

Performance under Severe Constraints: Under severe bandwidth limitation ( $\omega_n = 0.5$ ), the conventional matched filter (CMF) exhibited EVM values exceeding 50%, rendering demodulation infeasible. The proposed DMF reduced EVM to below 30% under identical conditions, representing a relative performance improvement of over 40%.
Intermediate Regime: In the transition region ( $\omega_n \in [0.65, 0.75]$ ), the DMF provided consistent EVM improvements of 5–15% over the CMF.
Graceful Degradation: As bandwidth constraints were relaxed ( $\omega_n \ge 0.85$ ), the DMF performance converged with the CMF. The neural network learned correction terms close to zero, effectively defaulting to the analytically optimal filter, thereby avoiding performance degradation in favorable conditions.
Comparison with Equalizers:
- A 192-tap LMS linear equaliser operating at the symbol rate provided no measurable improvement over the CMF, confirming that pulse-shape distortion is irreversibly lost after decimation.
- A Multi-Layer Perceptron (MLP) post-equaliser operating at the symbol rate matched or was outperformed by the DMF in the Inter-Symbol Interference (ISI)-dominated regime (moderate to severe bandwidth limits). While the MLP showed marginal advantages in noise-dominated regimes, the DMF offers a superior complexity-performance trade-off by operating pre-decimation without requiring pilot symbols or periodic retraining.
Generalization: The DMF demonstrated robustness across modulation formats (4-QAM and 16-QAM) and showed the ability to generalize when a model trained on 16-QAM was applied to 4-QAM without retraining.

Significance and Claims
The paper claims that the proposed DMF framework successfully bridges the gap between classical communication theory and data-driven optimization. Its primary significance lies in:

Hybrid Strategy: It combines the reliability and interpretability of analytically derived matched filters with the adaptability of machine learning by learning residual corrections rather than replacing the entire receiver structure.
Pre-Decimation Intervention: By adapting the matched filter itself prior to symbol-rate sampling, the method compensates for pulse-shape distortions that symbol-rate equalizers cannot recover.
Efficiency and Interpretability: The use of a compact, physics-informed feature set allows for a small neural network architecture (102,528 parameters) that ensures rapid convergence and maintains physical interpretability, unlike high-dimensional waveform-based deep learning models.
Robustness: The system demonstrates significant performance gains under severe bandwidth constraints while gracefully degrading to conventional performance when conditions are favorable, eliminating the risk of ML-induced instability.

The authors conclude that this approach offers a practical, pilot-free, and block-adaptive solution for bandwidth-limited optical links, particularly in scenarios where ISI is the dominant impairment. Future work is identified as investigating atmospheric impairments (turbulence, scintillation) and comparing the DMF against fractionally spaced equalizers and complete DSP chains.