⚡ electrical engineering

Explainable deformable matched filtering reveals measurable departures from classical receiver theory in optical wireless communications

This paper introduces an explainable deformable matched-filter framework that uses machine learning to learn low-dimensional corrections to classical matched filters based on physical receiver states, thereby improving optical wireless communication performance while providing measurable insights into the specific limitations of classical receiver theory.

Original authors: Paul Anthony Haigh

Published 2026-09-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Anthony Haigh

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the invisible highways of modern life, data travels as pulses of light, zipping through fiber-optic cables or bouncing through the air in wireless links. To make sense of this flood of information, engineers rely on a mathematical tool called a matched filter. Think of this tool as a perfectly tuned ear that listens for a specific sound pattern. If the sound arrives exactly as expected, the filter catches it with maximum clarity. This concept is a cornerstone of communication theory, a rule that has guided the design of receivers for decades. However, the real world is rarely perfect. As light travels, it gets distorted by the hardware that sends and receives it, by the limits of the equipment's speed, and by the quirks of the environment. These imperfections mean the signal arriving at the receiver is never quite the same as the one that left the transmitter. For years, engineers have treated these distortions as mere engineering glitches to be fixed, rather than clues about why the perfect theory fails in practice.

A researcher at Queen Mary University of London has now taken a fresh look at this gap between theory and reality. Instead of trying to replace the classic matched filter with a completely new, mysterious machine-learning system, they built a framework that keeps the old filter at the center but allows it to bend. They call this approach a deformable matched filter. In their experiments, they used a machine-learning model not to do the heavy lifting of decoding the signal, but to act as a guide. This guide looks at the current state of the receiver—how the signal is behaving right now—and suggests a small, specific adjustment to the filter's shape. The filter then applies this tweak before processing the data. By doing this, the researcher turned the filter itself into a measuring device. The amount and type of bending the filter requires to work well became a direct, readable record of exactly how the real-world conditions were breaking the rules of classical theory.

The researcher tested this idea on a free-space optical communication link, a setup where light travels through the air between a transmitter and a receiver over a distance of about twenty centimeters. They ran the system through a vast array of conditions, testing ten different ways of encoding data and four distinct types of signal distortion. Across 1,600 different scenarios, the deformable filter consistently outperformed the standard, unchanging filter. On average, it reduced the error in the received signal by 18.1 percent. But the true discovery was not just that the system worked better, but what the adjustments revealed. The researcher found that the filter did not bend in the same way for every type of signal. Different families of data formats, such as pulse amplitude modulation and carrier-less amplitude and phase modulation, required their own unique patterns of adjustment. This proved that the mismatch between theory and reality is not a single, generic problem. Instead, it is a structured phenomenon that depends heavily on the specific design of the signal being sent.

One of the most revealing findings came from comparing two similar signal types: duobinary and modified duobinary. The standard duobinary signal required the filter to make a large, low-frequency adjustment even when the system was running under normal conditions. This indicated that the mismatch was built into the very nature of that signal's interaction with the channel. When the researcher switched to the modified duobinary signal, which was designed to avoid those problematic low frequencies, the filter barely needed to bend at all. The error rate dropped, and the required adjustment vanished. This showed that the deformable filter could pinpoint exactly which part of a signal design was causing the trouble. It acted less like a generic equalizer and more like a diagnostic tool, identifying the specific spectral components that were failing to match the theoretical assumptions.

The researcher also looked at how the system reacted when they artificially limited the bandwidth, simulating a scenario where the equipment could not handle the full speed of the signal. As the bandwidth shrank, the filter's required adjustment grew in a smooth, predictable way. It began to suppress the low-frequency parts of the signal that were no longer supported by the channel, while carefully avoiding the high-frequency parts that were too weak to recover. This behavior demonstrated that the machine learning was not just guessing; it was learning a physical response to a physical constraint. The adjustments were not random noise but a coherent strategy to reshape the filter to match the reality of the channel.

By analyzing the internal state of the learning model, the researcher discovered that the system organized itself around the type of signal being used. The different signal families occupied distinct regions in the model's internal map, while the various types of distortion simply shifted the system within those regions. This confirmed that the primary factor determining how a receiver fails is the structure of the signal itself, not just the severity of the noise or distortion. The complexity of the required adjustment also varied; some signal types needed only a simple tweak, while others required a complex reshaping involving many different components.

This work establishes a new way to use machine learning in communication systems. Rather than hiding the decision-making process inside a black box, the researcher kept the classical theory visible and used the learning algorithm to measure the distance between that theory and the real world. The result is a framework where every improvement in performance is tied to a measurable departure from the ideal. It shows that the gap between analytical theory and practical systems is not a flaw to be ignored, but a landscape that can be mapped and understood. By treating the learned correction as a scientific observation rather than just a performance boost, the researcher has provided a method to investigate the limits of classical receiver theory, revealing that the way a system adapts tells us as much about the physics of the channel as it does about the quality of the connection.

Technical Summary: Explainable Deformable Matched Filtering in Optical Wireless Communications

Problem Statement
Matched filtering is a foundational result in communication theory, providing the optimal linear receiver for signals in additive white Gaussian noise (AWGN) when the received waveform is a noisy version of a known transmit pulse. However, practical optical wireless communication (OWC) systems rarely satisfy the strict assumptions required for matched-filter optimality. Factors such as transmitter bandwidth limitations, receiver filtering, AC coupling, device nonlinearities, and hardware implementation distort the received waveform, creating a systematic mismatch between the theoretical matched filter and the filter that is optimal in practice.

While modern receivers often employ machine learning (ML) to outperform classical matched filters (CMF), these "neural receivers" typically replace the classical architecture entirely. Consequently, they function as black boxes that optimize symbol recovery but offer little insight into why the classical theory fails or what specific physical mechanisms drive the performance gap. This leaves a critical gap: if a learned receiver outperforms the CMF, the improvement itself remains unexplained, and the nature of the receiver mismatch remains opaque.

Methodology
To address this, the author introduces an Explainable Deformable Matched Filter (DMF) framework. Rather than replacing the matched filter with a neural network, the framework preserves the CMF as the central processing element and uses ML to learn a constrained, low-dimensional deformation of it.

Architecture: The communication waveform is processed exclusively by the matched filter. A Kolmogorov-Arnold Network (KAN) does not process the waveform directly; instead, it takes a compact set of 24 physically interpretable receiver-state descriptors (e.g., RMS amplitude, spectral entropy, autocorrelation) extracted from the signal and predicts a set of deformation coefficients.
Deformation Mechanism: The final filter is expressed as $h_{DMF} = h_{CMF} + \Delta h$ , where $\Delta h$ is a learned deformation represented by a low-dimensional expansion using a Discrete Cosine Transform (DCT) basis (24 components for a 192-tap filter).
Interpretability: Because every learned parameter is defined relative to the theoretical matched filter solution, the deformation serves as a measurable representation of receiver mismatch. Persistent non-zero deformations indicate a departure from the assumptions underlying classical theory.
Experimental Setup: The framework was validated on a free-space optical (FSO) testbed (IM/DD, 850 nm, ~600 MHz bandwidth). The study evaluated 10 signaling formats (PAM, CAP, Duobinary, Modified Duobinary) across 1,600 conditions, including four impairment classes: baseline, bandwidth-limited, non-linear, and combined impairments.

Key Results

Performance Improvement: The DMF consistently reduced the Error Vector Magnitude (EVM) across all signaling formats and impairment conditions. The median relative EVM reduction was 18.1% (interquartile range 10.9–29.7%), with gains ranging from 0.9% to 82.6%. Bandwidth-limited conditions yielded the largest median improvement (30.8%).
Signaling-Specific Regimes: The study found that different signaling families occupy distinct "deformation regimes." There is no universal deformation-to-gain relationship. For instance, Carrier-less Amplitude and Phase (CAP) modulation showed a strong monotonic relationship between deformation magnitude and performance gain, particularly under bandwidth constraints. In contrast, Duobinary (DB) formats exhibited large deformations even under baseline conditions, suggesting intrinsic mismatch between the signaling structure and the channel.
Spectral Mechanisms: Frequency-domain analysis revealed that the learned deformations respond systematically to physical impairments. Under bandwidth limitations, the DMF progressively attenuated excess low-frequency energy (below ~0.1 normalized frequency) rather than attempting unstable inversion of high-frequency regions. This confirms the deformation adapts to the specific spectral constraints of the channel.
Latent State Organization: Analysis of the KAN's latent receiver state showed that signaling family is the dominant organizing factor, not the specific impairment type. While impairments cause systematic shifts within these family-dependent regions, they do not form globally separable clusters. This indicates that receiver mismatch is structured by the interaction between the signaling format and the channel, rather than being a simple function of impairment severity.
Complexity: The effective rank of the learned deformation coefficients varied by signaling format and impairment, ranging from 1.15 (DB) to 7.42 (CAP). This demonstrates that mismatch cannot be corrected by a single scalar gain or bandwidth adjustment but requires coordinated changes across multiple deformation modes.

Significance and Claims
The paper posits that the primary contribution is not a new neural receiver architecture that simply outperforms the CMF, but rather an interpretable framework for experimentally investigating the gap between analytical receiver theory and practical systems.

Measuring Mismatch: The learned deformation acts as a "measurable signature" of departures from classical optimality. It transforms the receiver from a black-box optimizer into a diagnostic tool that can identify where and how theoretical assumptions fail (e.g., identifying that Duobinary signaling inherently mismatches the optical channel's low-frequency response).
Scientific Insight: By expressing learned corrections relative to theoretical optima, the framework allows machine learning to interrogate analytical models. It reveals that receiver mismatch is not arbitrary but is structured, signaling-dependent, and physically interpretable.
Design Strategy: The work establishes deformable matched filtering as a practical design strategy that bridges the gap between theory and implementation without discarding the theoretical foundation. It suggests that future receiver design and signaling optimization (e.g., Modified Duobinary) can be guided by understanding the specific spectral and structural components of mismatch identified by the deformation.

The author explicitly states that this is not a claim of a universally superior receiver architecture compared to unconstrained neural networks or adaptive equalizers, but rather a novel coordinate system for measuring and understanding the limitations of classical matched-filter theory in real-world optical wireless environments.

Technical Summary: Explainable Deformable Matched Filtering in Optical Wireless Communications

More like this