Signal Space-Transformed Expectation Propagation for Symbol Detection in ISI Channels

Technical Summary: Signal Space-Transformed Expectation Propagation for Symbol Detection in ISI Channels

Problem Statement
Iterative message passing detection based on Expectation Propagation (EP) has established itself as a near-optimum framework for signal processing in Multiple-Input Multiple-Output (MIMO) and Inter-Symbol Interference (ISI) channels. In this framework, a linear minimum mean square error (LMMSE) estimator iterates with a non-linear symbol-wise demapper. While effective for many scenarios, conventional EP-based detection suffers from significant performance degradation in channels with strong ISI. This failure occurs when the initial LMMSE estimate is inaccurate, leading to a poor approximation of the true posterior distribution, particularly when dealing with discrete priors. Furthermore, while the optimal Bahl–Cocke–Jelinek–Raviv (BCJR) detector offers superior performance, its computational complexity scales exponentially with the channel memory ($O(M^{L+1})$), rendering it infeasible for channels with long impulse responses.

Methodology
The authors propose an EP-based detector that operates within a transformed signal space rather than the original channel space. The core methodology involves three key components:

1. Linear Channel Shortening: A linear receive filter ($W$) transforms the original system model ($y = Hx + n$) into a transformed model ($y_F = Fx + \epsilon_F$). The target channel matrix $F$ is constrained to have reduced memory ($\nu < L$) compared to the original channel. This transformation is designed to enhance the initialization of both the linear and non-linear estimators.
2. Iterative Estimation in Transformed Space: Instead of the conventional loop between an LMMSE estimator and a symbol-wise demapper, the proposed scheme iterates between:
   • A Linear Estimator (LE): Based on a channel shortening filter, operating on the transformed signal space.
   • A Non-Linear Estimator (NLE): A trellis-based BCJR detector. Crucially, this detector operates on the reduced memory $\nu$ of the transformed system model, not the original channel memory $L$. This allows the BCJR complexity to scale as $O(M^{\nu+1})$, making it feasible even when the original channel memory $L$ is too large for a standard BCJR.
3. Deliberate Mismatched Initialization: To accelerate convergence, the authors propose a specific initialization strategy for the first iteration ($\ell=0$). While the EP framework typically assumes a fully-factorized (diagonal) prior covariance, the authors initialize the LE using the full covariance matrix $V_A = FF^T$ derived from the channel shortening filter. However, the message passing prior is still initialized with the diagonal elements of this matrix ($v^{(0)}_A = \text{diag}(FF^T)$). This "mismatch" effectively recovers the optimal channel shortening filter estimates in the first step without violating the diagonal assumption required for subsequent message passing iterations.

Key Contributions

• Transformed Signal Space EP: The introduction of an EP framework that operates in a signal space defined by a linear channel shortening filter, enabling the use of reduced-memory BCJR detectors within an iterative EP loop.
• Performance-Complexity Trade-off: The method introduces hyperparameters (transformed memory $\nu$ and iteration count $N_{It}$) that allow for a fine-granular trade-off between detection performance and computational complexity.
• Convergence Acceleration: The proposal of a deliberate mismatch between the initialized messages and the covariance used in the linear estimator for the first iteration, which significantly improves convergence speed and initial estimation accuracy.

Results
The proposed approach was evaluated on two channel models: the well-known Proakis-C ISI channel (representing a worst-case strong ISI scenario) and a dataset of 26,161 CIRs from an indoor non-line-of-sight wireless measurement campaign (DICHASUS).

• Proakis-C Channel (8-PAM): At a symbol-wise mutual information (SMI) of 2 bits per channel use, the proposed detector with reduced memory $\nu=2$ achieved a 5 dB gain over the conventional EP detector ($\nu=0$). Increasing the memory to $\nu=3$ provided an additional 1 dB gain, reducing the gap to the optimal full-memory BCJR detector to 2 dB.
• DICHASUS Indoor Channels (16-QAM): At a Symbol Error Rate (SER) of $2 \times 10^{-3}$, the proposed detector showed a gain of more than 2 dB compared to the conventional EP detector. A gain of 0.7 dB was observed when comparing $\nu=2$ against $\nu=1$.
• Complexity Analysis: The results demonstrate that for a fixed complexity number (weighted sum of operations), increasing the reduced memory $\nu$ yields better performance. The "mismatched initialization" strategy significantly improved convergence for the Proakis-C channel, though its impact varied for the DICHASUS channel.

Significance and Claims
The paper claims that the proposed method offers a favorable performance-complexity trade-off over conventional EP-based detection. By operating in a transformed signal space, the method enables the use of BCJR detectors with reduced memory, making near-optimal detection feasible for channels where the standard BCJR is computationally prohibitive. The authors emphasize that the approach is particularly effective for channels with strong ISI where initial linear estimates are typically inaccurate, demonstrating improvements of up to 6 dB at 2 bits per channel use in specific configurations. The work suggests that the transformed signal space approach effectively enhances the initialization of both linear and non-linear estimators, leading to robust detection performance without the exponential complexity penalty of full-memory optimal detection.