Doing More With Less: Towards More Data-Efficient Syndrome-Based Neural Decoders

Technical Summary: Achieving More with Less: Towards Data-Efficient Syndrome-Based Neural Decoders

Problem Statement
While substantial research effort has been invested in developing advanced neural decoder architectures (such as Neural Belief Propagation and syndrome-based neural decoders), comparatively little attention has been paid to the quality and construction of training data. Current literature relies predominantly on "on-demand" data generation, where noisy received words are dynamically generated during training. This approach contrasts with standard Deep Learning practices that utilize fixed datasets, hinders fair benchmarking comparisons, and makes it difficult to estimate the data volume required for specific performance levels. Furthermore, on-demand training often targets "true" error patterns (the actual channel errors), leading to a discrepancy between the training objective and the ultimate goal of approximating optimal Maximum-Likelihood Decoding (MLD), which inevitably makes errors. The authors posit that insufficient attention to the construction of training data is a primary reason why neural decoders often fail to approach MLD performance on practical codes.

Methodology
The study focuses on syndrome-based neural decoders (SBND) and specifically employs two model-free architectures: a stacked Gated Recurrent Unit (GRU) network and a Transformer-based Error Correction Code Transformer (ECCT). The authors propose a shift from dynamic on-demand generation to a curated paradigm utilizing fixed datasets. Their methodology encompasses three core strategies:

1. Fixed Datasets: Instead of generating data on-the-fly, the authors construct static datasets offline. This enables precise control over the training distribution and allows for reproducible benchmarking.
2. Target Selection (MLD vs. True Errors): The authors argue that neural decoders should be trained to replicate the output of an MLD decoder rather than raw channel errors. They replace true error patterns ($e_{chan}$) with MLD error patterns ($e_{ML}$), derived using an Ordered Statistics Decoder (OSD) with a maximum post-processing order of $i_{max} = \lfloor d_{min}/4 \rfloor$.
3. Optimized Data Distributions: Recognizing that standard channel noise distributions (binomial) overrepresent low-weight error patterns that are easy to learn, the authors investigate four heuristics for constructing training sets to better cover the error region relevant to Frame Error Rate (FER):
   • Reference: Direct collection of MLD error patterns from OSD Monte Carlo simulations (adhering to the natural channel distribution).
   • Uniform Weight: Sampling to achieve a uniform distribution of Hamming weights up to a threshold $w_{max}$.
   • Biased Input Noise: Using Importance Sampling to generate input noise that favors error events near MLD decision boundaries.
   • Uniform Syndromes: Filtering samples to ensure all non-zero syndromes are equally represented in the dataset.

Key Results
Experiments were conducted on short block codes, primarily the BCH codes (31, 21, 5) and (63, 51, 5), comparing the proposed strategies with fixed datasets against traditional on-demand training.

• Data Efficiency: Training with fixed datasets containing MLD error patterns is significantly more data-efficient. For the (31, 21) code, models trained on 4 million fixed samples achieved performance comparable to models trained on 1 billion on-demand samples (a reduction in sample complexity by two orders of magnitude).
• Target Alignment: Training to correct MLD error patterns ($e_{ML}$) consistently results in lower Frame Error Rates (FER) than training on true error patterns ($e_{chan}$), particularly when using smaller datasets or at lower Signal-to-Noise Ratios (SNR).
• Distribution Impact: Datasets constructed with distributions deviating from the natural channel distribution (specifically uniform weight and uniform syndrome distributions) performed better than those following the natural channel distribution. For instance, a GRU model trained on 32 million samples with optimized distributions outperformed a model trained on 3 billion on-demand samples on the (63, 51) code.
• Model Efficiency: The ECCT architecture demonstrated a superior performance-to-sample-complexity ratio compared to the GRU architecture. Notably, a smaller ECCT model (304K parameters) trained on a curated dataset of 50 million samples outperformed a larger ECCT model (1.7M parameters) trained on 128 million on-demand samples.
• Performance Ceiling: The study observed that both architectures reached a performance plateau (stabilizing at an FER of approximately 0.04 for the tested SNR), regardless of further increases in data volume. This suggests that model capacity, rather than data volume, becomes the limiting factor for approaching MLD.

Significance and Claims
The work claims that the field of neural decoding has overlooked the critical role of training data quality. The authors emphasize that by introducing fixed datasets, targeting MLD error patterns, and optimizing training data distributions, existing neural decoder architectures can achieve superior performance with significantly fewer training examples.

The significance of this work lies in its demonstration that "more with less" can be achieved not by inventing new, more complex neural architectures, but by refining the data engineering process. The authors emphasize that these findings apply to model-free decoders (such as SBND) and likely extend to model-based decoders (such as Neural Belief Propagation). They conclude that careful curation of training data is crucial for reducing training costs, improving generalization, and enabling fair benchmarking comparisons of neural decoding techniques.