Lossless compression of simulated radio interferometric visibilities

Technical Summary: Lossless Compression of Simulated Radio Interferometric Visibilities

Problem Statement
Modern radio astronomical observatories generate vast volumes of visibility data, typically stored as complex 32-bit single-precision floating-point values. A significant storage burden arises during data processing, specifically in direction-dependent calibration and deconvolution, where forward-predicted model data must be stored alongside observed visibilities. In direction-dependent self-calibration, the volume of these model columns can reach ten times the size of the original observed data. While lossy compression techniques (e.g., Dysco) are effective for noisy observed data by exploiting Gaussian distributions, they are unsuitable for model data. Applying quantization-based lossy compression to noiseless model data introduces artificial noise, which corrupts the calibration process where observed data is fitted against these models. Existing generic lossless algorithms (e.g., GZIP) fail to exploit the specific spectral and temporal smoothness of radio astronomical model data, resulting in suboptimal compression ratios.

Methodology: Simulated Signal Compression (Sisco)
The authors present Sisco, a novel lossless compression framework designed specifically for noiseless, forward-predicted interferometric model data. The method integrates seamlessly with existing Casacore-based workflows and operates through four distinct stages:

1. Prediction: The algorithm performs two-dimensional polynomial extrapolation on the real and imaginary components of visibility values across time and frequency dimensions. It tests various orders of extrapolation (0th to 3rd order) and averaging schemes. The prediction is computed using integer arithmetic on the mantissa and exponent of the floating-point values to ensure precision and reversibility.
2. Residual Formation: The predicted value is subtracted from the actual data value. To ensure exact reversibility, the algorithm aligns the exponent of the predicted value with the data value before subtracting their mantissas. Special handling is applied for NaN, infinity, and overflow cases.
3. Data Grouping: The resulting residuals are decomposed into sign, mantissa, and exponent components. These are grouped into five separate streams: one for exponents, one for the sign and the first mantissa byte, and three for the remaining mantissa bytes. This reordering ensures that similar values are adjacent in the data stream, significantly improving entropy reduction.
4. Deflate Compression: The grouped streams are compressed independently using the libdeflate library, which combines Lempel-Ziv 1 (LZ77) dictionary encoding and Huffman encoding.

The implementation is designed to handle the standard ordering of Measurement Sets (time-slow, baseline-fast) by maintaining a small buffer of previous timesteps, allowing for on-the-fly compression with minimal memory overhead. It supports combination with Baseline-Dependent Averaging (BDA).

Key Results
The authors evaluated Sisco on diverse datasets from LOFAR, MeerKAT, and MWA, as well as simulated sine waves and pure Gaussian noise.

• Compression Efficiency: For noiseless forward-predicted data, Sisco achieves an average compression ratio reducing data volume to 24% of the original size.
  • Optimal Configuration: A combination of linear prediction in the time direction and quadratic prediction in the frequency direction yielded the best overall results.
  • Dataset Variance: Compression efficiency varied by dataset smoothness. Smooth data (e.g., MeerKAT forward-predicted models) compressed to 13%, while less predictable data compressed to 38%.
  • Component Analysis: The 8-bit exponent data proved highly compressible (reducing to ~3–15% of original size), while the 24-bit mantissa and sign data were the primary target of the prediction algorithms.
• Noisy Data: When applied to observed (noisy) data or pure Gaussian noise, prediction algorithms worsened compression ratios (e.g., pure noise compressed to 84% vs. 92% with prediction), confirming that Sisco is optimized for smooth, noiseless signals.
• Performance: On a commodity AMD Ryzen 7 2700X processor, Sisco achieved a throughput of 534 MB/s for generated data and 340 MB/s for real disk I/O at deflate level 9. The process is I/O bound on modern SSDs but CPU-bound on slower storage.
• BDA Integration: Combining Sisco with BDA resulted in a total size reduction of over a factor of 13 compared to raw data, with Sisco providing an additional 2.7% improvement over full-resolution BDA data.

Significance and Claims
The paper claims that Sisco provides a lossless solution for the specific problem of storing forward-predicted model data, addressing a gap left by existing lossy techniques that are incompatible with calibration requirements. By achieving an average compression factor of greater than 4 (reducing data to ~24%), Sisco significantly mitigates the storage and transport costs associated with multi-stage calibration and direction-dependent self-calibration, which are critical for current and future facilities like the Square Kilometre Array (SKA).

The authors emphasize that the method is transparent to users; integrated as an open-source storage manager within Casacore, it allows software packages like DP3 and WSClean to write compressed data without manual configuration. The decompression is automatic during data access. The authors modestly note that while the current implementation is CPU-based, the computational cost is minor compared to calibration and imaging tasks, and future GPU implementations could further improve speed. They also discuss the potential for extending Sisco to a lossy algorithm but note that such an extension would require a fundamentally different approach to error bounds compared to existing methods like Dysco.