Requirement-Driven Lossy Compression of Weather and Climate Fields

Technical Summary: Requirement-Driven Lossy Compression of Weather and Climate Fields

Problem Statement
Weather and climate data archives are expanding faster than storage capabilities, necessitating lossy compression. However, operational adoption is hindered by a lack of trust; a reconstruction is only usable if its error is bounded by explicit, machine-checkable accuracy requirements. The engineering challenge is not merely to minimize distortion at a fixed rate, but to maximize the compression ratio subject to constraints defined by external domain scientists and verified by reference checkers the compressor does not control. This work addresses five distinct compression problems involving operational atmospheric data, ranging from satellite fields with missing values to large ensembles of ERA5 variables with diverse, per-variable accuracy requirements.

Methodology
The author proposes a unified architecture designed to solve structurally different compression problems through a requirement-driven approach rather than data-driven heuristics.

1. Codec Selection via Requirement Analysis: Instead of selecting a codec based on data statistics, the system analyzes the structure of the accuracy requirement (e.g., absolute vs. relative bounds, pointwise vs. mean bounds, bounds on derivatives). Based on this structure, it selects from four codec families:

   • Zero-anchored uniform grid: For absolute bounds, snapping values to a grid anchored at zero to preserve exact zeros (crucial for precipitation and mean bounds).
   • Log-ratio grid: For pointwise relative bounds, transforming the data via $\log_2|x|$ and applying a uniform grid.
   • Transform codec (SPERR/SZ3) with optional repair: For smooth fields, these are run at coarser settings than their nominal guarantee. If a pointwise bound is violated, a correction layer repairs specific points.
   • Correction-layer-only: A constant (zero) prediction wrapped in safeguards, serving as a baseline for constant-dominated fields.

2. Entropy Stage: All families terminate in a shared entropy stage consisting of tokenization of distinct values, a bitmap index to lift frequent tokens from the literal stream, and raw LZMA2 compression. Missing values (NaN) are handled either via a packed bitmap or by rounding in floating point so NaN survives transforms as a distinct token.

3. Checker-in-the-Loop Parameter Search: The system employs a one-parameter search where a "looseness" parameter $p$ scales the nominal setting of the selected codec. The search starts at $p=1$, doubles until the reference checker rejects the reconstruction, and then bisects to find the optimal setting. For families using correction layers, the search takes the maximum ratio over evaluated settings rather than the loosest admissible one, as the ratio curve is non-monotonic.

4. Fidelity Criterion: The author introduces a criterion to reject "admissible-but-degenerate" reconstructions. If a mean-error bound exceeds the field's own variability, a codec might reconstruct a constant field (erasing the data) while technically satisfying the bound. The system enforces a constraint that the reconstruction must retain a minimum number of distinct values and maintain an error relative to the field's standard deviation ($\sigma$).

Key Results
The system was evaluated on five problems (P1–P5) with zero violations against reference checkers:

• P1 (Satellite Water-Vapour Path): Achieved a compression ratio of ×47.44 on a field with 69% missing values (NaN), outperforming baselines like SZ3 (×30.3) and SPERR (×20.1).
• P2 (Precipitation): Achieved ×19.07 on a field with exact zeros and a skewed distribution. Here, a uniform log-ratio grid outperformed transform codecs, which failed to maintain bounds on the noisy data.
• P3 (Specific Humidity Derivative): Achieved ×83.88 by constraining a longitudinal derivative. The transform codec (SPERR) with a correction layer was highly effective because errors in smooth fields are spatially correlated, requiring few repairs.
• P4 & P5 (ERA5 Variables):
  • P4 (16 Pressure-Level Variables): Median ratio of ×55.68 (vs. baseline ×46.14). Transform codecs dominated (9/16 variables).
  • P5 (226 Single-Level Variables): Median ratio of ×50.76 (vs. baseline ×22.88). Uniform grids dominated (170/226 variables), particularly for zero-inflated surface fields.
  • Fidelity Impact: Without the fidelity criterion, 28 of the 226 single-level variables would have been reconstructed as constant fields with ratios up to ×35,194. Applying the criterion reduced the maximum ratio to ×6,530 but ensured the data was not erased.

Significance and Claims
The paper claims three primary findings that generalize beyond the specific datasets:

1. Requirement-Codec Interaction: The optimal choice between transform coding and uniform quantization is governed by the interaction between field regularity and the bound type. Smooth fields under pointwise bounds favor transform coding, while noisy or zero-inflated fields under mean bounds favor zero-anchored grids.
2. Cost of Correction: A correction layer guarantees admissibility, but its efficiency depends on the fraction of points needing repair. It is profitable for derivative constraints on smooth fields (P3) but inefficient for pointwise constraints on noisy fields (P2).
3. Admissibility is Insufficient: The paper demonstrates that satisfying a published accuracy requirement is not a sufficient acceptance criterion if the bound allows for the erasure of the field's variability. The author argues that requirement-driven compression results must be reported alongside a fidelity statement (e.g., distinct value count and relative error) and that codecs should be published as rebuildable configurations to ensure independent verification.

Additional Observations
The evaluation process revealed two significant issues in the problem definitions themselves:

• Derivative Constraint Error: The evaluation code for P3 divided the derivative by $357.5^\circ$ instead of the intended $2.5^\circ$, making the bound roughly 143× looser than documented. The author reported their results against the implemented (looser) constraint while explicitly noting the discrepancy.
• Mean-Error Bounds: For 28 variables, the published mean-error bound exceeded the field's own mean magnitude, technically allowing a constant-field reconstruction. This highlights a flaw in how some requirements are formulated for benchmarking.

The work concludes that a single architecture, driven by requirement analysis and verified by external checkers with fidelity guards, can effectively solve diverse atmospheric compression problems while maintaining scientific integrity.