Research on compression technology of meteorological big data based on satellite communication

Technical Summary: Compression Technology for Meteorological Big Data Based on Satellite Communication

Problem Statement
The paper addresses the critical bottleneck in transmitting high-resolution numerical weather prediction (NWP) data to ocean-going vessels via satellite. While global meteorological data (e.g., from ECMWF) offers high spatial and temporal resolution, its volume is massive (e.g., ~15GB per start time for ECMWF). Existing satellite communication systems used in maritime environments, specifically China's Beidou-3 and Tiantong-1, face severe limitations. Beidou-3 short messages are restricted to approximately 14,000 bits (regional) or 560 bits (global), while Tiantong-1, though offering higher bandwidth (384 Kbps), suffers from high power consumption and latency. Consequently, current maritime communication methods can only support simple data exchanges, failing to deliver the massive datasets required for accurate numerical forecasting at sea. The core challenge is to compress these datasets significantly without losing the essential meteorological characteristics required for weather analysis.

Methodology
The authors propose a hybrid compression algorithm combining lossy and lossless compression techniques, specifically tailored for point cloud data representing meteorological fields (using Mean Sea Level Pressure, PRMSL, as the primary case study). The process involves four main stages:

1. Scale Separation and Data Processing:

   • The original high-resolution grid data (0.125°) is decomposed into a multi-layer grid nesting scheme (G100, G200, G300) corresponding to different weather system scales (synoptic and mesoscale).
   • This involves extracting core eigenvalues (MAX/MIN) from the embedded grid matrices to represent the weather system's intensity and spatial distribution.

2. Primary Lossy Compression (Dimensionality Reduction Plane Octree Coding):

   • To eliminate redundancy while preserving core characteristics, the authors design a "dimensionality reduction plane octree coding" scheme.
   • The data is structured into a reduced-dimensional plane octree where each node represents a volume element.
   • Morton coding is applied to the child nodes of the octree structure. If a child node contains data, it is coded as '1'; otherwise, '0'.
   • Nodes with a code of '00000000' (indicating no data in any of the eight directions) are removed, effectively de-duplicating the dataset.
   • Data Fusion: To address "point cloud voids" created by this removal, a data fusion method is applied. If a grid contains fewer than two data points after compression, it is filled with data from the coarser scale (DDLarge) to prevent errors in isobar analysis or weather system boundary detection.

3. Secondary Lossless Compression (Triple Matrix Pointer Symmetric Cryptographic Table):

   • To further reduce size without data loss, a lossless compression algorithm based on a "triple matrix pointer symmetric cryptographic table" is applied.
   • Pressure Transformation: PRMSL values (typically 600–1400 hPa) are shifted by -500, reducing the integer requirement from 4 bits to 3 bits.
   • Matrix Slicing: The global grid is sliced into 32 regions (45°×45°).
   • Pointer Encoding: The algorithm utilizes three nested matrix levels (G300, G200, G100). Instead of transmitting raw values, the system transmits pointers (row/column indices) referencing the location of non-empty values within these matrices.
   • The data stream is organized as: Region Code (QB) $\rightarrow$ G300 pointers $\rightarrow$ G200 pointers $\rightarrow$ G100 pointers $\rightarrow$ Actual PRMSL value. This structure is encrypted using a symmetric cryptographic table for secure transmission.

4. Transmission:

   • The compressed data is transmitted via Tiantong-1 satellite and is theoretically convertible into Beidou-3 short messages.

Key Contributions

• Hybrid Algorithm Design: The paper introduces a novel two-stage compression framework that integrates scale-aware lossy compression (octree coding) with pointer-based lossless compression.
• Meteorological Feature Preservation: Unlike generic compression, the lossy stage is explicitly designed to preserve the "core meteorological characteristics" of different scale weather systems, ensuring that the reconstructed data remains scientifically valid for analysis.
• Satellite Compatibility: The algorithm is specifically optimized to fit within the strict constraints of maritime satellite communication (Beidou-3 short messages and Tiantong-1 bandwidth), enabling the transmission of NWP data that was previously impossible.
• Field Validation: The study includes real-world testing on the Yuezhanyuke No.2 research vessel, validating the algorithm under actual oceanic conditions.

Results
The study utilized ECMWF numerical prediction data from July to December 2021 for testing:

• Compression Efficiency:
  • Original Data: ~13.9 MB (for a single element, PRMSL, in a specific region).
  • After Lossy Compression: Reduced to ~29–31 KB (Average lossy compression rate: 0.21%).
  • After Lossless Compression: Reduced to ~1.6 KB (Average lossless compression rate: 5.28%).
  • Total Compression: The total compression ratio achieved is approximately 0.112‰ (a reduction factor of nearly 10,000 times).
• Transmission Performance (Tiantong-1 Satellite Test):
  • Uncompressed Data: Average transmission time was 578.2 seconds with frequent interruptions and failures.
  • Lossy Compressed: Average time reduced to 12.4 seconds.
  • Fully Compressed: Average transmission time dropped to 5.2 seconds with stable, failure-free transmission.
• Beidou-3 Conversion: The compressed regional dataset (QB(2,7)) fits into 1 Beidou-3 short message. A global single compressed dataset requires 32 messages, and a 10-day global dataset requires 1,664 messages.

Significance and Claims
The paper claims that this technology solves the transmission bottleneck for marine meteorological big data. By reducing a 13.9 MB file to less than 1.7 KB, the algorithm enables the delivery of high-resolution numerical forecast products to ships in remote ocean areas where bandwidth is limited and unstable.

• Timeliness: The transmission of a regional dataset takes only 5.2 seconds via Tiantong-1, or can be sent via Beidou-3 short messages within minutes, meeting the real-time needs of ocean navigation.
• Reliability: Field tests demonstrated that the compressed data transmission is stable and reliable, unlike uncompressed data which frequently failed.
• Practical Application: The compressed data can be decrypted on the ship using the symmetric cryptographic table to restore the data for analysis, graphic drawing, and system integration.
• Intellectual Property: The authors note that the core algorithm and transmission optimization scheme are covered by two granted invention patents (CN116683915B in China and NL2037367 in the Netherlands), confirming the novelty and technical maturity of the proposed method.