TerraCodec: Compressing Optical Earth Observation Data

The paper introduces TerraCodec, a family of publicly available, pretrained neural codecs for Earth observation data that utilizes a Temporal Transformer and a novel Latent Repacking method to achieve significantly higher compression rates than classical codecs while enabling zero-shot cloud inpainting.

The Gist

Imagine the Earth is a giant, living library, and satellites are the librarians taking millions of photos every day. These aren't just regular photos; they are multispectral time-lapses. They capture the planet in 12 different "colors" (including invisible ones like infrared) and take pictures of the same spot every few days to watch seasons change, crops grow, or floods rise.

The problem? This library is getting too heavy to carry. The data is so massive that storing it and sending it back to Earth is like trying to mail a library's worth of books in a single envelope.

Enter TerraCodec, a new team of "digital movers" designed to shrink these massive files without losing the important details. Here is how they do it, explained simply:

1. The Problem: Why Old Movers Fail

Traditional compression tools (like JPEG or MP4) are like old-school movers who only know how to pack a single room. They treat every photo as a separate, static object.

• The Issue: Earth observation isn't static. A forest in July looks different than in December, but the trees are still there. Old tools throw away the "memory" of the previous photo, forcing them to re-pack the same tree trunks and river banks over and over again. This wastes space.
• The Result: To get a clear picture, you have to send a huge file.

2. The Solution: TerraCodec's "Smart Packing"

TerraCodec is a family of AI tools trained specifically on Earth data (using images from the Sentinel-2 satellite). It uses three main tricks:

A. The "Contextual Memory" (TEC-TT)

Imagine you are watching a movie. If you see a frame where a character is walking, you don't need to be told exactly where their feet are in the next frame; you can guess they moved forward.

• How it works: TerraCodec's "Temporal Transformer" (TEC-TT) is like a super-smart movie editor. It looks at the last few photos and learns the patterns of the Earth. It knows that clouds move slowly, rivers don't jump, and crops grow gradually.
• The Magic: Instead of sending the whole new photo, it only sends the changes. "The cloud moved left," "The water level rose." This is why it can shrink files by 3 to 10 times compared to old methods while keeping the picture just as sharp.

B. The "Smart Suitcase" (Latent Repacking)

Usually, AI compression tools are like suitcases that only fit one specific size of clothes. If you want a smaller suitcase (lower quality/smaller file), you have to buy a whole new suitcase and repack everything from scratch.

• The Innovation: TerraCodec introduces Latent Repacking. Imagine a suitcase with a magical zipper.
  • The "Essentials" (Early Tokens): The first few items you pack are the most important: the shape of the land, the big rivers, the general layout.
  • The "Details" (Later Tokens): The later items are the fine details: the texture of the leaves, the ripples in the water.
• The Benefit: You can unzip the suitcase halfway and still have a perfect map of the terrain. You don't need a new suitcase; you just decide how many "zippers" (tokens) you want to open based on how much space you have. This allows one single model to handle any file size you need.

C. The "Weather Predictor" (Zero-Shot Cloud Inpainting)

Satellite photos are often ruined by clouds. Usually, you need a special AI trained just to remove clouds.

• The Trick: Because TerraCodec is so good at understanding how the Earth should look over time, it can act like a weather forecaster. If it sees a photo with a cloud, it looks at the previous photos, guesses what the ground looks like underneath, and "paints" the cloud away.
• The Result: It can fix cloudy photos without ever being taught how to do it specifically. It just uses its general knowledge of the Earth. It beats all other current methods at this task.

3. Why Does This Matter?

• For Scientists: They can store years of data on a single hard drive instead of a warehouse.
• For Disaster Response: When a flood hits, they can send high-quality maps back to Earth instantly, even with slow internet connections.
• For the Planet: Less data transmission means less energy used by satellites and data centers.

The Bottom Line

TerraCodec is like upgrading from a clumsy, manual mover to a robotic, predictive packing system. It doesn't just squish the data; it understands the story of the Earth. By remembering the past and predicting the future, it throws away the unnecessary repetition, leaving us with a tiny file that still tells the whole story.

The best part? The creators have opened the doors and shared the blueprints (code and models) with the world, so anyone can use this technology to help monitor our planet.

Technical Summary

1. Problem Statement

Earth Observation (EO) satellites, such as those in the Copernicus program, generate massive streams of multispectral, multi-temporal image data. This creates critical bottlenecks in storage, transmission, and processing.

• Unique Challenges: Unlike natural images, EO data is multispectral (up to 12+ bands), high-bit-depth (16-bit), and multi-temporal (captured at regular intervals from constant viewpoints).
• Current Limitations:
  • Fragmented Research: Learned compression for EO is fragmented, lacking large-scale, publicly available pretrained codecs.
  • Underexplored Temporal Redundancy: Prior work focuses on single-image compression, ignoring the strong temporal dependencies in satellite time series (seasonal patterns vs. object motion).
  • Fixed-Rate Constraints: Most neural codecs are trained for a single rate-distortion setting, requiring retraining for different bitrates.
  • Standard Inefficiency: Classical codecs (JPEG, JPEG2000, HEVC) are not optimized for multispectral data or temporal sequences, often treating bands independently.

2. Methodology

The authors introduce TerraCodec (TEC), a family of learned neural codecs pretrained on the SSL4EO-S12 dataset (Sentinel-2 L2A data). The approach includes three main components:

A. Architecture Variants

1. TEC-FP (Factorized Prior): A lightweight image codec using a fully factorized entropy model. It assumes independent latent variables, offering fast parallel encoding but limited expressiveness.
2. TEC-ELIC (Efficient Learned Image Compression): A stronger image codec based on the ELIC architecture. It utilizes a hyperprior with spatial and channel context (checkerboard convolutions and grouped channel context) to predict per-latent mean and scale, significantly improving rate-distortion performance.
3. TEC-TT (Temporal Transformer): A video codec extending the VCT (Video Compression Transformer) architecture.
   • Mechanism: It encodes frames using an ELIC-based encoder, tokenizes the latent space, and uses a masked autoregressive transformer to model temporal dependencies.
   • Context: It predicts the current frame's latent distribution based on the two previous frames ($t-1, t-2$) without relying on hand-crafted motion estimation.
   • Training: Trained end-to-end on rate-distortion loss with a $\lambda$-schedule emphasizing low-rate regimes.

B. Latent Repacking (Flexible-Rate Training)

To overcome the fixed-rate limitation of neural codecs, the authors propose Latent Repacking for the FlexTEC model:

• Concept: Instead of standard spatial tokens, the latent channels are sliced and reordered. Early tokens encode global scene structure, while later tokens refine local details.
• Training Strategy: During training, a random token budget $K$ is sampled. The remaining $T-K$ tokens are masked with a learned "mask token." The rate loss is computed only on unmasked tokens, with dynamic scaling to prevent information collapse into the first few tokens.
• Inference: Users can select a compression level by transmitting only the first $K$ tokens. The transformer predicts the missing tokens based on the learned temporal prior, allowing flexible bitrate control from a single checkpoint.

C. EO-Specific Design Choices

• Native 12-Band Support: Handles 16-bit reflectance data directly.
• Per-Band Standardization: Standardizes each spectral band individually to preserve band-specific statistics crucial for downstream analysis.
• Pretraining: Trained on a large-scale global dataset (244k locations) to learn robust domain-specific priors.

3. Key Contributions

1. TerraCodec Suite: The first publicly available, large-scale pretrained neural codecs for Sentinel-2 multispectral and multi-temporal data.
2. Superior Compression: Demonstrates that learned codecs significantly outperform classical standards (JPEG2000, HEVC) in rate-distortion performance.
3. Latent Repacking: A novel method enabling flexible-rate transformer models (FlexTEC) that operate across varying bitrates without retraining.
4. Downstream Utility: Demonstrates that compression models possess implicit knowledge useful for tasks beyond compression, specifically zero-shot cloud inpainting.

4. Results

Rate-Distortion Performance

• Efficiency: TerraCodec achieves 3–10× higher compression than classical codecs at equivalent image quality.
• Metrics: On the SSL4EO-S12 validation set, TEC-TT achieves up to 22.6% rate reduction compared to image-only codecs (TEC-ELIC) when utilizing two frames of temporal context.
• Comparison: At an SSIM of 0.999, the lightweight TEC-FP achieves 5× lower bitrate than WebP. TEC-TT outperforms HEVC and JPEG2000 across both PSNR and SSIM metrics.

Flexible-Rate Performance (FlexTEC)

• FlexTEC performs close to or better than fixed-rate models across a wide range of token budgets ($K=1$ to $16$).
• It successfully maintains global structure even at very low token counts, with later tokens refining details.

Downstream Tasks

• Cloud Inpainting (AllClear Benchmark): TEC-TT was applied to remove clouds without task-specific fine-tuning.
  • Performance: It outperformed all heuristic baselines (e.g., LeastCloudy) and prior zero-shot neural methods.
  • Metrics: Achieved 32.86 dB PSNR and 0.917 SSIM on the full test set, significantly surpassing the previous best (UnCRtainTS at 29.01 dB).
  • Robustness: Maintained high performance even on the hardest 10% of samples (99% cloud coverage).
• Downstream Analysis: Fine-tuning segmentation and classification models on TerraCodec-compressed data resulted in minimal performance degradation (<1.0 percentage point drop) at moderate compression rates, while reducing data size by up to 380×.

5. Significance

• Paradigm Shift: TerraCodec establishes neural codecs as a viable and superior alternative to classical standards for Earth Observation, moving beyond natural image compression.
• Temporal Modeling: It proves that modeling long-range temporal dependencies in satellite data (seasonal patterns) yields significant compression gains, a previously underexplored area.
• Operational Impact: The ability to compress data by an order of magnitude without losing analytical utility addresses critical storage and transmission bottlenecks in the Copernicus program and similar initiatives.
• Open Science: The release of code, models, and pretrained weights under a permissive license facilitates future research and adoption in the geospatial community.

In conclusion, TerraCodec bridges the gap between advanced neural compression techniques and the specific requirements of Earth Observation, offering a scalable, efficient, and versatile solution for the future of satellite data management.