GeoNDC: A Queryable Neural Data Cube for Planetary-Scale Earth Observation

The paper introduces GeoNDC, a queryable neural data cube that encodes planetary-scale Earth observation archives into a compact, continuous spatiotemporal implicit neural field, enabling high-fidelity on-demand queries, cloud-free reconstruction, and significant compression (up to 95:1) directly on consumer hardware.

The Gist

Imagine you have a massive library containing every photograph ever taken of the Earth from space, going back 20 years. This library is so huge that it would fill millions of hard drives. Currently, to find a picture of a specific forest on a specific day, you have to hunt through thousands of separate files, wait for them to load, and often deal with missing pages because clouds blocked the camera that day.

GeoNDC is a revolutionary new way to organize this library. Instead of storing millions of individual photos, it stores one single, tiny "magic recipe" that can recreate any photo of Earth, at any time, from any angle, instantly.

Here is how it works, using some everyday analogies:

1. From a Photo Album to a "Living Recipe"

The Old Way (The Photo Album):
Think of current Earth data like a giant photo album. If you want to see how a city grew over 20 years, you have to flip through 7,000 separate photo pages (one for every 8 days). If a page is torn out (clouds), you have to guess what was underneath. Storing this album takes up a massive amount of space (hundreds of gigabytes).

The GeoNDC Way (The Magic Recipe):
GeoNDC doesn't save the photos. Instead, it learns the "rules" of how the Earth looks and changes. It's like a chef who doesn't save a photo of every cake they've ever baked. Instead, they memorize the recipe.

• If you ask, "What did the Amazon rainforest look like on July 4th, 2015?" the recipe instantly "bakes" that specific image for you.
• If you ask, "What did it look like on July 5th?" it bakes that one too.
• Because it's a recipe (a mathematical model), it takes up almost no space. The paper shows they compressed 20 years of global data (168 GB) into a file smaller than a single high-definition movie (0.44 GB). That's a 380-to-1 shrinkage.

2. The "Smart Inpainting" Artist

One of the biggest problems with satellite photos is clouds. They act like a smudge on a camera lens, hiding the ground below.

• The Old Way: Scientists usually try to fix this by taking photos from different days and stitching them together, or by guessing based on the pixels right next to the cloud. This often looks blurry or fake.
• The GeoNDC Way: Because GeoNDC has learned the "story" of the Earth's seasons, it acts like a master artist who knows how plants grow. If a cloud covers a field in June, GeoNDC doesn't just guess; it looks at how that field looked in May and July, understands the natural growth pattern, and paints in the missing part so perfectly that it looks like the cloud was never there. It fills in the gaps with high accuracy, even for huge cloud banks.

3. The "Instant Search" Engine

Imagine you want to study how a specific tree in your backyard changed color over 20 years.

• The Old Way: You have to open 7,000 different files, find the one pixel for your tree in each file, and copy the data. This is slow and requires a supercomputer to handle the memory.
• The GeoNDC Way: You just ask the "recipe." The computer instantly calculates the color of that tree for every single day in 20 years in a fraction of a second. You can do this on a regular laptop or even a web browser. It turns the data from a "static library" into an "interactive app."

4. Why This Matters

This isn't just about saving space; it's about speed and accessibility.

• For Scientists: They can analyze global climate change without needing a massive server farm. They can ask complex questions like, "Show me the growth rate of vegetation in Africa during drought years," and get an answer instantly.
• For Everyone: It means we can have a "Google Earth" that isn't just a map, but a living, breathing history of our planet that you can explore, query, and analyze right from your phone.

The Catch (The Fine Print)

The paper is honest about the limitations:

• Cooking vs. Eating: Creating the "recipe" (training the model) takes a lot of computing power and time. But once it's made, using it is incredibly fast and cheap.
• It's an Approximation: It's not a perfect 1:1 copy of the original raw data (it's "lossy"). However, for 99% of scientific uses, the quality is so high that you can't tell the difference. It's like a high-quality MP3 vs. a CD; for most people, the music sounds the same, but the file is much smaller.

In short: GeoNDC turns the Earth's history from a dusty, heavy archive of millions of files into a lightweight, interactive, and intelligent "time machine" that fits in your pocket.

Technical Summary

1. Problem Statement

Earth Observation (EO) systems have accumulated massive spatiotemporal archives (e.g., MODIS, Landsat, Sentinel) essential for monitoring planetary change. However, these archives face a critical bottleneck:

• Discrete Storage: Data is stored as discrete raster files (GeoTIFF, NetCDF), leading to massive storage requirements and inefficient transmission.
• Access Friction: Querying specific spatiotemporal points requires opening, streaming, and decoding thousands of individual files, making interactive analysis slow and I/O-bound.
• Data Gaps: Optical observations are frequently interrupted by clouds, shadows, and sensor artifacts. Traditional workflows rely on separate preprocessing steps (e.g., Maximum Value Compositing, interpolation) to fill gaps, which often introduce spectral biases or temporal discontinuities.
• Lack of Unification: Existing solutions treat storage, compression, and reconstruction as separate stages. There is no unified representation that is simultaneously compact, geospatially queryable, and continuous in time.

2. Methodology: The GeoNDC Framework

GeoNDC (Geographic Neural Data Cube) reformulates EO archives from discrete tensors into a continuous spatiotemporal implicit neural field. Instead of storing pixel values, the archive is represented as a learned function $\Phi_\theta(x, y, t)$ that maps spatial coordinates and time to physical values.

A. Mathematical Formulation

The goal is to learn a function $\Phi_\theta$ such that for any query $(x, y, t)$, the output $\hat{v}$ approximates the surface state. The model is trained using a masked objective function that minimizes the error only on valid observations ($M=1$), ignoring contaminated pixels ($M=0$). This allows the model to learn the underlying continuous manifold without being forced to fit noise or missing data.

B. Architecture: Decoupled Dual-Branch Design

To handle the anisotropy of Earth data (sharp spatial boundaries vs. smooth temporal evolution), GeoNDC employs a factorized architecture:

1. Static High-Resolution 2D Branch: Uses a multi-resolution 2D HashGrid to encode spatial coordinates $(x, y)$. This preserves high-frequency spatial details (e.g., coastlines, urban edges) across the entire time series without temporal degradation.
2. Dynamic Coarse 3D Temporal Branch: Uses a 3D HashGrid with spatial downscaling (scaling factor $s < 1$) to encode $(sx, sy, t)$. This forces the network to learn smooth, regional temporal trends (phenology) while suppressing temporal striping artifacts and reducing memory costs.
3. MLP Decoder: Concatenates the spatial and temporal embeddings into a high-dimensional vector, which is decoded by a lightweight Multi-Layer Perceptron (MLP) to predict the surface state.

C. Handling Missing Data & Residuals

• Implicit Gap Filling: Because the model learns a continuous spatiotemporal field, it can infer values for cloud-covered regions by leveraging the learned temporal context and neighboring valid observations. No external gap-filling algorithm is required.
• Sparse Residual Layer: To preserve critical local details that the neural approximation might miss, a sparse residual layer is added. Residuals exceeding a threshold $\tau$ are quantized and stored using entropy coding, ensuring high-fidelity reconstruction while maintaining compression.

D. Storage Protocol (.gndc)

GeoNDC introduces a unified binary storage format containing:

1. Global Geospatial Header: Metadata (CRS, bounding box, normalization).
2. Neural Payload: Quantized parameters of the HashGrids and MLP.
3. Physical Correction Layer (Optional): Validity bitmasks and sparse residuals.

3. Key Contributions

1. Unified Representation: GeoNDC is the first framework to unify compact storage, direct spatiotemporal querying, and continuity-aware reconstruction into a single executable neural model.
2. Massive Compression: It achieves orders-of-magnitude compression (e.g., 95:1 to 380:1) compared to optimized raster baselines (Int16/Float64) while preserving dominant spatiotemporal dynamics.
3. Native Continuity: It transforms "data as files" into "data as models," enabling arbitrary-time reconstruction and derivative calculations (e.g., $\partial NDVI/\partial t$) via automatic differentiation.
4. AI-Native Accessibility: The model can be deployed on consumer hardware (GPUs) and even executed entirely in web browsers via WebGPU, enabling interactive exploration of planetary-scale archives without backend servers.

4. Experimental Results

The authors evaluated GeoNDC on three distinct datasets:

• Sentinel-2 (High-Resolution, 10m):

  • Reconstruction: Achieved $R^2 > 0.96$ in valid regions.
  • Gap Filling: Under simulated cloud occlusions (up to 2km), GeoNDC maintained $R^2 > 0.85$, significantly outperforming linear interpolation ($R^2 \approx 0.63$) which suffered from spectral bias.
  • Compression: Compressed a 4.2 GB spatiotemporal volume to 292 MB.

• Global MODIS MCD43A4 (Planetary Scale, 20 years, 5km):

  • Compression: Reduced a 168 GB (Float64) archive to 0.44 GB (approx. 380:1 compression). Even against a conservative Int16 baseline (42 GB), it achieved a 95:1 reduction.
  • Fidelity: Mean $R^2 > 0.98$ across all spectral bands and years.
  • Query Efficiency: Retrieving a 20-year time series for a single pixel took 8 ms (GeoNDC) vs. 612 ms (GeoTIFF), an 81x speedup.
  • Memory: The entire 20-year archive fits in 2.2 GB VRAM, whereas loading just one year of GeoTIFFs requires ~77 GB RAM.

• HiGLASS Biophysical Products (LAI & FPAR):

  • Synergistic Encoding: Jointly encoded LAI and FPAR into a single model, achieving near-perfect accuracy ($R^2 > 0.98$) and sub-quantization error levels.
  • Efficiency: Reduced 7.2 GB of separate archives to a single 385 MB model.

5. Significance and Impact

• Paradigm Shift: GeoNDC proposes a fundamental shift from data as static files to data as executable models. This allows planetary archives to be treated as queryable, continuous functions rather than discrete raster stacks.
• Democratization of Analysis: By compressing multi-decadal global archives into files small enough for standard laptops and web browsers, GeoNDC removes the barrier of institutional high-performance computing infrastructure for global Earth system science.
• Scientific Transparency: The framework distinguishes between observed data and neural reconstruction via validity masks and optional residuals, ensuring scientific rigor while enabling "analysis-ready" data layers.
• Future Infrastructure: It lays the groundwork for AI-native geospatial infrastructures where storage, access, and reconstruction are intrinsic properties of the data representation itself, facilitating real-time, interactive planetary monitoring.