GeoSANE: Learning Geospatial Representations from Models, Not Data

1. Problem Statement

The remote sensing (RS) domain has seen a proliferation of Foundation Models (RSFMs) trained on diverse satellite data (optical, SAR, multispectral) for various tasks (classification, segmentation, detection). However, this landscape is fragmented:

• Specialization vs. Unification: Each RSFM specializes in a specific subset of sensors, resolutions, or objectives. No single model captures the full breadth of geospatial knowledge.
• Redundancy and Inefficiency: Users must repeatedly choose, retrain, or select between models for new tasks. The collective knowledge of all existing models is underutilized.
• Limitations of Current Approaches:
  • Pre-training from scratch is data and compute-intensive.
  • Model Merging (e.g., averaging weights) often fails when models have different architectures or divergent tasks, leading to performance degradation.
  • Knowledge Distillation/Pruning requires a large teacher model and often struggles to generate high-quality lightweight models without significant performance loss.

Core Question: Can we learn a unified representation of geospatial knowledge directly from the weights of existing heterogeneous models, rather than from raw satellite data, to generate new, task-specific models on demand?

2. Methodology: GeoSANE

GeoSANE (Geospatial SANE - Sequential Autoencoder for Neural Embeddings) is a weight-space learning framework that treats neural network parameters as a data modality. It operates in three stages:

A. Data Collection (Model Retrieval)

• Source: The authors curated a heterogeneous collection of 103 remote sensing models from Hugging Face, totaling approximately 38 billion parameters.
• Diversity: The collection includes various architectures (ViT, Swin, ResNet, UNet, YOLO), modalities (RGB, Multispectral, SAR, Hyperspectral), and tasks (flood detection, land cover, object detection).
• Preprocessing: A custom loader handles corrupted or undocumented checkpoints by inferring architecture from state dictionaries (e.g., identifying convolutional stems or patch embeddings).

B. Learning Shared Latent Representation (Training)

GeoSANE employs a Transformer-based Weight-Space Autoencoder:

1. Tokenization: Model weights are reshaped into 2D matrices per layer and divided into fixed-size tokens. A binary mask distinguishes real parameters from padding.
2. Positional Embedding: Tokens are augmented with 3D positional embeddings indicating sequence position, layer index, and within-layer position.
3. Architecture:
   • Encoder: Maps the sequence of weight tokens to a latent embedding sequence ($Z$).
   • Decoder: Reconstructs the original weight tokens from the latent space.
   • Projection Head: Maps latent embeddings to a lower-dimensional space for contrastive learning.
4. Loss Function: A combination of Reconstruction Loss (MSE on masked weights) and Contrastive Loss (NT-Xent). The contrastive loss uses augmented views (noised versions) of the same model to ensure the latent space captures robust structural features.
   • Innovation: Unlike prior work requiring homogeneous models, GeoSANE normalizes loss at runtime, enabling training on heterogeneous architectures and public repositories.

C. On-Demand Model Generation

Once trained, GeoSANE can generate weights for a target architecture:

1. Prompting: A user provides a "prompt model" (e.g., an ImageNet-pretrained ViT-L or Swin-B).
2. Encoding: The prompt's weights are tokenized and encoded into the latent space ($Z_{prompt}$).
3. Sampling: A Kernel Density Estimator (KDE) is fitted around $Z_{prompt}$. Samples are drawn from this local distribution to explore structurally similar but geospatially informed regions of the latent space.
4. Decoding: Sampled latent vectors are decoded into synthetic weight tokens and de-tokenized into neural network weights.
5. Output: The result is a functional model with the target architecture but weights initialized with aggregated geospatial knowledge, ready for fine-tuning.

3. Key Contributions

1. New Paradigm: Shifts the paradigm from data-centric pre-training to weight-centric generation. It learns from existing models rather than raw satellite imagery.
2. GeoSANE Framework: A scalable encoder-decoder approach capable of learning from heterogeneous models (different architectures, modalities, tasks) and generating weights for arbitrary target architectures on demand.
3. Comprehensive Validation: Extensive empirical evaluation across 10 diverse datasets, covering optical, multispectral, and radar modalities, and classification, segmentation, and detection tasks.

4. Experimental Results

GeoSANE was evaluated on 10 datasets (e.g., EuroSAT, RESISC-45, BigEarthNet, Sen1Floods11, DIOR) and GEO-Bench.

• vs. Training from Scratch: GeoSANE-generated models consistently outperform models trained from scratch.
  • Example: On fMoW, accuracy improved by +23.2%; on BigEarthNet, by +18.9%.
• vs. State-of-the-Art RSFMs: GeoSANE matches or surpasses leading foundation models (SatMAE, Scale-MAE, CROMA, DOFA, etc.) despite not being pre-trained on massive satellite datasets.
  • Example: Outperformed CROMA on Sen12Flood (+1.8%) and BigEarthNet (+0.1%).
• vs. Model Merging: GeoSANE significantly outperforms simple weight averaging or DARE (Drop And REscale) merging, proving that learning a latent manifold is superior to linear interpolation.
• vs. Compression (Pruning/Distillation): For lightweight models (e.g., ResNet-18, MobileNetV2), GeoSANE-generated weights outperform models obtained via magnitude pruning, variational dropout, or knowledge distillation.
  • Example: On fMoW (11M params), GeoSANE achieved 53.5% vs. 43.1% for the best distillation baseline.
• Generalization: The method generalizes across backbones (ViT, Swin, ResNet) and modalities. Models generated from a single latent space perform well on tasks ranging from flood segmentation to building extraction.

5. Significance and Impact

• Democratization of Geospatial AI: Allows users to generate specialized models without the massive compute costs of pre-training on petabytes of satellite data.
• Knowledge Unification: Successfully aggregates the fragmented knowledge of 70+ existing RSFMs into a single, coherent latent space.
• Efficiency: Enables the generation of lightweight, high-performance models directly, bypassing the need for complex distillation pipelines or teacher models.
• Scalability: As the number of open-source geospatial models grows, the GeoSANE latent space becomes richer, offering a scalable path to unify and transfer geospatial knowledge.

In conclusion, GeoSANE demonstrates that learning from models is a viable and powerful alternative to learning from data, offering a flexible, efficient, and high-performing framework for the future of remote sensing foundation models.