Any Model, Any Place, Any Time: Get Remote Sensing Foundation Model Embeddings On Demand

To address the challenges of heterogeneity in remote sensing foundation models, this paper introduces rs-embed, a Python library that enables users to retrieve embeddings from any supported model for any location and time range through a unified, single-line interface.

The Gist

Imagine you are a chef trying to cook a giant, global stew. You have a recipe that calls for "vegetables," but you have a problem: every farmer in the world sends you their vegetables in a different box, wrapped in different paper, and cut into different shapes. One sends you whole carrots, another sends you carrot juice, and a third sends you carrots that are already cooked but in a box that doesn't fit your pot.

To make your stew, you'd have to spend all your time unwrapping, chopping, and reshaping these vegetables just to get them into the pot. By the time you're ready to cook, you're exhausted, and you can't even taste-test the different recipes to see which one is best.

This is exactly the problem the paper "Any Model, Any Place, Any Time" is solving for the world of Remote Sensing.

Here is the breakdown in simple terms:

1. The Problem: The "Vegetable Chaos"

In the world of Earth observation, scientists use powerful AI models (called Foundation Models) to look at satellite images and understand what's happening on the ground (like predicting crop yields or tracking deforestation).

Currently, using these models is a nightmare because:

• Different Boxes: Some models give you the raw code, others give you pre-cooked answers.
• Different Shapes: One model needs a square image, another needs a rectangle. One needs 3 colors (Red, Green, Blue), another needs 12 different "super-colors."
• Hard to Compare: If you want to see which model is better at predicting corn growth, you have to build a custom machine for each model to feed it the data. It's slow, expensive, and confusing.

2. The Solution: The "Universal Adapter" (rs-embed)

The authors built a tool called rs-embed. Think of this as a universal kitchen adapter or a smart translator.

Instead of you having to go to every farm, unwrap every box, and chop every vegetable yourself, you just tell the adapter:

"I want to see what the corn fields in Illinois looked like in July 2019, using the top 5 best AI models."

The adapter does the rest:

• It fetches the data: It goes to the satellite archives (like Google Earth Engine) and grabs the right images.
• It does the prep work: It cuts the images into the exact shape and size each specific AI model needs.
• It runs the models: It feeds the data to all the different AIs at once.
• It gives you a standard result: Instead of getting 5 different messy formats, you get 5 neat, standardized lists of numbers (called embeddings) that you can immediately compare.

3. How It Works (The Magic Behind the Curtain)

The paper describes a system with three main layers, which we can imagine as a highly efficient assembly line:

• The Spec Layer (The Order): You write a simple "order" (a single line of code) saying Where (location), When (time), and Which Models you want.
• The Provider Layer (The Delivery Truck): This part goes out to the satellite data warehouses, grabs the raw images, and cleans them up (removing clouds, fixing colors) so they are ready for the AIs.
• The Embedder Layer (The Chefs): This is where the AI models live. The tool feeds the cleaned images to the models. Some models cook the image instantly; others just look up a pre-cooked answer from a database.
• The Orchestra (The Conductor): This is the brain that manages the traffic. It makes sure the data flows smoothly, doesn't crash the computer, and handles errors (like if a satellite image is missing) without stopping the whole process.

4. Why Does This Matter? (The Taste Test)

The authors tested their tool by trying to predict corn yields in Illinois.

• Before: A researcher would spend weeks setting up different systems to test different models.
• With rs-embed: They ran the experiment with a single line of code.

They found that while some models were great at predicting average yields, they all struggled with "outliers" (fields that were incredibly good or incredibly bad). Because the tool made it so easy to compare them, they could quickly see why they failed and learn how to improve them.

They also visualized the "thoughts" of 16 different AI models. It was like looking at 16 different artists painting the same landscape. Some focused on the rivers, others on the roads, but they all captured the general shape of the land. This helps scientists understand what each model is actually "seeing."

The Bottom Line

rs-embed turns a chaotic, hours-long technical headache into a one-click experience.

It allows scientists to stop worrying about how to get the data and start focusing on what the data means. It's like turning a library where every book is written in a different language and stored in a different room, into a library where you just ask a librarian, "Show me the books about space," and they hand you a perfectly organized stack of translated, ready-to-read books.

The Goal: "Any Model, Any Place, Any Time." No matter which AI you want to use, no matter where on Earth you are looking, and no matter what time of year, the tool gets you the answer instantly.

Technical Summary

1. Problem Statement

The rapid proliferation of Remote Sensing Foundation Models (RSFMs) has created a significant bottleneck in practical adoption and scientific evaluation due to heterogeneity and fragmentation:

• Inconsistent Release Formats: Some models release only precomputed embeddings, while others release only model weights, forcing users to manually fetch imagery and run inference.
• Fragmented Deployment: Models rely on diverse interfaces (e.g., Hugging Face vs. custom repositories) and specific framework versions, creating high configuration and compatibility costs.
• Input Specification Mismatches: Inconsistent definitions for input data (e.g., varying band counts like RGB vs. 6-band vs. 12-band Sentinel-2, different resolutions, and preprocessing requirements) complicate fair downstream comparisons and benchmarking.
• Workflow Complexity: The current workflow requires extensive "glue code" to handle data acquisition, preprocessing, model loading, and inference, making it difficult to compare models across different locations and time ranges.

2. Methodology: The rs-embed Library

The authors propose rs-embed, a Python library designed to unify the workflow around the Region of Interest (ROI). It abstracts away the complexity of data fetching and model inference through a modular, four-layer architecture:

A. Specification Layer

Users define requests using standardized specifications rather than raw code:

• Spatial Spec: Defined via bounding boxes or point buffers with Coordinate Reference Systems (CRS).
• Temporal Spec: Defined by year or time ranges (left-closed, right-open intervals) with observation synthesis strategies (e.g., median compositing).
• Output Spec: Determines the embedding format:
  • Pooled Mode: Aggregates spatial dimensions into a fixed-length vector ($z \in \mathbb{R}^d$) for retrieval/tabular tasks.
  • Grid Mode: Preserves spatial context ($z \in \mathbb{R}^{h \times w \times d}$) for pixel-level tasks.
• Sensor Spec: Defines raw imagery requirements (data source, bands, resolution, cloud limits, compositing method).

B. Provider Layer

This layer decouples heterogeneous data sources from model inference:

• Wraps cloud APIs (e.g., Google Earth Engine) into standardized numeric tensors.
• Handles projection, resampling, and spatiotemporal filtering.
• Converts observations into a consistent $(C, H, W)$ NumPy/Xarray format, hiding authentication and query complexity.

C. Embedder Layer

The core engine for feature extraction, utilizing an object-oriented design:

• Unified Interface: A base Embedder class with standard APIs (get_embedding, get_embeddings_batch, describe) wraps diverse models.
• Dual Acquisition Modes:
  • On-the-fly: Runs forward inference on raw imagery provided by the Provider Layer.
  • Precomputed: Queries pre-stored embeddings in the cloud (e.g., Alpha Earth) based on spatial/temporal specs without executing a deep learning graph.

D. Orchestration & Execution

To handle large-scale processing, the system implements a high-performance parallel pipeline:

1. Orchestration: Validates parameters and splits workloads into sub-batches to manage memory.
2. Prefetch: Deduplicates and fetches inputs in parallel via thread pools, caching data to avoid redundant downloads.
3. Inference: Reuses embedder instances to avoid repeated weight loading; prioritizes batch APIs for throughput.
4. Export: Asynchronously writes results to disk (.npz/.netcdf) to overlap I/O with computation.

• Resilience: Features failure isolation (point/model-level), bounded retries with exponential backoff, and structured manifests for partial success recovery.

3. Key Contributions

1. Unified ROI-Centric Interface: Enables users to retrieve embeddings from any supported model for any location and time with a single line of code.
2. Standardized Benchmarking Infrastructure: Provides a reproducible environment to fairly compare models by enforcing consistent input preprocessing, sensor specifications, and output formats.
3. Scalable Engineering: Delivers a high-throughput, fault-tolerant pipeline capable of generating embeddings at scale, supporting both on-the-fly inference and precomputed retrieval.
4. Open Ecosystem: The library is open-source, supporting extensibility to new models and data providers (e.g., Microsoft Planetary Computer).

4. Experimental Results

The authors validated rs-embed through two primary experiments:

• Use Case: Maize Yield Mapping (Illinois, USA):

  • Task: Regression to predict maize yield using embeddings from multiple RSFMs as features.
  • Setup: 991 sampling points, training a Random Forest regressor on embeddings from 2019.
  • Result: The Agrifm model achieved the highest $R^2$. However, the study highlighted that even top models struggle with extreme outliers (very high/low yields), demonstrating the utility of rs-embed for rapid model comparison and identifying limitations.

• Embedding Visualization:

  • Task: Visualizing and comparing embeddings from 16 different models over a specific location in Shanghai (2022).
  • Result: Despite differences in training objectives and architectures, the embeddings generally captured key land-cover structures (e.g., rivers). The tool successfully visualized the varying spatial resolutions and channel dimensions of different models, confirming the library's ability to harmonize disparate outputs.

5. Significance and Future Impact

• Lowering Barriers: rs-embed drastically reduces the engineering overhead required to utilize RSFMs, making advanced representation learning accessible to domain scientists who are not ML experts.
• Reproducibility: By standardizing the input/output pipeline, it enables fair, apples-to-apples comparisons of foundation models, which is critical for scientific progress.
• Scalability: The architecture supports the generation of massive embedding datasets required for training downstream models or creating global benchmarks.
• Extensibility: While currently focused on remote sensing, the ROI-centric design is naturally extensible to broader geospatial modalities, potentially serving as a unified embedding layer for diverse Earth observation data types.

The paper concludes that rs-embed is a foundational step toward a composable, open ecosystem for remote sensing AI, facilitating the transition from isolated model research to unified, large-scale geospatial applications.