LEPA: Learning Geometric Equivariance in Satellite Remote Sensing Data with a Predictive Architecture

This paper introduces LEPA, a learned architecture that conditions on geometric augmentations to accurately predict transformed satellite image embeddings, effectively overcoming the limitations of standard interpolation in non-convex geospatial foundation model manifolds and significantly improving geometric adjustment performance.

The Gist

Imagine you have a massive library of satellite photos of the Earth. To make these photos easier to use, scientists have already processed them into a special "summary code" (called an embedding). Think of this code like a compact, high-tech recipe card that perfectly describes the flavor of a specific patch of land.

Instead of downloading terabytes of raw photos, researchers just grab these recipe cards. It's fast, cheap, and efficient.

The Problem: The "Grid" Mismatch

Here's the catch: These recipe cards are arranged in a rigid, fixed grid, like tiles on a bathroom floor. But what if a user wants to look at a specific area that doesn't line up perfectly with those tiles? Maybe they want to zoom in, rotate the view to look at a mountain from a different angle, or shift the window slightly.

In the old days, if you wanted to change the angle of a photo, you'd just rotate the picture. But with these "recipe cards," you can't just spin the cards around. If you try to average two cards together (like mixing two paint colors to get a new shade) to fill a gap, you get a mess. The resulting "recipe" describes a landscape that doesn't exist in reality. It's like mixing a recipe for a chocolate cake with one for a pizza and expecting a delicious new dish—it just doesn't work because the "flavor space" is too complex.

The Solution: LEPA (The "Magic Translator")

The authors of this paper, LEPA, realized that instead of trying to mix the cards, we need a translator.

Imagine you have a master chef (the Predictor) who knows exactly how the recipe changes when you rotate the dish or zoom in.

1. The Old Way (Interpolation): You try to guess the new flavor by averaging the old ones. Result: A weird, inedible mush.
2. The LEPA Way: You tell the chef, "Hey, I'm rotating the view 30 degrees." The chef, who has studied the geometry of the landscape, instantly writes a new, perfect recipe card for that rotated view without ever needing to look at the original photo again.

How They Built It

They trained this "chef" using a game called Joint-Embedding Predictive Architecture (I-JEPA).

• The Game: They showed the AI a picture, then gave it a version of the picture that was rotated or resized.
• The Task: The AI had to guess what the "recipe card" for the rotated picture would look like, based only on the original recipe card and the instruction "rotate 30 degrees."
• The Result: The AI learned the rules of geometry. It learned that if you turn a forest 90 degrees, the "forest recipe" changes in a very specific, predictable way.

Why This Matters

• Speed & Cost: Before, if you wanted to look at a rotated area, you had to send the raw satellite photo back to a supercomputer to generate a new recipe. That takes time and money. With LEPA, you just ask the "translator" to do the math instantly.
• Accuracy: The paper tested this by asking, "If we rotate the image, can we find the matching recipe?"
  • The old method (averaging) got it right less than 20% of the time.
  • LEPA got it right over 80% of the time.

The Analogy in a Nutshell

Think of the satellite data as a 3D puzzle.

• Old Method: If you want to see the puzzle from a different angle, you try to glue the existing pieces together in a new shape. It looks broken and wrong.
• LEPA Method: You have a holographic projector. You tell the projector, "Show me the puzzle from the left," and it instantly generates a perfect, new view of the puzzle without needing to rebuild the whole thing from scratch.

This breakthrough means we can finally use these powerful, pre-made satellite summaries for any shape, size, or angle a user needs, making Earth observation faster, cheaper, and much more flexible.

Technical Summary

Here is a detailed technical summary of the paper "LEPA: Learning Geometric Equivariance in Satellite Remote Sensing Data with a Predictive Architecture."

1. Problem Statement

Geospatial foundation models generate precomputed embeddings (compact feature vectors) for satellite remote sensing data to reduce storage and computational costs. However, a critical limitation arises when users define Areas of Interest (AOIs) that do not align with the fixed grid of these precomputed embeddings.

• The Challenge: To align embeddings with user-defined geometries (e.g., rotation, scaling, translation), standard approaches rely on latent-space interpolation (e.g., bilinear interpolation of patch embeddings).
• The Failure Mode: The paper demonstrates that embedding manifolds are highly non-convex. Consequently, linearly combining or interpolating patch embeddings yields vectors that do not correspond to realistic inputs, leading to severe geometric mismatches and information loss.
• Goal: Develop a method to geometrically transform embeddings (equivariance) without re-running the expensive encoder on raw image data.

2. Methodology

The authors propose LEPA (Learned Equivariance-Predicting Architecture), a framework based on Joint-Embedding Predictive Architectures (JEPAs).

Core Concept: LEPA

Instead of averaging vectors, LEPA trains a predictor to learn the mapping between geometric transformations in image space and their corresponding transformations in embedding space.

• Architecture:
  • Student Encoder: Processes the original (unmodified) image context to produce patch embeddings.
  • Teacher Encoder: Processes a geometrically transformed version of the image (e.g., rotated, scaled) to generate target embeddings.
  • Predictor: Takes the student's patch embeddings (context) and transformation parameters (e.g., rotation angle, scale factor) as input. It predicts the target embeddings that would result from applying the transformation to the image.
• Training Strategy:
  • The model is trained using an Image World Model approach. The predictor is conditioned on augmentation parameters to learn to "undo" or "simulate" geometric transformations directly in the latent space.
  • Inductive Biases: The authors experiment with novel conditioned positional encodings, where patch indices are centered around the image center rather than a corner, to better reflect geometric shifts.
  • Datasets: The model is pre-trained on ImageNet-1k and NASA/USGS Harmonized Landsat-Sentinel (HLS) data.

Comparison Baselines

• Standard Interpolation: Nearest-neighbor or bilinear interpolation of patch embeddings.
• Prithvi-EO-2.0: A state-of-the-art MAE-based foundation model used to test the limits of existing architectures.
• I-JEPA: Standard Joint-Embedding Predictive Architecture without specific geometric conditioning.

3. Key Contributions

1. Demonstration of Interpolation Failure: The paper empirically verifies that standard interpolation on patch embeddings fails due to the non-convex nature of the embedding manifold, resulting in unrealistic representations.
2. LEPA Architecture: Introduction of a predictive architecture that conditions on geometric parameters to transform embeddings directly, avoiding the need for repeated, expensive encoder inference passes.
3. High-Performance Equivariance: The proposed method achieves high geometric equivariance, significantly outperforming traditional interpolation methods.
4. Architectural Insights:
   • Evaluation of CLS tokens (classification tokens) shows they improve performance on ImageNet but offer mixed results on diverse remote sensing data (HLS).
   • Analysis of positional encodings reveals that while conditioning on transformation parameters is crucial, the specific method of conditioning (MLP vs. positional encoding) impacts performance differently depending on the training objective.

4. Experimental Results

The evaluation focuses on Mean Reciprocal Rank (MRR), a metric measuring how well the predicted transformed embedding matches the true transformed embedding.

• Baseline Performance: Standard interpolation and unmodified I-JEPA models achieve an MRR below 0.2, indicating poor geometric alignment.
• LEPA Performance:
  • LEPA increases MRR to ~0.7 without fine-tuning.
  • With predictor fine-tuning (training the predictor solely to predict augmentations while keeping the encoder frozen), MRR exceeds 0.8.
• Semantic Segmentation (PANGAEA Benchmark):
  • LEPA and I-JEPA models trained on HLS data perform competitively against specialized remote sensing models (e.g., TerraMind, RemoteCLIP, DOFA).
  • Models trained on ImageNet-1k also show strong performance, particularly on datasets with skewed class distributions (e.g., Marine Debris), suggesting the learned representations are robust even when pre-trained on natural images.
• Qualitative Analysis: Visualizations (PCA projections) show that LEPA predictions closely follow the trajectory of true image-space transformations, whereas interpolation results in artifacts and deviation from the target manifold.

5. Significance and Impact

• Efficiency: LEPA enables the dynamic adjustment of precomputed embedding grids to match arbitrary user-defined geometries without re-processing terabytes of raw satellite imagery. This drastically reduces computational costs and latency in Earth Observation workflows.
• Robustness: By learning the geometric structure of the embedding manifold, LEPA provides a reliable alternative to linear interpolation, which is mathematically unsound for non-convex manifolds.
• Foundation Model Advancement: The work bridges the gap between general foundation models and specific remote sensing needs, demonstrating that predictive architectures can learn complex geometric equivariances, a property often lacking in standard self-supervised models.
• Future Directions: The paper suggests that smaller predictors could suffice (reducing inference cost) and that better inductive biases (e.g., relative positional encodings like RoPE) could further enhance performance.

In summary, LEPA solves the "geometric mismatch" problem in satellite remote sensing by replacing unreliable interpolation with a learned predictive model, enabling accurate, efficient, and scalable manipulation of geospatial embeddings.