What's in an Earth Embedding? An Explainability Analysis of Location Encoders

Technical Summary: What's in an Earth Embedding? An Explainability Analysis of Location Encoders

Problem Statement
Geographic Implicit Neural Representations (INRs) learn to map any Earth coordinate to a "location embedding," implicitly encoding vast geospatial data into neural network weights. These embeddings serve as compact, general-purpose representations for downstream tasks like temperature mapping, air quality prediction, and species identification. However, a critical gap exists: there are no principled tools to audit what specific geographic or semantic information these embeddings capture. Unlike traditional image models where inputs correspond to single images, location embeddings correspond to latitude/longitude coordinates, making their internal information content opaque. Current evaluation relies solely on predictive performance on benchmarks, which fails to explain why embeddings succeed or when they might fail due to reliance on spurious proxies (e.g., infrastructure rather than poverty indicators).

Methodology
The authors propose a framework to decompose location embeddings into three complementary, human-interpretable feature sets: sparse latent concepts, natural language concepts, and visual features.

1. Sparse Latent Concepts (Unsupervised):

   • Approach: The authors employ Sparse Autoencoders (SAEs), specifically the BatchTopK method, to disentangle location embeddings into a sparse dictionary of concepts.
   • Mechanism: An SAE projects embeddings into a higher-dimensional space under a sparsity constraint, activating only a small number of neurons ($k=20$) per sample.
   • Metrics: To interpret these neurons, they introduce two metrics:
     • Visual Monosemanticity (MS): Measures if a neuron activates for images with similar visual features (using correlation between activation and visual similarity).
     • Geographic Monosemanticity (GeoMS): Measures if a neuron activates over a local or global geographic region (using inverse Haversine distance).
   • Datasets: SAEs were trained on two distributions: Uniformly At Random (UAR) across landmasses (for SatCLIP, Climplicit) and human-visited areas from the Geo-YFCC dataset (for GeoCLIP).

2. Natural Language Concepts (Supervised/Dictionary-based):

   • Approach: The authors adapt the Sparse Linear Concept Embeddings (SpLiCE) framework to decompose embeddings into a predefined dictionary of 2,000 geospatial text concepts (e.g., "city," "forest").
   • Alignment: Since SpLiCE requires a shared semantic space between location and text, the authors either utilized models pre-trained with CLIP alignment (GeoCLIP) or manually aligned other encoders by freezing the location encoder and fine-tuning a text encoder (OpenCLIP ViT-L) on the Git-10M dataset.
   • Optimization: They apply $\ell_1$ regularization to encourage sparsity, aiming for 5–20 active concepts per example, and measure reconstruction via cosine similarity and Mean Squared Error (MSE).

3. Visual Features (Feature Attribution):

   • Approach: The authors use CLIP Surgery to generate saliency maps, identifying which regions of input imagery (natural or satellite) most influence the model's prediction.
   • Adaptation: Since location encoders lack a "null class" (unlike language models), the authors use the North Pole as a heuristic "null" embedding to approximate noise.
   • Pipeline: For satellite imagery, they computed saliency maps, thresholded them into binary masks, cropped salient regions, and clustered these crops using t-SNE and DBSCAN to identify recurring visual patterns (e.g., roads, roundabouts).

Key Results

• Sparse Latent Concepts:

  • Reconstruction: SatCLIP and Climplicit embeddings were reconstructed with near-perfect $R^2$ scores (>0.92) by the SAE, whereas GeoCLIP achieved lower scores (~0.52–0.65).
  • Interpretability: High-monosemanticity neurons captured distinct geographic patterns. Examples include neurons activating specifically for Amazon rainforests, Australian deserts, and Middle Eastern archaeological sites.
  • Artifacts: The analysis successfully identified visual artifacts in the pretraining data, such as specific anomalies in Sentinel-2 imagery around Greenland.

• Natural Language Concepts (SpLiCE):

  • Structure Preservation: SpLiCE maintained directional structure in the embedding space (cosine similarity > 0.29) while using significantly fewer active concepts than the original embedding dimension.
  • Encoder Differences:
    • GeoCLIP: Showed fine-grained, specific signals in highly photographed areas (e.g., Paris) but struggled with less photographed regions (e.g., Siberia).
    • SatCLIP: Captured broader land-cover trends (e.g., "tundra," "desert") more consistently across regions.
    • Climplicit: Showed mixed interpretability, performing well on coarse land-cover signals (Sahara) but less coherently on others.

• Visual Features (Saliency Maps):

  • Natural Imagery (GeoCLIP): Attention focused on distinctive architectural landmarks, natural features (trees, sky), and textual cues (street signs).
  • Satellite Imagery (SatCLIP): Attention concentrated on structural boundaries, including road patterns (e.g., roundabouts in European cities vs. suburban clusters in US cities), land-water edges, and large urban structures like stadiums.
  • Consistency: The visual features highlighted by saliency maps aligned with the semantic concepts found in the text decomposition (e.g., urban structures vs. natural biomes).

Significance and Claims
The paper claims to provide the first step toward interpretable geospatial representations for geographic INRs. By demonstrating that location embeddings can be decomposed into human-interpretable representations (latent, textual, and visual) while retaining high reconstruction capability, the authors reveal that these "black box" encoders encode systematic geographic structures such as forests, deserts, urban features, and climate signals.

The authors emphasize that these methods offer complementary insights:

• For Users: The ability to audit which encoder captures specific signals (e.g., urban infrastructure vs. broad biomes) aids in selecting the right model for downstream tasks.
• For Developers: The framework enables debugging by identifying concept gaps, irrelevant visual features, or low-quality data artifacts (as seen in the Greenland anomaly).

The paper remains modest, acknowledging that current interpretability requires manual effort (e.g., inspecting neuron activations, aligning spaces) and that future work should focus on "interpretable-by-design" frameworks or deeper tailoring of these methods specifically for geospatial models.