🤖 machine learning

From Pixels to Patches: Pooling Strategies for Earth Embeddings

This paper introduces the EuroSAT-Embed dataset to demonstrate that advanced pooling strategies, particularly those leveraging distributional statistics like covariance, significantly outperform standard mean pooling in aggregating pixel-level geospatial embeddings for region-level classification, especially under challenging geographic distribution shifts.

Original authors: Isaac Corley, Caleb Robinson, Inbal Becker-Reshef, Juan M. Lavista Ferres

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Isaac Corley, Caleb Robinson, Inbal Becker-Reshef, Juan M. Lavista Ferres

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to identify different types of neighborhoods in a city using a high-tech satellite camera.

The Problem: The "Pixel" vs. The "Neighborhood"

Your satellite camera is incredibly powerful. It takes a picture of the ground and breaks it down into millions of tiny dots called pixels. For every single dot, the camera doesn't just see "green" or "brown"; it creates a complex, secret code (an embedding) that describes exactly what that dot is (e.g., "this specific pixel is a pine tree," "this one is a red roof," "this one is a wet patch of grass").

However, your job isn't to identify individual trees or roofs. You need to identify the whole neighborhood (a "patch"). A neighborhood might be a mix of houses, trees, roads, and cars.

The Mistake Most People Make:
Traditionally, when people tried to summarize a whole neighborhood from these millions of tiny codes, they used a method called Mean Pooling.

The Analogy: Imagine you want to know the "average personality" of a whole classroom. The old way was to take every student's personality, add them all up, and divide by the number of students.
The Flaw: If you have a classroom with 50 quiet students and 1 loud, energetic student, the "average" personality might come out as "slightly quiet." You've lost the energy of that one loud student! In a neighborhood, if you just average everything, you might miss the fact that it's a chaotic mix of a factory, a park, and a school. You lose the variety and the extremes.

The Experiment: Testing New Ways to Summarize

The authors of this paper asked: "What if we don't just take the average? What if we look at the whole story of the neighborhood?"

They created a new dataset called EuroSAT-Embed (think of it as a giant library of these secret codes for 81,000 different neighborhoods). They tested 13 different ways to summarize a neighborhood, from simple to complex.

Here are the main characters in their story:

The Average (Mean Pooling): The old standard. Simple, but boring. It ignores the drama and variety within the neighborhood.
The "Stats" Package (Stats Pooling): This is the paper's favorite. Instead of just the average, it reports:
- The Average (typical pixel).
- The Minimum and Maximum (the most extreme pixels).
- The Standard Deviation (how much things vary).
- The Metaphor: Instead of saying "The class is average," this method says, "The class is mostly quiet, but there is one very loud student, and the energy levels vary wildly." This captures the texture of the neighborhood.
The "Covariance" Package: The super-advanced detective. It looks at how different features relate to each other (e.g., "Whenever there is a red roof, there is usually a driveway nearby"). It's very accurate but requires a lot of memory, like carrying a massive encyclopedia.

The Results: Why "Variety" Wins

They tested these methods in two scenarios:

The Easy Test (Random Split): They tested on neighborhoods they had seen before, just mixed up. Everyone did okay.
The Hard Test (Spatial Split): They tested on entirely new cities they had never seen. This is where the real world lives.

The Big Discovery:

Mean Pooling (The Average) failed badly in the new cities. Because it ignored the variety, it got confused when the "average" neighborhood looked different in a new place.
Stats Pooling (The Variety) was a superhero. It improved accuracy by up to 6% and reduced the "confusion gap" by 50%.
Why? Because real-world neighborhoods are messy. A "residential" area in one city might look different from a "residential" area in another. By keeping track of the variability (the min, max, and spread), the model could recognize the pattern of a neighborhood even if the specific houses looked different.

The Takeaway: What Should You Do?

The authors give a simple "Three-Tier Strategy" for anyone using these satellite models:

Tier 1 (The Baseline): If you are lazy or have very limited computer power, use Mean Pooling. It's okay, but you're leaving performance on the table.
Tier 2 (The Sweet Spot): Use Stats Pooling. It takes a little more space (4 times bigger than the average), but it gives you a massive boost in accuracy, especially when you move to new locations. This is the recommended default.
Tier 3 (The Power User): If you have a supercomputer and need the absolute highest accuracy possible, use Covariance Pooling. It's the most detailed but also the heaviest.

In a Nutshell

Don't just take the "average" of a picture. Real life is full of extremes and variations. By capturing the min, max, and spread of the data (the "Stats"), you can understand the world much better, especially when you travel to new places. It's the difference between describing a city as "an average of its parts" versus describing it as "a vibrant, chaotic mix of extremes."

1. Problem Statement

The paper addresses a critical bottleneck in the application of Geospatial Foundation Models (GFMs): the input-label resolution mismatch.

Context: Modern GFMs (e.g., AlphaEarth, OlmoEarth, Tessera) generate dense, pixel-level embedding vectors (e.g., 10m resolution). However, many downstream remote sensing tasks (land cover classification, crop monitoring) operate on coarser spatial units (patches, fields, or regions) with a single label per patch.
The Challenge: To use these fixed, pre-computed pixel embeddings for patch-level tasks, practitioners must aggregate the dense pixel vectors into a single patch representation.
The Limitation of Current Practice: The default aggregation method is Mean Pooling. The authors argue that mean pooling discards valuable intra-patch variability (heterogeneity) and performs poorly when the model encounters spatial distribution shifts (i.e., testing on geographic regions unseen during training).
The Constraint: This study focuses on the fixed-embedding setting, where the underlying encoder is unavailable or inaccessible. Therefore, the aggregation method must be post-hoc and training-free (or fitted only on the training split of the downstream task, not the encoder).

2. Methodology

A. Dataset: EuroSAT-Embed

To rigorously test pooling strategies without re-training encoders, the authors introduced EuroSAT-Embed, a new benchmark dataset containing 81,000 embedding GeoTIFFs.

Source: Derived from three distinct GFMs:
1. AlphaEarth (64-dim)
2. OlmoEarth-Nano (128-dim)
3. Tessera (128-dim)
Base Data: Based on the EuroSAT land-cover dataset (10 classes, 64×64 pixel patches, 10m resolution).
Splits: Evaluation was conducted on two splits:
1. Random Split: Standard random shuffling.
2. Spatial Split: Geographically disjoint split (partitioned by longitude) to simulate real-world spatial distribution shifts and prevent data leakage via spatial autocorrelation.

B. Pooling Strategies Benchmarked

The authors evaluated 13 aggregation methods categorized into three groups:

First-Order Statistics (Training-Free):
- Mean: Standard average.
- Max: Maximum activation per channel.
- Generalized Mean (GeM): Interpolates between mean and max.
- Center-Weighted Mean: Down-weights boundary pixels.
- Median: Robust central tendency.
Distributional Statistics (Training-Free):
- Std: Standard deviation.
- Mean + Std / Mean + Max: Concatenation of complementary signals.
- Stats: Concatenation of Min, Max, Mean, and Std (4× dimensionality).
- Percentiles: Five summary quantiles.
- Median + IQR: Robust location and spread.
- Covariance: Extracts the upper triangle of the pixel covariance matrix (dimensionality scales as $D(D+1)/2$ ).
Parametric Methods (Fitted on Train Set):
- PCA: Projects mean-pooled embeddings to lower dimensions.
- Bag of Visual Words (BoVW): Clusters embeddings via k-means and uses normalized histograms.

C. Evaluation Protocols

Probes: Two downstream classifiers were used to test the quality of the pooled representations:
1. k-Nearest Neighbors (kNN): Tests geometric similarity (k=5, cosine distance).
2. Linear Probe: Multinomial logistic regression (tests linear separability).
Metrics: Accuracy on Random vs. Spatial splits and the Generalization Gap (accuracy drop from random to spatial split).

3. Key Contributions

EuroSAT-Embed Release: The release of three aligned, pixel-level embedding datasets derived from major GFMs, enabling standardized benchmarking of post-hoc aggregation.
Comprehensive Benchmark: A controlled evaluation of 11 training-free and 2 parametric pooling methods across three encoders, two probes, and two split types.
Empirical Findings: Demonstration that distributional statistics significantly outperform mean pooling in spatial generalization scenarios, challenging the industry standard of using mean pooling as a default.

4. Results

A. Performance on Spatial Splits (Generalization)

Mean Pooling: Suffers significantly under spatial shift. For linear probes, it achieved an average accuracy of 87.3% on spatial splits, with a generalization gap of 8.8% (drop from random split).
Distributional Statistics (Stats Pooling): The Stats method (min/max/mean/std) achieved the best balance, reaching 93.0% average accuracy on spatial splits. It reduced the generalization gap to 3.8%, a 57% reduction relative to mean pooling.
Covariance Pooling: Achieved the highest absolute accuracy on two of the three encoders (91.0% on AlphaEarth, 94.4% on Tessera) for linear probes, though it increases dimensionality significantly.
kNN Results: Trends were similar, with Stats leading on spatial splits (88.0% avg) and Center-Weighted Mean performing slightly better on random splits.

B. Accuracy vs. Robustness

The study found no trade-off between accuracy and robustness. Methods that performed best on spatial splits (Stats, Covariance, Mean+Max) also maintained high accuracy on random splits. In contrast, Mean pooling was both less accurate and less robust.

C. Dimensionality Trade-offs

Stats Pooling: Increases dimensionality by 4× (e.g., 128-d $\to$ 512-d).
Covariance Pooling: Increases dimensionality to $D(D+1)/2$ (e.g., 128-d $\to$ ~8,256-d), offering peak accuracy at a high computational/storage cost.

5. Significance and Recommendations

This paper fundamentally shifts the design paradigm for using Geospatial Foundation Models in downstream tasks. It argues that pooling is a first-order design choice, not a trivial preprocessing step.

Proposed Three-Tier Strategy for Practitioners:

Baseline: Use Mean Pooling only as a minimal reference point.
Default Recommendation: Use Stats Pooling (min/max/mean/std). It offers a 4× dimensionality increase but provides a massive boost in spatial generalization (+6% accuracy on spatial splits) and is robust across different encoders and probes.
Peak Performance: Use Covariance Pooling when storage and dimensionality are not constraints and maximum accuracy is required.

Broader Impact:
The findings suggest that the "heterogeneity" within a land-cover patch (e.g., the mix of rooftops, roads, and vegetation in a residential area) contains critical semantic signals that mean pooling erases. Preserving these distributional statistics is essential for building robust Earth observation systems that can generalize to new geographic regions without re-training the foundation model.