🤖 machine learning

TESSERA v2: Scaling Pixel-wise Earth Foundation Models

This paper presents the largest controlled scaling study for pixel-wise Earth-observation foundation models, revealing that pretraining loss is a poor predictor of downstream performance and establishing a compute-allocation rule that enables the creation of compact, high-performing distilled student models which outperform existing products while supporting flexible, low-cost Matryoshka embeddings.

Original authors: Zhengpeng Feng, Sadiq Jaffer, Ira Shokar, Jovana Knezevic, James Ball, Pedro Sousa, Mark Elvers, Madeline Lisaius, Clement Atzberger, Robin Young, Aneesh Naik, Niall Robinson, David Coomes, Anil Madha

Published 2026-08-07

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhengpeng Feng, Sadiq Jaffer, Ira Shokar, Jovana Knezevic, James Ball, Pedro Sousa, Mark Elvers, Madeline Lisaius, Clement Atzberger, Robin Young, Aneesh Naik, Niall Robinson, David Coomes, Anil Madhavapeddy, Srinivasan Keshav

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand the Earth by looking at a giant, messy photo album. Every page is a picture of a specific spot on the planet, but the photos are taken at different times, by different cameras, and half of them are covered in clouds or fog. This is the daily reality for scientists who study our planet using satellites. They need to turn these chaotic, incomplete snapshots into a clear, usable map of what the Earth is doing—whether it's tracking how forests grow, counting crops, or spotting pollution. To do this, they use something called "foundation models." Think of these models as super-smart students who have read every satellite photo ever taken. Instead of giving you the raw, heavy photos, these students summarize the information into a tiny, efficient "ID card" (an embedding) for every spot on Earth. This ID card tells you everything you need to know about that location without needing to carry around the whole photo album. The big question, however, has always been: How do we train these students to be the best possible without wasting a fortune on electricity and time?

This paper, titled TESSERA V2, is a massive experiment to answer exactly that question. The researchers ran 395 different training sessions to figure out the perfect recipe for building these Earth-observing AI models. They discovered a few surprising things. First, they found that the usual way of judging a student's progress—looking at their "homework score" (the pretraining loss)—is actually a terrible predictor of how well they will do in the real world. It's like a student getting an A on a practice test but failing the final exam; the paper shows that relying on this score wastes a huge amount of computing power. Instead, they found the winning strategy is to make the "brain" of the model (the encoder) much bigger and feed it more data, while keeping the part that organizes the answers (the projector) small and fixed.

But there's a catch: a giant brain is expensive to run. If you build a super-smart teacher, it costs too much to use every day. So, the team used a clever trick called "distillation." They trained one giant, 2-billion-parameter "teacher" model using their new rules, and then taught four smaller, "student" models to copy its thinking. These students are tiny compared to the teacher but still incredibly smart. Even cooler, these students can spit out answers in different sizes, like a Russian nesting doll (Matryoshka). You can ask for a tiny 16-dimensional answer that takes up very little storage space but keeps 92% of the accuracy, or a full 128-dimensional answer if you need every last detail. The result is a family of models that are cheaper to store, faster to run, and more accurate at real-world tasks than any other open or private system tested, proving that sometimes, the best way to scale up is to train a giant and then shrink it down.

Technical Summary: TESSERA V2 – Scaling Pixel-Wise Earth Foundation Models

Problem Statement

The application of Earth observation (EO) to downstream tasks faces significant bottlenecks: data preparation is labor-intensive (radiometric calibration, cloud correction, cross-sensor harmonization), ground-truth labels are scarce and region-specific, and raw imagery contains irregular sampling cadences and cloud occlusions. While "embeddings-as-data" products aim to provide analysis-ready vector representations of the Earth's surface, existing systems suffer from rigid specifications. They often offer a single fixed embedding dimension and model size, forcing all users to incur the same storage and I/O costs regardless of their specific budget or task requirements. Furthermore, the scaling laws governing how to allocate pretraining compute (between encoder size, projector size, and data volume) in the EO domain remain poorly understood, with many studies relying on pretraining loss as a proxy for downstream performance—a proxy that may be unreliable due to the unique characteristics of satellite data (e.g., cloud dominance and orbital sampling).

Methodology

1. Downstream-Driven Scaling Study

The authors conducted the largest controlled scaling study for EO to date, involving 395 training runs within a fixed pixel-wise Barlow Twins family.

Architecture: A fixed architectural family of pixel-wise Sentinel-1/2 encoders was selected via a preliminary sweep.
Variables: The study swept across three dimensions: encoder size ( $N_{enc}$ , 16 widths from 7M to 278M parameters), projector size ( $N_{proj}$ , 4 widths), and training data volume ( $D$ , 0.03M to 3,202M d-pixels).
Evaluation: Every model was evaluated on 15 diverse downstream tasks (classification, change detection, regression) from the ALPHAEARTH suite.
Compute Metric: Total training FLOPs were calculated using a specific formula accounting for three encoder passes per step (two augmented views + one mix-up view) and the Barlow Twins cross-correlation cost.

2. Key Findings on Scaling Laws

The study yielded two critical empirical findings that contradict standard language/vision scaling assumptions:

Finding F1 (Loss is a Poor Proxy): The converged pretraining loss barely predicts downstream performance ( $|Pearson\ r| < 0.2$ ). Selecting models based solely on loss requires roughly 2–5× more compute to achieve the same downstream score as selecting based on actual task performance.
Finding F2 (Optimal Allocation): As the training budget grows, the optimal strategy is to increase encoder capacity and training data together, while keeping the projector size fixed. The projector acts as a "disposable training scaffold" rather than a scaling axis. The fitted power laws are $N_{enc} \propto C^{0.36}$ and $D \propto C^{0.63}$ , while $N_{proj} \propto C^{0.011}$ (statistically indistinguishable from zero).

3. Teacher-Student Distillation and MATRYOSHKA Embeddings

To bridge the gap between large, high-performance teachers and deployable products, the authors employed a distillation strategy:

Teacher Training: A massive 2B-parameter pixel-wise encoder was trained using the derived scaling rule (oversized encoder + matched data, fixed projector) on ~14 billion d-pixels.
Distillation: The frozen 2B teacher was used to distill a family of compact students (N, S, M, L sizes, ranging from 1M to 44M parameters).
MATRYOSHKA Representation: Unlike self-supervised pretraining, which identifies subspaces only up to rotation, distillation against a fixed teacher provides the necessary signal to order embedding coordinates. This allows the students to produce nested embeddings where prefixes of dimensions $d \in \{16, 32, 64, 128\}$ are valid representations. A 16-dimensional prefix retains ~92% of the performance of the full 128-dimensional embedding at 1/8th the storage cost.
Inference Optimization: The system uses adaptive bucket sampling, packing variable-length observation sequences into the smallest fixed-length bucket that fits, filling residuals via midpoint resampling. This avoids discarding observations or duplicating data arbitrarily.

Key Results

Performance: The distilled TESSERA v2-2B-L (44M parameters) outperforms every open and proprietary embedding product tested, including ALPHAEARTH (480M parameters) and OlmoEarth-L (308M parameters).
- On a held-out suite of 14 datasets, TESSERA v2 achieved a composite score of 0.647, compared to 0.614 for the next best systems.
- On the 15-task ALPHAEARTH suite, it scored 0.593, surpassing ALPHAEARTH (0.560) and TESSERA v1 (0.541).
Efficiency: The 44M student model achieves state-of-the-art results at an inference cost two orders of magnitude lower than the 2B teacher and significantly lower than larger proprietary models.
Storage Adaptivity: The MATRYOSHKA prefixes allow users to trade storage for performance. The 16-dimensional prefix maintains ~92% of the full score at 1/8th the storage, while the 32-dimensional prefix maintains ~97%.
Artifact Reduction: TESSERA v2 significantly reduces the along-track striping and tile-seam discontinuities visible in v1, while maintaining inter-annual stability.

Significance and Claims

The paper claims to provide a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models, shifting the paradigm from "train large and hope loss correlates" to "train large, select by downstream performance, and distill."

Redefining Scaling for EO: The authors assert that EO scaling laws differ fundamentally from language/vision; relying on pretraining loss is inefficient and misleading. The proposed allocation rule (grow encoder and data, fix projector) is presented as the optimal path for EO.
Distillation as a Necessity for Nested Embeddings: The paper claims that nested embeddings (MATRYOSHKA) cannot be effectively learned via self-supervised prefix losses due to coordinate symmetry issues. Instead, distillation against a fixed teacher is required to supply the ordering signal that makes prefixes usable at different dimensionalities.
Product Viability: TESSERA v2 is presented not just as a model, but as a deployable product family that offers adaptivity. It allows users to select model sizes and embedding dimensions that fit their specific budget constraints without retraining, addressing a key gap in existing "analysis-ready" data products.

The authors plan to release global 10m annual embeddings covering 2017–2025 as version 2 of the TESSERA product, making the code and checkpoints available to the community.