📊 statistics

FLUXtrapolation: A benchmark on extrapolating ecosystem fluxes

The paper introduces FLUXtrapolation, a new benchmark designed to evaluate machine learning models' ability to extrapolate ecosystem fluxes under increasingly difficult distribution shifts, thereby addressing the critical challenge of accurately upscaling sparse tower measurements to global scales.

Original authors: Anya Fries, Jacob A Nelson, Martin Jung, Markus Reichstein, Jonas Peters

Published 2026-05-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anya Fries, Jacob A Nelson, Martin Jung, Markus Reichstein, Jonas Peters

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to predict the weather. You have a notebook full of data from 207 specific weather stations scattered across the globe. However, these stations are clustered mostly in Europe and North America, leaving huge gaps in the tropics, deserts, and remote wilderness.

The goal of this paper is to create a test (called FLUXtrapolation) to see how well computer models can predict what's happening in those empty spots, or in future years, based only on the data from the stations they've already seen.

Here is a breakdown of the paper's ideas using simple analogies:

1. The Big Problem: "The Missing Map"

Ecosystems constantly exchange carbon, water, and energy with the air (like trees breathing in CO2 and releasing water vapor). We measure this at specific "towers" (weather stations). But to understand the whole planet's climate, we need to know what's happening everywhere, not just where the towers are.

The Analogy: Imagine you only have photos of your friends' faces in your hometown. You want to draw a picture of a stranger you've never met in a different country. You have to guess their features based on general rules (like "people have two eyes"), but you don't know their specific details (like their unique nose shape).
The Challenge: The computer model is trained on the "hometown" data (the towers) and asked to predict the "stranger" (the rest of the world). This is called upscaling.

2. The Three Levels of Difficulty (The Test Scenarios)

The authors created a benchmark with three levels of difficulty, getting harder each time, to see where models break down.

Level 1: Time Travel (Temporal Extrapolation)
- The Setup: The model learns from a specific tower in 2015–2018 and is tested on the same tower in 2019–2022.
- The Analogy: You learn to predict the weather in your own backyard for next week based on last week's data. It's a bit of a stretch, but the location is the same.
- The Result: This is the "easiest" test. The environment hasn't changed much, just the time.
Level 2: Moving to a New Town (Random Spatial Extrapolation)
- The Setup: The model learns from a group of towers and is tested on different towers it has never seen before.
- The Analogy: You learn to predict the weather in Chicago, and then you are asked to predict the weather in Denver. The climate is different, but it's still a temperate city.
- The Result: This is harder. The model has to adapt to new landscapes and climates it hasn't seen.
Level 3: The Extreme Heat (Temperature-Based Spatial Extrapolation)
- The Setup: The model is tested on the hottest, most extreme locations (like deep tropics) that are very different from where it was trained.
- The Analogy: You learned to predict weather in a cool forest, and now you must predict the weather in a scorching desert. The rules of the game might have changed entirely.
- The Result: This is the hardest test. The model often fails here because the "rules" (how plants react to heat) are different, and some important clues (like soil type) are missing from the data.

3. The Two Types of "Surprises"

The paper explains that models fail for two different reasons, which they call shifts:

Covariate Shift (The "Different Inputs" Problem):
- What it is: The model sees data it has never seen before (e.g., a temperature of 45°C when it only saw 20°C before).
- Analogy: You are a chef who only cooks with apples. Suddenly, someone hands you a durian fruit. You don't know how to handle it because the ingredients are new.
Conditional Shift (The "Different Rules" Problem):
- What it is: Even if the inputs look familiar, the relationship between the input and the output changes because of hidden factors (like soil quality or underground water) that the model can't see.
- Analogy: You know that "rain makes grass grow." But in a specific hidden location, the soil is so salty that rain actually kills the grass. The rule changed, but your model doesn't know about the salty soil.

4. How They Tested the Models

Instead of just asking, "How close was the average guess?" (which can hide big mistakes), they looked at:

The "Tail" Errors: Did the model make huge, catastrophic mistakes on specific days or places? (Like a weather forecast that is usually right but fails completely during a storm).
Different Time Scales: They checked if the model was good at predicting the next hour, the next week, or the whole year. A model might be good at hourly guesses but terrible at predicting seasonal trends.

5. What They Found (The Pilot Study)

They ran a "practice round" using standard computer models (like XGBoost and simple neural networks).

The Surprise: When looking at the average error, all the smart models performed almost the same. They all looked equally good.
The Real Story: When they looked at the worst-case scenarios (the "tails") and different time scales, the models started to separate. Some were much better at handling the extreme heat or the long-term trends than others.
The "Domain Generalization" Trap: The authors tried fancy techniques designed to help models handle "new" data (like Domain Generalization). Surprisingly, these fancy techniques didn't consistently beat the standard, simpler models. In fact, sometimes the simple models were more robust.
The "Hidden Driver" Issue: The model struggled most with NEE (Net Ecosystem Exchange, which tracks carbon). Why? Because carbon exchange depends heavily on hidden biological processes (like what's happening in the soil) that the model can't see. It's like trying to guess a person's mood without seeing their face, only knowing the weather outside.

Summary

FLUXtrapolation is a new, tougher exam for AI models trying to predict Earth's climate. It forces these models to prove they can handle:

New places they've never visited.
Extreme conditions they've never seen.
Hidden factors that change the rules.

The paper concludes that while current models are okay at the basics, they struggle with the hardest parts of the job. To truly understand our planet's future, we need models that don't just memorize the past but can handle the unknown, especially when the "rules" of nature change in ways we can't fully see.

Technical Summary: FLUXtrapolation

Problem Statement

Ecosystem fluxes—the exchange of carbon, water, and energy between terrestrial ecosystems and the atmosphere—are central to modeling the Earth system. While direct observations exist via eddy-covariance towers (e.g., FLUXNET), these sites are sparsely distributed globally. Consequently, producing global flux estimates requires upscaling: training models on observed sites using globally available covariates to predict fluxes in unobserved regions.

This upscaling task is a challenging domain generalization problem characterized by two distinct types of distribution shift:

Covariate Shift ( $P_X$ ): Differences in the marginal distribution of inputs (climate, ecosystem types, environmental conditions) between observed tower sites and target unobserved regions.
Conditional Shift ( $P_{Y|X}$ ): Changes in the relationship between inputs and targets due to unobserved drivers (e.g., soil properties, understory characteristics) that vary across ecosystems but are missing from global datasets.

Existing benchmarks often fail to reflect the specific spatiotemporal nature of flux upscaling, relying on single aggregate metrics that may mask failures on specific sites or fail to capture performance across scientifically relevant temporal scales.

Methodology: The FLUXtrapolation Benchmark

The authors introduce FLUXtrapolation, a benchmark designed to evaluate machine learning methods under progressively harder distribution shifts. The benchmark utilizes hourly data from 207 FLUXNET sites (2015–2022) to predict three target variables: Net Ecosystem Exchange (NEE), Gross Primary Productivity (GPP), and Evapotranspiration (ET).

1. Extrapolation Scenarios

The benchmark defines three scenarios with increasing difficulty:

Temporal Extrapolation: Predicting future years (2019–2022) at sites observed during training (2015–2018). This induces mild shift, primarily in $P_{Y|X}$ .
Random Spatial Extrapolation: Predicting at unseen sites (40 randomly selected sites) while retaining overlap in covariate support. This induces moderate $P_X$ shift.
Temperature Spatial Extrapolation: Predicting at unseen sites with the highest average annual temperatures. This compounds spatial generalization with explicit covariate shift toward warmer conditions.

2. Distribution Shift Diagnostics

To quantify the nature of the shifts, the authors employ two diagnostics:

Covariate Shift ( $P_X$ ): Measured via the balanced accuracy of a domain classifier distinguishing training from test inputs. Higher accuracy indicates larger shift.
Conditional Shift ( $P_{Y|X}$ ): Quantified by the percentage increase in RMSE from training to test after reweighting the test set to match the training marginal distribution of $X$ (using importance weights derived from the domain classifier).

3. Evaluation Protocol

The evaluation moves beyond simple mean or median error:

Domain-Level Analysis: Performance is assessed per test domain (site or site-year) rather than pooled, using the median and 90th percentile of RMSE to capture typical and tail performance.
Multi-Scale Temporal Aggregation: Predictions are evaluated at hourly resolution and aggregated to weekly, seasonal, and site-mean levels. Additionally, anomalies and inter-annual variability (IAV) are assessed to test performance on deviations from seasonal norms.
Skill Score: A summary metric relative to a linear regression baseline is used to compare methods across scenarios with varying intrinsic difficulty.

Key Results (Pilot Study)

A pilot study evaluated several baselines, including linear regression, XGBoost, MLPs, and domain generalization (DG) methods (CORAL, MMD, Group DRO).

Baseline Separation: Under median hourly RMSE, top baselines (XGBoost, MLP, CORAL) performed similarly. However, separation emerged significantly when evaluating the 90th percentile of RMSE (tail performance) and across different temporal aggregations (e.g., site-mean, IAV).
Failure Modes: Standard baselines performed poorly on inter-annual variability and anomalies, often approaching the performance of a constant predictor. This suggests current methods struggle to capture ecosystem dynamics beyond average seasonal behavior.
Domain Generalization Efficacy: DG methods did not consistently outperform Empirical Risk Minimization (ERM) baselines like XGBoost.
- For temporal extrapolation (low $P_X$ shift, moderate $P_{Y|X}$ shift), DG methods offered no advantage.
- For spatial extrapolation (high $P_X$ shift), MMD performance dropped sharply, while CORAL remained competitive with ERM.
- Overall, leveraging explicit shift information did not lead to reliable improvements, a finding consistent with prior DG benchmarks.
Flux Differences: Results varied by target variable. NEE showed the least separation between methods, likely due to its stronger dependence on unobserved biological processes compared to GPP and ET.

Significance and Claims

The paper positions FLUXtrapolation as a realistic and relevant challenge for machine learning under distribution shift, grounded in domain expertise. Its significance is threefold:

Scientific Relevance: Progress on this benchmark directly supports the scientific goal of improving flux upscaling, which is critical for carbon-cycle science and climate policy monitoring (e.g., European forest-carbon monitoring, Amazon dieback risk).
Methodological Rigor: By explicitly defining scenarios with varying shift structures and evaluating performance across tails and temporal scales, the benchmark reveals failure modes that aggregate metrics obscure.
Broader Applicability: The design and lessons learned may transfer to other Earth system science problems characterized by rich temporal data but sparse spatial observations, such as predicting plant traits, tree water use, and soil properties.

The authors conclude that while the benchmark successfully separates methods under specific evaluation criteria, effectively leveraging shift information for model design remains an open challenge. Future work should investigate whether strong methods can generalize to new sites in the expanded FLUXNET network and whether foundation models can mitigate the impact of hidden variables.