💰 quantitative finance

A harmonised dataset for Earth system foundation models

This paper introduces WorldTensor, a harmonised global dataset that aligns diverse environmental and socioeconomic variables onto a standardised 0.25° grid and annual framework to enable the training of multimodal Earth system foundation models that capture coupled human-environment dynamics.

Original authors: Carlos Rodriguez-Pardo, Massimo Tavoni

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Carlos Rodriguez-Pardo, Massimo Tavoni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the Earth as a giant, complex machine with two main engines: the natural world (weather, oceans, forests) and the human world (cities, farms, power plants, economies). For a long time, scientists have built "super-brains" (called foundation models) to understand how these engines work. However, there's been a major problem: the super-brains have only been fed data about the natural engine. They know everything about rain and wind, but they are almost blind to how humans build cities, burn fuel, or move money.

This paper introduces WorldTensor, a massive new "instruction manual" designed to fix this blind spot. Think of WorldTensor not as a single book, but as a giant, perfectly organized library where every single fact about the planet—from a drop of rain to a factory smokestack—is written on the same type of paper, in the same language, and filed in the same drawer.

Here is how the paper breaks it down:

1. The Problem: A Messy Pile of Puzzles

Before WorldTensor, trying to study the Earth and humans together was like trying to solve a puzzle where half the pieces are from a picture of the ocean, and the other half are from a picture of a city, but they are all different sizes, different shapes, and printed in different languages.

Some data is daily (like weather); some is only every few years (like population counts).
Some data is a grid of squares (like a map); some is just a list of dots (like power plant locations).
Some maps use different coordinate systems, making it impossible to line them up.

Scientists had to spend months just cleaning and aligning this data before they could even start their research.

2. The Solution: The "WorldTensor" Library

The authors created WorldTensor, a harmonized dataset that acts like a universal translator and a master organizer.

The Grid: They took hundreds of different data sources and forced them all onto a single, standard grid (0.25 degrees, roughly the size of a large city block). Imagine taking a high-resolution photo, a sketch, and a list of numbers, and printing them all on the exact same graph paper so they line up perfectly.
The Time: They standardized time to a yearly rhythm. If a source had daily data, they summarized it into a yearly average. If a source only had data every 5 years, they filled in the gaps to make a smooth yearly timeline.
The Content: It's a "multimodal" mix. It doesn't just have weather; it has:
- Nature: Oceans, ice, soil, forests, and air quality.
- Humans: Population, GDP (wealth), power plants, roads, and even conflict zones.
- The Mix: How humans change nature (like cutting down forests) and how nature affects humans (like floods hitting cities).

3. How They Built It (The Kitchen Analogy)

Think of the data sources as raw ingredients from different markets. Some are fresh fish (daily weather), some are dried beans (annual census), and some are spices (point locations of factories).

Regridding: They took the "fish" and chopped it into the same size pieces as the "beans."
Rasterizing: They took the "spices" (dots on a map) and spread them out so they looked like a smooth layer of seasoning on the grid.
Standardizing: They made sure every ingredient was measured in the same units (like converting cups to grams) and labeled clearly.
Quality Control: They checked the ingredients to make sure they weren't spoiled. If a number was physically impossible (like a temperature of 500°C on Earth), they flagged it. They also checked that the "map" didn't have tears or seams where the edges didn't match.

4. What It Looks Like

The paper shows that this dataset is huge. It contains over 52,000 files covering the years 1900 to 2025.

It includes 14 different "departments" (like Climate, Energy, Human Systems, Hazards).
It allows a computer to see that a change in emissions (human) is linked to a change in air quality (nature), or that urbanization (human) is linked to flood risk (nature).

5. Why It Matters (The "Super-Brain" Training)

The main goal of WorldTensor is to train Foundation Models.

Before: A super-brain learned to predict the weather but didn't understand that building a dam changes the river flow, or that a war stops people from farming.
Now: With WorldTensor, a super-brain can learn the coupled dynamics. It can learn how human actions and natural forces push and pull against each other on a global scale.

The paper validates this by showing that the data works. When they tested it against real historical events (like the 1997 El Niño or the 2020 pandemic), the data correctly showed the spikes and dips in temperature, emissions, and conflict. It proved that the "library" is accurate and ready for computers to read.

Summary

WorldTensor is a massive, clean, and organized dataset that finally puts the "Human" and "Earth" engines on the same page. It turns a chaotic pile of mismatched maps and numbers into a single, coherent picture of our planet, allowing AI to finally learn how we live on and with the Earth, rather than just studying the Earth in isolation.

Technical Summary: WorldTensor – A Harmonised Dataset for Earth System Foundation Models

Problem Statement
Earth system foundation models have advanced rapidly in simulating physical climate and weather dynamics, primarily leveraging dense reanalysis and satellite data. However, a critical gap remains: these models largely omit the human systems that drive emissions, alter land use, build infrastructure, and determine vulnerability. While the necessary data to describe coupled human–Earth dynamics exists across reanalyses, remote sensing, emissions inventories, and socioeconomic indicators, it remains fragmented. These sources differ in native resolutions, projections, temporal frequencies (ranging from daily to sparse multiyear snapshots), and metadata conventions. This fragmentation hinders the development of truly multimodal foundation models capable of learning coupled dynamics across environmental and human systems at a planetary scale. Current efforts are often limited to the Global North or lack a unified representation that allows for joint reasoning across physical and socioeconomic domains.

Methodology
The authors present WorldTensor, a harmonised global dataset designed to serve as a unified training resource for machine learning workflows. The dataset aligns hundreds of environmental and socioeconomic variables to a standardised 0.25° spatial grid (matching ERA5) and an annual temporal framework.

The construction of WorldTensor involved four core principles: shared spatial support, shared temporal resolution, multimodal domain coverage, and a machine-readable format. The methodology included:

Source Selection: A curated set of high-quality variables was selected based on relevance to coupled human–Earth dynamics (covering physical climate, land/ocean processes, emissions, infrastructure, hazards, and socioeconomic activity) and global coverage. Sources were excluded if they lacked global extent, reliable temporal alignment, or reproducibility.
Spatial Harmonisation: All gridded inputs were regridded to the canonical 0.25° grid. Continuous fields used bilinear resampling, while categorical layers used nearest-neighbor assignment to preserve class boundaries. Point, line, and polygon datasets (e.g., power plants, conflict events) were rasterised into continuous fields or density maps using geometry-specific rules. Special attention was paid to longitude seam handling, polar edge behavior, and avoiding spatial artifacts.
Temporal Harmonisation: Heterogeneous source frequencies were converted to a common annual release model. Daily and monthly data were aggregated using variable-appropriate statistics (e.g., mean, sum, max, min). Sparse socioeconomic datasets were interpolated between anchor years. Static variables (e.g., bathymetry, soil texture) were released as time-invariant layers.
Data Model and Packaging: The dataset is distributed as compressed NetCDF files adhering to CF metadata conventions. The structure is modular: temporal variables are stored as <domain>/<variable>/<YYYY>.nc, while static variables reside in static/<group>/<variable>.nc. The dataset includes 757 released units (658 temporal variable families and 99 static layers), totaling approximately 46 GB.

Key Contributions

WorldTensor Dataset: A unified, harmonised global dataset spanning 14 top-level domains: climate, extremes, air quality, emissions, land use, vegetation, hydrology, cryosphere, ocean, agriculture, energy, human systems, hazards/conflict, and static context.
Standardised Representation: The dataset provides a single representation for variables that were previously incompatible, enabling models to learn cross-domain couplings (e.g., between emissions and atmospheric composition, or land-use change and hydrology).
Reproducible Infrastructure: The authors provide a complete processing pipeline, including source-specific download modules, harmonisation scripts, and automated quality control tools.
Machine Learning Interfaces: The release includes PyTorch examples (WorldTensorYearDataset and WorldTensorPatchDataset) that demonstrate how to ingest the data into foundation model training workflows, handling both temporal and static layers.

Results and Technical Validation
The paper validates WorldTensor through a five-layer framework:

Physical Plausibility: 54 representative variables were tested against physical bounds; fewer than 0.1% of pixels exceeded expected ranges. Global area-weighted means for temperature, precipitation, NDVI, population, and methane emissions matched authoritative benchmarks (WMO, ERA5, MODIS, GPW, EDGAR).
Internal Consistency: Land-use state fractions were verified to sum correctly against the land budget (excluding ice/water). Cumulative hazard and infrastructure variables were confirmed to be monotonically non-decreasing.
Temporal Signal Fidelity: The dataset successfully captured historical anomalies without prior knowledge, including the 2015–16 El Niño (temperature spike), the Syrian civil war (conflict fatalities), the Pinatubo eruption (solar radiation dip), and the 2020 COVID-19 emissions perturbation (attenuated at annual resolution).
Spatial Structure: Empirical semivariograms confirmed that broad spatial autocorrelation is preserved across domains. Environmental variables showed long-range autocorrelation (e.g., temperature ~~2580 km), while human-system variables like GDP remained more localised (~~300 km).
Multivariate Coherence: Independent Component Analysis (ICA) on non-static variables recovered distinct eco-climatic zones (boreal forests, tropical rainforests, arid regions) without geographic supervision, confirming the data encodes geographically meaningful structure.
Feature Utility: Geospatial embedding analysis showed that Random Convolutional Features (RCF) extracted from WorldTensor patches significantly outperformed simple Fourier positional encoding for predicting held-out variables (mean $R^2$ of 0.63 vs. 0.34), demonstrating the dataset preserves fine-grained spatial texture and multi-variable co-occurrence patterns.

Significance and Claims
The paper positions WorldTensor as a research infrastructure rather than a benchmark tied to a specific model class. Its primary significance lies in addressing the data fragmentation that constrains the next generation of Earth system foundation models. By co-locating environmental, human, and static dimensions in a single resource, WorldTensor provides the "raw material" for models to learn how human systems modify and respond to planetary dynamics.

The authors claim that this unified representation is essential for tasks such as:

Training general-purpose geospatial embeddings.
Transfer learning across environmental and socioeconomic prediction tasks.
Downscaling or gap-filling sparse socioeconomic fields.
Feature generation for climate-impact and risk models.

The paper acknowledges limitations, noting that the annual resolution is less suited for sub-daily weather forecasting or event-level detection of short-lived extremes, and that the 0.25° grid may smooth fine-grained urban structures or introduce apparent precision for sparse point data. However, it asserts that WorldTensor successfully enables the study of coupled human–Earth dynamics at a planetary scale, moving beyond the physical climate system to include the human drivers and responses that shape global change.