🤖 machine learning

GHGbench: A Unified Multi-Entity, Multi-Task Benchmark for Carbon Emission Prediction

This paper introduces GHGbench, a unified open benchmark for predicting company- and building-level carbon emissions that addresses data fragmentation by providing large-scale, harmonized datasets and reveals that out-of-distribution generalization is the primary challenge, where tabular foundation models and multimodal remote-sensing embeddings significantly outperform traditional baselines.

Original authors: Yifan Duan, Siyuan Zheng, Lihuan Li, Chao Xue, Flora Salim

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Duan, Siyuan Zheng, Lihuan Li, Chao Xue, Flora Salim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how much pollution two very different things produce: a giant corporation and a single office building.

For a long time, trying to do this with computers has been like trying to compare apples and oranges using different scales, different units of measurement, and data scattered across a thousand different locked filing cabinets. Some data was hidden behind paywalls, some was in different languages, and some was just missing.

GHGbench is the new "universal translator" and "testing ground" the authors built to fix this mess. Think of it as a giant, open-source video game level where different AI models can race to see who is best at predicting carbon emissions.

Here is a simple breakdown of what they did and what they found:

1. The Two "Tracks" of the Game

The benchmark has two distinct levels, because companies and buildings are very different beasts.

The Company Track (The Corporate Giant): This level uses data from over 12,000 companies. It's like trying to guess a company's pollution based on its size, what industry it's in (like banking or tech), and its financial reports. The authors gathered 32,000 records of this data.
The Building Track (The Concrete Block): This level is much harder. It uses data from nearly 100,000 individual buildings across 26 cities in the US, Australia, and Singapore. It's like trying to guess how much electricity a specific house uses just by looking at its address, its size, and the weather. They harmonized data from 13 different city portals into one single, clean format.

2. The Rules of the Race

The authors didn't just let the AI models guess randomly. They set up strict rules to see if the models were actually smart or just memorizing the answers:

The "Same Neighborhood" Test: Can the model predict emissions for buildings it has seen before? (Easy mode).
The "New City" Test: Can the model predict emissions for a city it has never seen before? (Hard mode). This is like teaching a student in New York and then testing them in Sydney.
The "Time Travel" Test: Can the model predict next year's emissions based on this year's data?

3. The Big Surprises (What They Found)

The authors ran many different types of AI (from simple math models to fancy "foundation models" that are like super-intelligent brains) and found three major things:

Surprise #1: Buildings are harder than companies.
Predicting a company's pollution is like guessing a person's weight based on their height and job title; it's fairly predictable. Predicting a building's pollution is like guessing a person's weight based on their address, because it depends on invisible things like how many people are inside, what time they turn on the lights, and how old the fridge is. The data doesn't tell you these things, so the AI struggles more.
Surprise #2: The "New City" problem is huge.
The biggest gap in performance wasn't between the different AI models; it was between "seeing the data before" and "seeing it for the first time."
- The Analogy: Imagine a student who memorizes the answers to a math test. They get 100% on that test. But if you give them a test with the same math problems but in a different country's currency, they might fail.
- The Result: Most models failed miserably when moved to a new city. However, one specific model called TabPFN (a "table foundation model") was the first to show it could actually learn the rules of the game rather than just memorizing the answers, beating the old standard models when moving to new cities.
Surprise #3: Satellite photos are the secret weapon for new cities.
When the AI models got stuck trying to guess emissions in a new city, adding satellite images of the buildings helped.
- The Analogy: If you only have a list of addresses (text), you might guess a house in a snowy city is heated differently than one in a hot city. But if you can see the roof in a satellite photo, you can tell if it's covered in snow, has solar panels, or is surrounded by trees. The satellite images gave the AI a "visual clue" that helped it generalize to places it had never seen before.

4. The "Catastrophic" Failures

The paper also highlights where the AI breaks down completely.

The "City Transfer" Crash: When moving from a city with lots of data to a city with very little data, some models didn't just get a little worse; they crashed completely, giving predictions that were wildly wrong (like predicting a building emits negative pollution).
The "Sector Factor" Ceiling: Trying to guess emissions just by looking up a "standard pollution number" for an industry (e.g., "All banks emit X amount") is too blunt. It misses the specific details of individual companies, leading to huge errors.

Summary

GHGbench is a toolkit that finally lets researchers compare AI models fairly. It shows that while AI is getting good at predicting pollution for companies it knows well, predicting pollution for new buildings in new cities is still very hard. However, using a specific type of "foundation model" combined with satellite photos is the most promising path forward to solving this puzzle.

The authors have made all their code, data recipes, and results open to everyone, so other scientists can build on this work without having to start from scratch.

Technical Summary of GHGbench: A Unified Multi-Entity, Multi-Task Benchmark for Carbon Emission Prediction

Problem Statement
Quantitative prediction of greenhouse gas (GHG) emissions at the entity level is critical for climate policy, finance, and urban operations. However, the field lacks a unified, open benchmarking ecosystem. Existing resources are fragmented: company-level studies often rely on licensed data feeds (e.g., CDP, Bloomberg) with incompatible schemas, while building-level data is scattered across city portals with varying reporting standards. Consequently, it is difficult to compare models on open, reproducible tasks that reflect real-world shifts in geography, time, and data schema. Furthermore, current benchmarks typically evaluate either companies or buildings in isolation, failing to address the structural differences in prediction difficulty between these two entity types or to test generalization under distribution shifts (e.g., cross-region or cross-city transfer).

Methodology
The authors introduce GHGbench, an open dataset and benchmark designed to jointly evaluate company- and building-level carbon emission prediction under a shared task suite.

Dataset Construction:
- Company Track: Contains 32,830 company-year records from 12,087 firms (2018–2023). It integrates public GHG disclosures (Scope 1+2 and Scope 3) with financial signals (revenue, EBITDA, market cap) and sectoral data (ExioML/EXIOBASE factors). The data is released as a reconstruction script using free public APIs to respect source terms.
- Building Track: Harmonizes 491,591 building-year records from 13 open sources across 26 metropolitan areas in the U.S., Australia, and Singapore. It unifies property taxonomies, coordinates, and annual GHG targets. The dataset is augmented with NASA POWER climate covariates and Sentinel-2 + Clay multimodal remote-sensing embeddings.
Benchmark Design:
- Tasks: The benchmark defines canonical splits for in-distribution regression, cross-region/city transfer (leave-one-out), temporal hold-out, cross-property-type stress testing, and short-horizon forecasting.
- Evaluation Protocol: All headline results are reported as the mean over multiple seeds with 95% confidence intervals derived from 1,000-resample paired-bootstrap tests on shared test rows. This ensures statistical rigor in model ordering claims.
- Baselines: The evaluation spans naive references, classical machine learning (Ridge, Random Forest, XGBoost, LightGBM), neural networks (MLP, FT-Transformer), time-series foundation models (Chronos, TimesFM, Moirai), and a tabular foundation model (TabPFN v2). An LLM panel is also evaluated for the company track.

Key Contributions

Unified Open Dataset: The first open dataset and benchmark to jointly evaluate company and building emissions across 26 metros and 12,000+ firms, featuring canonical splits and multimodal signals.
Rigorous Evaluation Framework: A multi-task suite with strict statistical protocols (paired-bootstrap significance testing) to distinguish genuine model improvements from variance, particularly in out-of-distribution (OOD) settings.
Open-Source Release: Includes code, data reconstruction recipes, and a quickstart kit to ensure reproducibility without requiring access to proprietary data feeds.

Key Results and Findings
The paper reports three primary benchmark-level findings:

Cross-Entity Difficulty Asymmetry: Building emissions are structurally harder to predict than company emissions. While company emissions are dominated by firm size (revenue is the top predictor), building emissions depend on unobserved factors like occupancy schedules, runtime, and occupant behavior. Consequently, the best building model (TabPFN v2, $R^2 \approx 0.48$ ) significantly underperforms the best company model (XGBoost, $R^2 \approx 0.88$ ).
Dominance of the ID-to-OOD Gap: The performance gap between in-distribution (ID) and out-of-distribution (OOD) settings dwarfs any performance differences between models within the same setting.
- In the building track, cross-city transfer causes a collapse in performance for tree-based models (e.g., LightGBM $R^2$ drops from ~0.41 to ~0.03).
- Notably, TabPFN v2 is the first baseline to open a paired-bootstrap-significant gap over tuned gradient-boosted trees on a multi-city, multi-source building-emissions task in the OOD setting ( $R^2 = 0.482$ vs. $0.401$ for LightGBM). This advantage is attributed to in-context pretraining rather than architecture alone.
Multimodal Lift in Transfer Settings: Multimodal remote-sensing embeddings (Sentinel-2 + Clay) provide negligible gains in in-distribution settings (where tabular features like latitude/longitude suffice) but offer measurable improvements in cross-city transfer. They act as a generalizable geographic prior where tabular generalization breaks.

Systematic Failure Modes
The benchmark exposes specific failure modes:

Catastrophic City Transfer: Performance varies wildly by city; some cities (e.g., Seattle, Hobart) show negative $R^2$ for tree models, indicating severe distribution shifts.
Sector-Factor Lookup Ceiling: Simple sector-factor lookups (e.g., ExioML) fail to capture firm-level magnitude, often over-predicting by orders of magnitude for specific sub-sectors (e.g., financial services).

Significance and Claims
The authors position GHGbench as a necessary step toward open, comparable evaluation of carbon-emission prediction, moving the field away from reliance on paywalled data. The paper modestly claims that the benchmark reveals where current deep learning and foundation models succeed (in-context tabular learning) and where they fail (cross-city generalization without multimodal priors). It identifies the "sector-factor lookup ceiling" and "catastrophic city transfer" as systematic challenges that future research must address. The work does not claim to solve carbon prediction but provides the infrastructure to measure progress rigorously.