🤖 AI

AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting

The paper introduces AirQualityBench, a global multi-pollutant benchmark that evaluates air-quality forecasting models under realistic conditions of uneven coverage and structured missingness, revealing that strong performance on sanitized datasets does not reliably transfer to fragmented, real-world monitoring streams.

Original authors: Xing Xu, Xu Wang, Yudong Zhang, Huilin Zhao, Zhengyang Zhou, Yang Wang

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xing Xu, Xu Wang, Yudong Zhang, Huilin Zhao, Zhengyang Zhou, Yang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the weather. In the past, scientists taught these robots using "perfect" weather data: clean, complete, and neatly organized spreadsheets where no numbers were missing. It was like teaching a student to drive on a perfectly smooth, empty racetrack with no traffic lights or potholes.

The problem? Real life isn't a racetrack. Real air quality monitoring is messy. Some sensors break, some stations are far apart, and some pollutants are harder to track than others. When you take a robot trained on a perfect track and put it on a bumpy, real-world road, it often crashes.

AirQualityBench is a new "driving test" designed to see if air quality prediction models can actually handle the messy reality of the real world.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The "Sanitized" Trap

Most current air quality benchmarks are like photoshopped maps.

The Old Way: If a sensor was broken and missing data for a day, researchers would just "fill in the blank" with a guess (like drawing a straight line between two points). They also often converted all the numbers into a generic "score" (like a grade of 0 to 100) to make them easier to compare.
The Issue: This hides the real challenges. In the real world, data is often missing in specific patterns (e.g., gas sensors work less often than dust sensors), and different places measure things in different units (some use grams, some use milligrams). By "cleaning" the data, researchers were testing models on a fantasy world, not the real one.

2. The Solution: The "Real-World" Gym

The authors created AirQualityBench, which is like a gym with obstacles instead of a smooth treadmill.

The Size: It's massive. Instead of looking at just one city, it covers 3,720 monitoring stations all over the globe.
The Time: It tracks data from 2021 to 2025.
The Messiness: They did not fill in the missing numbers. If a sensor was broken, the data is marked as "missing." The models have to learn to predict the future even when they don't have all the past information.
The Real Units: They didn't convert everything to a generic score. They kept the real physical units (like micrograms per cubic meter). This means the error you see is the actual amount of pollution the model got wrong, not just a math score.

3. The Six "Pollutants" (The Characters)

The benchmark tracks six different types of air pollutants, each acting like a different character with its own personality:

PM2.5 & PM10: These are tiny dust particles. They are everywhere and have the most data.
O3, NO2, SO2, CO: These are gases. They are much harder to track, with many more gaps in the data (like a character who shows up late to the party).
The Challenge: A model has to predict all six at once, even though some of them are missing data much more often than others.

4. The Results: Who Actually Passed the Test?

The authors tested many different types of AI models (some based on graphs, some on time sequences, some on attention mechanisms) on this new, messy benchmark.

The Surprise: Models that looked like "stars" on the old, clean datasets often struggled here. Being good at predicting on a smooth track didn't mean they could drive on a bumpy road.
The Winners: The models that performed best were those that could handle fragmented data well. They didn't just rely on fancy, complex math; they were robust enough to say, "I don't have data from Station A, but I can guess based on Station B and the wind direction."
The Trade-off: The paper found a "Goldilocks" zone. The most accurate models were often very heavy and slow (like a luxury truck), while the fastest models were often too inaccurate (like a bicycle). The best models for real-world use need to find a balance between being smart enough to handle the mess and fast enough to run on real computers.

5. Why This Matters

Think of AirQualityBench as a stress test.

Before this, we were checking if models could solve a puzzle with all the pieces present.
Now, we are checking if they can solve the puzzle when half the pieces are missing, and the remaining pieces are different shapes and sizes.

The paper concludes that if a model can't handle the missing data and the real-world units, it probably isn't ready to be deployed in the real world to help cities manage their air quality. This benchmark forces researchers to build models that are not just "smart" in a lab, but "tough" enough for the real world.

In short: AirQualityBench stops us from pretending the world is perfect. It forces AI to learn how to predict air quality when the sensors are broken, the data is missing, and the units are confusing—just like real life.

Technical Summary: AirQualityBench

Problem Definition

Air-quality forecasting is a complex spatio-temporal prediction task influenced by regional transport, meteorology, emissions, and heterogeneous monitoring coverage. While recent deep spatio-temporal models have achieved strong results on curated regional datasets, their progress relies on benchmarks that often fail to reflect real-world deployment conditions. Existing benchmarks typically suffer from three critical limitations:

Limited Spatial Scope: They are often confined to single cities or administrative regions, severing planetary-scale transport patterns and failing to test model scalability across diverse climatic and topological conditions.
Sanitized Missingness: They frequently employ manual interventions (e.g., linear interpolation, mean imputation) to construct "clean" data tensors. This deviates from the physical reality of Missing Not At Random (MNAR) patterns, creating synthetic bias that masks model fragility when faced with fragmented, real-world observation streams.
Weak Physical Interpretability: They rely on unitless, normalized evaluation metrics, obscuring the toxicological significance of prediction errors and creating a gap between laboratory experiments and environmental regulatory requirements.

The central obstacle identified is not merely a lack of data volume, but the absence of a realistic evaluation protocol that accounts for missing observations, non-uniform spatial coverage, pollutant-specific scales, and computational costs.

Methodology: AirQualityBench

To address these gaps, the authors introduce AirQualityBench, a global multi-pollutant benchmark designed to evaluate forecasting models under realistic, fragmented conditions.

Dataset Construction

Scale and Scope: The benchmark aggregates hourly observations from 3,720 resident monitoring stations globally, spanning the period 2021–2025.
Pollutants: It covers six major pollutants: PM2.5, PM10, O3, NO2, SO2, and CO.
Data Source: Raw data is harvested from the OpenAQ platform. A "resident station" filter is applied, retaining only stations with a valid observation rate above 50% for at least one pollutant over the full five-year span.
Authentic Missingness: Unlike prior works, AirQualityBench preserves provider-native observation masks. It does not impute missing values. Instead, it uses a binary mask tensor ( $M \in \{0, 1\}^{T \times N \times C}$ ) to denote valid observations, forcing models to reason directly from fragmented historical streams.
Physical-Scale Evaluation: The benchmark retains provider-native unit metadata (e.g., $\mu g/m^3$ for particulates, $mg/m^3$ for CO). Predictions are inverse-transformed to these physical scales before error calculation, ensuring metrics are interpretable in real-world concentration units.
Graph Topology: A standardized spherical $k$ -NN graph is constructed using Haversine distance to account for Earth's curvature, providing a reproducible spatial prior for graph-based baselines.

Experimental Protocol

Temporal Split: Data is split chronologically to test generalization across unseen climatic cycles: 2021–2023 for training, 2024 for validation, and 2025 for testing.
Baselines: The authors evaluate a diverse set of representative spatio-temporal models, including:
- Graph-recurrent/convolutional methods (DCRNN, STGCN, GWN, AGCRN).
- Attention-based and Transformer architectures (ASTGCN, STTN, PDFormer, IGSTGNN).
- Decoupled and event-aware models (D2STGNN, MAGE, BiST).
Fairness: All models are evaluated under the same masking protocol. Missing values are handled via zero-filling (after normalization) with explicit binary mask inputs where applicable. Evaluation metrics (MAE, MSE, RMSE, MAPE) are computed only on valid future observations after inverse transformation.

Key Contributions

Global Multi-Pollutant Benchmark: A geographically diverse testbed with 3,720 stations and six pollutants, enabling the evaluation of scalability and cross-region heterogeneity beyond regional limits.
Mask-Aware Physical-Scale Evaluation: A protocol that preserves authentic observation incompleteness and reports errors on physical concentration scales, enabling physically interpretable metrics rather than normalized residuals.
Unified Baselines and Diagnostics: A comprehensive evaluation of state-of-the-art models under identical splits and masking protocols, revealing their behavior under pollutant-specific sparsity and non-uniform station coverage.

Results and Findings

The evaluation of representative models on AirQualityBench yields several insights:

Performance Transfer: Strong performance on sanitized, regional datasets does not reliably transfer to global, fragmented monitoring streams.
Model Sensitivity: Models with explicit spatio-temporal dependency modeling and decoupled spatial–temporal dynamics (e.g., D2STGNN, STTN) generally form the strongest tier. However, flexible adaptive or attention-heavy designs do not always improve performance, suggesting that authentic missingness and sparse non-uniform graphs make learned dependencies difficult to estimate reliably.
Temporal Regularities: Temporal-convolution-based methods remain highly competitive, particularly at longer horizons, indicating that the benchmark preserves strong diurnal and seasonal regularities exploitable by stable temporal backbones.
Accuracy–Efficiency Trade-off: The benchmark highlights a distinct trade-off. Higher accuracy often demands significant computational resources (memory and latency), while lightweight models tend to sacrifice forecasting quality under authentic missingness. For instance, some compact models incur substantial memory usage, while others with larger parameter counts remain efficient.

Significance and Claims

The paper positions AirQualityBench not as a final standard, but as a realistic testbed and a stress test for spatio-temporal models. Its significance lies in:

Bridging the Gap: It connects theoretical research with real-world deployment by exposing models to the exact conditions (missingness, heterogeneity, physical scales) that determine deployability in real sensing networks.
Diagnostic Value: It moves beyond simple leaderboard ranking to provide diagnostic insights into how models handle structured missingness, pollutant-specific sparsity, and the cost of scaling to non-uniform global networks.
Reproducibility: By releasing the processed tensors, authentic masks, station metadata, and evaluation scripts, the authors aim to ensure that reported results are reproducible from the same preprocessing pipeline.

The authors conclude that AirQualityBench serves as a foundation for developing forecasting models that are not only accurate on sanitized tensors but also reliable under fragmented real-world observation streams, facilitating the development of scalable, mask-aware, and physically interpretable air-quality forecasting systems.