🤖 AI

CarbonCLIP: Enhance Carbon Prediction from Satellite Imagery via Integrated Street-View Semantics and Temporal Context Training

CarbonCLIP is a multimodal distillation framework that enhances satellite-based urban carbon emission prediction by leveraging contrastive learning to integrate semantic insights from street-view images and temporal context from monthly data during pretraining, enabling accurate, scalable inference using only satellite imagery.

Original authors: Zeru Yang, Fang-Ying Gong, Steve H. L. Yim, Chau Yuen

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zeru Yang, Fang-Ying Gong, Steve H. L. Yim, Chau Yuen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how much pollution a city is producing just by looking at a photo taken from a satellite high in the sky.

The Problem: The "Bird's-Eye View" Blind Spot
Think of a satellite image like a bird flying over a city. You can see the shapes of buildings, the roads, and the parks. But from that height, you can't see the details that actually drive carbon emissions. You can't tell if a building is a busy factory or a quiet home just by its roof. You can't see the traffic jams, the greenery on the sidewalks, or the types of shops on the street corners.

Current methods are like trying to guess a person's job just by looking at the top of their head. It's a good start, but it misses the most important clues.

The Solution: CarbonCLIP (The "Time-Traveling Detective")
The authors created a new AI system called CarbonCLIP. Think of it as a detective who learns to solve a case by studying two different types of evidence before being asked to solve it with just one.

Here is how the training works, using a simple analogy:

The "Ground Truth" Tour (Street Views):
Imagine the AI is sent on a tour of the city on the ground. It looks at street-level photos (like Google Street View) and uses a super-smart robot brain (a Large Multimodal Model) to write a detailed report.
- Instead of just seeing a building, the robot writes: "This is a dense residential area with many balconies, a busy bus stop nearby, and lots of trees."
- This gives the AI a rich understanding of what is happening on the ground.
The "Time Machine" (Temporal Context):
The AI also learns that cities change with the seasons. In winter, people heat their homes; in summer, they use air conditioning. The AI is taught to recognize that "January" is different from "July," even if the buildings look the same. It learns to associate specific months with specific energy habits.
The "Magic Link" (Contrastive Learning):
Now, the AI plays a matching game. It looks at the satellite photo (the bird's-eye view) and tries to match it with the detailed ground report and the specific month. It learns: "Ah, this specific satellite shape of a building corresponds to 'busy commercial district' and 'winter heating season'."

It does this millions of times, connecting the dots between the high-altitude view and the ground-level reality.

The Final Trick: The "Satellite-Only" Superpower
Once the AI has finished this training, it undergoes a transformation.

During Training: It uses satellite photos, street-view reports, and calendar dates.
During the Real Job (Inference): It is told to forget the street views and the calendar. It must now predict carbon emissions using only the satellite photo.

Because it learned so deeply during training, it has "internalized" the ground-level details. When it looks at a satellite photo now, it doesn't just see a gray roof; it "remembers" that this roof likely belongs to a busy commercial area and that it's currently winter, so emissions should be higher. It has distilled the knowledge of the ground into the satellite image itself.

The Results
The researchers tested this in two very different cities: Beijing (which has four distinct seasons) and Singapore (which is tropical and rainy).

The Old Way: Previous AI models were like guessing the weather by looking at a cloud. They were okay, but often wrong.
CarbonCLIP: This new model was much more accurate in both cities. It proved that by "teaching" the satellite AI with ground-level stories and time-based context, it can make much better predictions without needing to see the ground again.

In Summary
CarbonCLIP is like teaching a student to read a map by first walking the streets and talking to locals. Once the student understands the neighborhood deeply, they can look at a map from a distance and accurately predict what's happening there, even without being on the ground. This allows cities to monitor their carbon footprint more accurately using only satellite data, which is available everywhere.

Technical Summary: CarbonCLIP

Problem Statement

Accurately estimating urban carbon emissions is critical for sustainable planning, yet existing approaches face significant hurdles. Traditional bottom-up inventory methods suffer from coarse spatiotemporal resolution and reporting delays. Conversely, data-driven models relying on remote sensing often fail to capture fine-grained drivers of emissions because optical satellite imagery lacks human-centric information (e.g., building facades, street-level greenery, signage) and vertical context. Furthermore, many current methods rely on heterogeneous auxiliary data (e.g., Points of Interest, socio-demographics, or static street-view imagery) that are not consistently available across different cities, limiting scalability. A key challenge is bridging the gap between top-down satellite views and ground-level human activities while accounting for the temporal volatility of emissions (e.g., seasonal energy consumption) without requiring region-specific data during inference.

Methodology: CarbonCLIP

The authors propose CarbonCLIP, a task-oriented multimodal distillation framework designed to enhance satellite-based carbon emission prediction. The core innovation is an asymmetric learning paradigm: the model leverages multimodal data (satellite, street-view, and temporal context) during pretraining but relies solely on satellite imagery during inference.

1. Framework Architecture

The framework consists of two main phases:

Phase 1: Dual-Branch Contrastive Pretraining: The model learns to align satellite imagery with ground-level semantics and temporal context.
- Spatial Branch (Street-View Semantics): Instead of using raw street-view images, the system employs Large Multimodal Models (LMMs), specifically Qwen2.5-VL, to automatically generate fine-grained textual descriptions of street-view panoramas. These descriptions capture urban structure, infrastructure, and activity levels. A text encoder (CLIP-based) converts these descriptions into semantic embeddings.
- Temporal Branch (Month Context): A dedicated Month Encoder maps observation months to learnable temporal embeddings. This encoder uses a multi-frequency sinusoidal encoding strategy combined with learnable coarse calendar-group embeddings (e.g., rainy/dry seasons) to capture cyclic temporal variations.
- Contrastive Learning: Two symmetric contrastive objectives are employed:
  1. Image-Text Alignment: Aligns satellite image embeddings with the corresponding street-view text embeddings.
  2. Temporal Alignment: Aligns satellite image features with month embeddings using a soft contrastive loss that respects the cyclic nature of months (e.g., December and January are close).
Phase 2: Satellite-Only Inference: After pretraining, the vision encoder and projection layers are frozen. A lightweight Multi-Layer Perceptron (MLP) is trained using only the satellite image embeddings (spatial + temporal) to predict monthly carbon emissions. No street-view or auxiliary data is required at this stage.

2. Data Processing

Satellite Data: High-resolution (3m) Planet imagery for Beijing and Singapore (2022), aggregated into $1 \text{ km}^2$ tiles.
Street-View Data: Panoramas from Google Maps (Singapore) and Baidu Maps (Beijing), sampled at 200m intervals along road networks.
Labels: Monthly carbon emissions from the ODIAC dataset ( $1 \text{ km}^2$ resolution).

Key Contributions

Multimodal Distillation for Satellite-Only Inference: The paper introduces a framework that distills fine-grained ground-level semantics and temporal priors into a unified satellite representation. This allows the model to retain multimodal awareness during inference while operating solely on satellite imagery, addressing the scalability issue of region-specific auxiliary data.
LMM-Generated Semantic Abstraction: Instead of direct pixel-level alignment with street-view images, the authors utilize LMMs to generate structured textual descriptions. This approach abstracts human-centric details (e.g., building functions, traffic) into semantic anchors, reducing noise from view-specific variations (e.g., occlusion, viewing angle) and enabling robust knowledge transfer.
Cyclic Temporal Alignment: The introduction of a dedicated month encoder with cyclic contrastive alignment allows the model to encode temporal priors associated with monthly emission variations, a factor often ignored in static visual models.
Dataset Construction: The authors constructed a dataset linking satellite tiles to LMM-generated street-view descriptions and plan to make it publicly available to facilitate future research.

Experimental Results

The method was validated on two distinct cities: Beijing (temperate, four seasons) and Singapore (tropical, rainy/dry seasons).

Performance: CarbonCLIP outperformed all baselines (ResNet, ViT, and UrbanCLIP) in both cities across all metrics ( $R^2$ $R^{2}$ , RMSE, MAE).
- In Beijing, CarbonCLIP achieved an annual $R^2$ of 0.728, significantly surpassing UrbanCLIP (0.642) and ViT (0.559).
- In Singapore, CarbonCLIP achieved an annual $R^2$ of 0.704, outperforming UrbanCLIP (0.576) and ResNet (0.569).
Temporal Stability: The model demonstrated consistent performance across all months, whereas baselines showed noticeable degradation during specific seasonal periods.
Ablation Studies:
- Removing the spatial branch (street-view text) caused significant performance degradation, confirming the necessity of fine-grained spatial semantics.
- Removing the temporal branch reduced performance, particularly in months with distinct energy-use patterns.
- Replacing text-based supervision with direct street-view image alignment resulted in lower performance, suggesting that LMM-generated text provides a more robust and abstract semantic target than raw visual embeddings.

Significance and Claims

The paper claims that CarbonCLIP offers a robust and scalable solution for satellite-based urban carbon modeling. Its primary significance lies in decoupling the need for ground-level data during deployment from the training process. By distilling multimodal knowledge into a satellite-centered representation, the framework enables accurate carbon estimation in cities where ground-level auxiliary data (like street views or detailed socio-economic stats) may be unavailable or inconsistent.

The authors emphasize that their approach effectively transfers "human-centric" semantics and temporal context into satellite representations, overcoming the "iso-spectral" limitations of top-down views where functionally distinct urban zones appear visually similar. The results validate that integrating LMM-generated semantics and cyclic temporal context significantly improves prediction accuracy compared to purely visual or static multi-source approaches.

The paper remains modest regarding limitations, acknowledging that spatial correlations between neighboring tiles are not explicitly modeled, temporal representations are currently coarse (monthly), and the pretraining phase still depends on the availability of street-view data and the quality of LMM generation.