💻 computer science

TerraDiT: Point-Conditioned Diffusion Transformer for Satellite Image Synthesis

TerraDiT is a novel point-conditioned diffusion transformer that enables flexible and semantically rich satellite image synthesis by replacing time-consuming pixel-level maps with an adaptive local attention mechanism driven by spatial points and their associated textual descriptions.

Original authors: Srikumar Sastry, Dan Cher, Brian Wei, Aayush Dhakal, Subash Khanal, Dev Gupta, Nathan Jacobs

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Srikumar Sastry, Dan Cher, Brian Wei, Aayush Dhakal, Subash Khanal, Dev Gupta, Nathan Jacobs

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a realistic map of a city from space, but instead of hiring an army of cartographers to draw every single building, road, and tree by hand, you just want to drop a few "pins" on a blank canvas and say, "Put a school here" or "Put a park there."

That is the core idea behind TerraDiT, a new AI system described in this paper. Here is a breakdown of how it works, using simple analogies.

The Problem: The "Pixel-Perfect" Bottleneck

Previously, if you wanted an AI to generate a satellite image with specific details (like a hospital in the corner or a river in the middle), you had to give it a dense, pixel-by-pixel map.

The Analogy: Imagine trying to tell a painter exactly where to put every single brushstroke on a canvas by handing them a complex, pre-drawn blueprint. It's accurate, but it takes forever to make the blueprint, and the painter is forced to follow it rigidly, leaving no room for creativity.
The Limitation: These blueprints (pixel-level maps) are expensive to create and often limit the AI to only drawing exactly what is on the map, with no flexibility.

The Solution: The "Pin and Prompt" Approach

TerraDiT changes the game by using points instead of blueprints.

The Analogy: Instead of a full blueprint, you now just drop a few pins on a blank map. Next to each pin, you write a short note (a text prompt).
- Pin 1: "School"
- Pin 2: "River"
- Pin 3: "Forest"
The Magic: The AI looks at where you dropped the pins and reads your notes. It then fills in the rest of the picture itself, deciding exactly how big the school should be or how the river curves, as long as it stays near your pin. This is much faster to set up and allows for more natural, varied results.

How It Works: The "Smart Spotlight"

The paper introduces a special tool called Adaptive Local Attention (ALA).

The Analogy: Imagine the AI is a spotlight operator in a theater. Usually, a spotlight shines on everything equally. But with ALA, the spotlight knows exactly where to focus based on your pins.
- If you put a pin for a "small shed," the spotlight knows to stay tight and small around that pin.
- If you put a pin for a "large park," the spotlight expands to cover a wider area.
The Result: The AI doesn't just guess; it uses a "spatial prior" (a learned sense of scale) to understand that a "house" needs a small area, while a "city" needs a large one. This ensures the generated image looks realistic and structured.

The "Training" Process

The researchers didn't just build this from scratch; they trained the AI in three steps, like a student learning a subject:

Level 1 (Unconditional): The AI learns what satellite images generally look like (clouds, oceans, cities) without any instructions.
Level 2 (Text): The AI learns to listen to text descriptions (e.g., "a city with red roofs").
Level 3 (Points): The AI learns to listen to your pins and text notes simultaneously.

They also taught the AI to "look" at the world the way satellite experts do by aligning its internal brain with a powerful satellite-specific vision model (DINOv3). This helps the AI understand the difference between a natural image (like a photo of a dog) and a satellite image (a photo of a dog from space).

The Results: Better and Faster

The paper tested TerraDiT against other top AI models.

Quality: It produced images that looked more realistic and matched the text instructions better than previous models.
Flexibility: Unlike models that force you to draw a perfect map, TerraDiT could generate many different, valid versions of a scene based on just a few pins.
Efficiency: It generated these images faster (lower latency) than many competitors.

Summary

TerraDiT is like giving a satellite artist a set of sticky notes and a pen instead of a rigid, pre-drawn blueprint. It allows users to guide the creation of complex space images with simple dots and words, making the process easier, faster, and more flexible, while still producing highly accurate and realistic results.

Technical Summary: TerraDiT

Problem Statement

Existing controlled satellite image generative models face a significant bottleneck: they rely on dense, pixel-level spatial maps (such as segmentation masks or OpenStreetMap layouts) to guide generation. While effective for control, acquiring these dense annotations is time-consuming, expensive, and often semantically limited to fixed concept sets. Conversely, standard text-to-image models lack the fine-grained spatial control necessary for specialized remote sensing applications like urban design and environmental monitoring. There is a need for a generative framework that offers flexible, annotation-friendly spatial control without the computational and data acquisition overhead of dense pixel-level supervision.

Methodology

The authors introduce TerraDiT, a Diffusion Transformer (DiT) designed for text-to-satellite image generation with point-based control. The approach integrates several domain-specific design choices:

1. Architecture and Training Strategy

TerraDiT is built upon the DiT architecture (specifically the SiT backbone) and utilizes Flow Matching for generative modeling. The training follows a three-stage procedure to minimize computational costs while ensuring stability:

Stage 1 (Unconditional): The model learns the overall distribution of high-resolution satellite imagery.
Stage 2 (Text Conditioning): Text-to-satellite generation is introduced using LongCLIP as the text encoder. The model incorporates dense tokens and a single embedding representation, with the latter added to the flow timestep embedding.
Stage 3 (Point Conditioning): The model is trained for point-controlled generation. Each point query consists of a 2D coordinate and a free-form textual prompt. Coordinates are encoded via 2D sine-cosine encoding followed by an MLP, and text is encoded via LongCLIP.

2. Adaptive Local Attention (ALA)

A core innovation is the Adaptive Local Attention (ALA) module, designed to regularize cross-attention between latent image tokens and point prompts. ALA comprises two components:

MetaRBF Module: A MultiHead Attention block followed by a feed-forward layer that predicts spatial extent parameters ( $\sigma_x, \sigma_y$ ) for each point query. These parameters define a 2D Radial Basis Function (RBF) centered at the point's location.
Local Attention Block: The predicted RBF kernels modulate the cross-attention scores, allowing the model to attend locally around each point query while respecting the predicted spatial extent of the concept. This prevents the model from imposing strict pose constraints, enabling diverse semantic variations.

3. Representation Alignment and Geolocation

Satellite Representation Alignment: Inspired by REPA, the authors align the hidden representations of the DiT with a pre-trained satellite-only encoder (DINOv3) during training. A linear projection maps DiT block representations to the DINOv3 space to improve domain-specific feature learning.
Geolocation Conditioning: The model leverages RANGE embeddings for geolocation awareness, integrated via adaptive layer normalization.

4. Dataset

The model is trained on an augmented version of the Git-10M dataset (2 million images at zoom level 17). To enable point-based conditioning, the authors co-register OpenStreetMap (OSM) vector data with satellite images, randomly sampling 10–50 points per image along with their associated OSM tags and relative coordinates.

Key Contributions

TerraDiT-XL/2- $\alpha$ : A state-of-the-art text-to-satellite image generative model that establishes a strong foundation for controlled remote sensing synthesis.
TerraDiT-XL/2- $\Sigma$ : A point-conditioned DiT that enables fine-grained spatial and semantic control using minimal point-based inputs, offering a flexible alternative to dense pixel-level supervision.
Adaptive Local Attention (ALA): A novel module that effectively regularizes cross-attention for point-based conditions, resulting in precise and spatially coherent generation without strict shape constraints.
Systematic Design Analysis: A comprehensive study of domain-specific factors for satellite image DiTs, including conditioning strategies, geolocation encoders, and representation alignment with satellite-specific encoders.

Experimental Results

The authors evaluated TerraDiT on four datasets: two in-domain held-out splits (Git-Rand-15k, Git-Spatial-15k) and two out-of-domain datasets (RSICD, FMoW).

Performance: TerraDiT variants outperformed state-of-the-art remote sensing generative models (e.g., GeoSynth, DiffusionSat, Text2Earth) and general-purpose models (SD3, PixArt) across all metrics (FID, LPIPS, SSIM, CLIP). Specifically, the point-conditioned variant (TerraDiT-XL/2- $\Sigma$ ) achieved the lowest FID (12.01 on Git-Rand-15k) and highest SSIM, surpassing the text-only variant.
Qualitative Quality: Visualizations demonstrate that TerraDiT generates high-fidelity imagery that accurately reflects point-query semantics. Unlike map-based controls, point queries allow for diverse structural variations while maintaining semantic consistency.
Training-Free Inpainting: In a zero-shot inpainting task on urban regions, TerraDiT-XL/2- $\Sigma$ achieved superior FID (16.81) and SSIM (0.7000) compared to geospatial baselines, demonstrating effective local semantic alignment with minimal annotation.
Efficiency: The models achieved the lowest latency for single-image generation compared to other remote sensing generative models.

Significance and Claims

The paper claims that TerraDiT represents a significant advancement in remote sensing generative modeling by bridging the gap between the flexibility of text-to-image generation and the spatial control required for satellite imagery. By replacing dense, expensive pixel-level maps with semantically rich point queries, the authors argue that TerraDiT provides a more annotation-friendly and computationally efficient framework. The work demonstrates that Diffusion Transformers, when combined with domain-specific representation alignment and novel attention mechanisms, can effectively capture the complex distributions of satellite imagery, outperforming traditional UNet-based approaches in both perceptual realism and structural consistency.