💻 computer science

SeasonScapes: Learning Large-scale Re-lightable 3D Landscapes with Seasonal Variation from Sparse Webcams

The paper introduces SeasonScapes, a framework and dataset that constructs large-scale, relightable 3D mountain landscapes with seasonal variations by projecting over 85,000 sparse webcam images onto a mesh and using conditional diffusion models to inpaint occlusions.

Original authors: Timo Kleger, Qi Ma, Deheng Zhang, Luc Van Gool, Danda Pani Paudel

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Timo Kleger, Qi Ma, Deheng Zhang, Luc Van Gool, Danda Pani Paudel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a perfect, 3D video game world of the Swiss mountains. The problem is, you don't have a drone flying everywhere, and you don't have a team of photographers. All you have are 32 old webcams scattered across the landscape, taking pictures at different times of day and different times of the year.

The paper "SeasonScapes" is about taking those sparse, scattered webcam photos and turning them into a massive, living, breathing 3D model that changes with the seasons.

Here is how they did it, broken down into simple steps:

1. The "Rough Draft" (The Skeleton)

First, the team started with a basic, low-resolution map of the mountains (like a digital elevation model) and a satellite photo. They turned this into a 3D mesh—a wireframe skeleton of the terrain.

The Problem: This skeleton was gray and empty. It didn't look like real mountains.
The Fix: They tried to "paint" the skeleton using the webcam photos. But because the webcams are far apart, huge chunks of the mountain were left blank (like trying to paint a giant mural with only a few tiny paintbrushes).

2. The "Magic Paintbrush" (AI Inpainting)

This is the core trick. To fill in the blank spots where no camera could see, they used a special AI tool called ControlNet.

The Analogy: Imagine you are looking at a photo of a mountain, but a cloud is covering the peak. You know what a mountain peak usually looks like. The AI acts like a very smart artist who looks at the visible parts of the mountain, the shape of the terrain (depth), and a text description, then "hallucinates" or guesses what the hidden peak should look like.
The Process: They didn't just guess randomly. They created a "path" for the AI to follow, moving the virtual camera around the mountain. As the camera moved, the AI filled in the missing textures, ensuring the grass, rocks, and snow looked consistent with the real photos.

3. The "Time Machine" (Seasons and Time of Day)

Most 3D models are static; they look the same at noon in July and midnight in December. This project wanted to capture change.

They collected over 85,000 images taken over a full year.
By training their system on these different timestamps, they created a model that knows what the mountain looks like when it's covered in summer green, winter snow, or golden autumn light.
They even trained a special "Gaussian" model (a fancy way of saying a collection of 3D dots) that can be "re-lit." This means you can tell the computer, "Show me this mountain at sunset," and it will adjust the shadows and colors accordingly, even if the original webcam photo was taken at noon.

4. The Results

They tested this on three different areas of the Swiss Alps.

Small areas: The AI did a great job filling in the gaps.
Large areas: It was harder, and sometimes the AI made small mistakes (like painting a valley that looked a bit "dreamy" or slightly wrong), but it was still much better than just using standard satellite maps, which often miss details on steep cliffs because they look from directly above.

In Summary

The paper presents a new toolkit called SeasonScapes. It takes a few scattered webcam photos of a huge mountain range and uses AI to:

Fill in the blanks where the cameras couldn't see.
Paint the textures so the 3D model looks real.
Simulate time, allowing you to see how the landscape changes from dawn to dusk and from summer to winter.

It's essentially turning a collection of static, disconnected webcam snapshots into a continuous, interactive 3D movie of the Swiss Alps.

Technical Summary: SeasonScapes

Problem Statement

Large-scale 3D landscape reconstruction from sparse, unconstrained views remains a significant challenge in computer vision. Existing methods struggle with camera calibration errors, limited scene coverage, and high scene complexity involving dynamic lighting and seasonal appearance variations. While large-scale datasets exist for static urban environments, there is a notable gap in high-quality, temporally rich datasets for natural, unbounded mountain landscapes that capture both seasonal shifts and diurnal lighting cycles. Furthermore, traditional mesh-based systems (e.g., Google Earth) rely on static geometry and blended satellite textures, failing to model view-dependent appearance or diverse illumination conditions with photometric consistency.

Methodology

The authors propose SeasonScapes, a framework and dataset designed to reconstruct large-scale, relightable 3D landscapes from sparse webcam imagery. The pipeline integrates multimodal data (webcams, satellite imagery, and elevation models) through the following stages:

1. Data Preprocessing and Initialization

Base Geometry: The process begins with a low-resolution Digital Elevation Model (IDEM) and a satellite color image ( $I_S$ ) of the target region (e.g., Bernese Highlands). A 3D point cloud is generated by mapping pixel heights from the IDEM and coloring them with $I_S$ . A 3D mesh ( $M$ ) is constructed using Poisson surface reconstruction.
UV Unwrapping: The mesh surface is unwrapped using the xatlas library to create a UV texture map, separating texture storage from geometry.
Webcam Alignment: Panoramic webcam feeds (360° cylindrical) are split into 4–6 perspective sub-images. These are color-graded for consistency and aligned with the 3D mesh.
Camera Optimization: Initial camera poses are derived from GPS coordinates but refined via a learning-based approach. The authors manually establish 2D–3D correspondences and optimize intrinsic (focal length) and extrinsic (rotation) parameters by minimizing the L1 pixel loss between projected 3D points and target 2D image points.

2. 3D Mesh Texturing and Inpainting

To address occlusions and sparse coverage, the framework employs a two-stage texturing process:

Initial Painting: Preprocessed webcam images are projected onto the UV map of the mesh. A progressive superposition strategy is used, where new textures are blended with existing ones based on a UV mask, ensuring that previously painted regions are preserved.
Iterative Inpainting: For regions not covered by webcam views, the system uses a depth-aware diffusion model (ControlNet) to inpaint missing areas.
- Trajectory: The system defines a novel view path with specific poses.
- Conditioning: For each step, an RGBD image is rendered. An uncolored mask is generated using HSV color space filtering to identify missing regions.
- Diffusion: A ControlNet pipeline conditions the inpainting on the depth map, an IP-Adapter (for appearance guidance from nearby webcams), and a text prompt.
- Projection: The generated inpainted image is projected back onto the local UV texture map and superimposed with the previous texture.

3. Relightable 3D Representation

The final stage involves training a Wild Gaussian model (an extension of 3D Gaussian Splatting) using the completed, temporally diverse dataset.

Dynamic Embeddings: The model incorporates time embeddings to capture seasonal and diurnal variations.
Appearance MLP: A Multi-Layer Perceptron (MLP) transforms the base color attributes of each Gaussian based on the viewing ray direction and time, enabling the reconstruction of dynamic scene properties and relighting capabilities.

Key Contributions

SeasonScapes Dataset: A high-quality, large-scale multimodal dataset covering 3,000 km² of alpine terrain. It includes over 85,000 webcam images from 32 locations across 13 timestamps, capturing full seasonal cycles and diurnal changes. It provides precise timestamp annotations and calibrated 2D–3D correspondences between ground-level imagery and georeferenced satellite data.
SeasonScapes Framework: A novel pipeline for creating relightable 3D landscapes. It uniquely handles large-scale, unbounded scenes by introducing linear or cubic spline inpainting trajectories between GPS coordinates and performing occlusion-aware painting via UV indexing and depth rendering, rather than relying solely on normal-based filtering.
Dynamic Relightable Modeling: The integration of dynamic embeddings into relightable Gaussian training, enabling effective modeling of temporal changes (seasons and time of day) in large-scale outdoor scenes.

Results and Evaluation

The authors evaluated the pipeline across three diverse test scenes: Grindelwald (small scale), Jungfrau (medium scale), and Bernese Highlands (large scale).

Quantitative Metrics: Performance was measured using LPIPS, PSNR, and SSIM. The Grindelwald scene achieved an LPIPS of 0.159, while the larger Jungfrau and Bernese scenes showed higher LPIPS values (0.191 and 0.216, respectively). The authors attribute the performance degradation in larger scenes to the increased proportion of unseen regions requiring synthesis.
Ablation Studies: Removing the initial painting stage resulted in a significant performance drop (up to 33.1% in LPIPS for Grindelwald), highlighting the necessity of the two-stage approach. Removing depth control led to "mountain hallucinations" in distant regions, while the IP-Adapter was shown to primarily modulate color tones.
Qualitative Comparison: Compared to Google Earth, the SeasonScapes approach demonstrated superior coloring on steep cliff faces by better capturing near-orthogonal views, avoiding the warping and detail loss common in top-down satellite perspectives. The method also showed greater adaptability to seasonal variations.

Significance and Limitations

The paper positions SeasonScapes as a step forward in creating digital twins of natural environments that are not only geometrically accurate but also photometrically consistent across time. By combining sparse webcam data with diffusion priors, the method enables the modeling of long-term environmental transitions and short-term illumination changes, advancing beyond static satellite-only reconstructions.

The authors acknowledge current limitations:

The manual 2D–3D correspondence process is time-consuming.
ControlNet struggles with inpainting very small missing regions.
The relightable Gaussian outputs occasionally exhibit blurry artifacts, likely due to calibration errors and the expressiveness constraints of the Wild Gaussian framework.

The work provides a foundation for applications in environmental monitoring, weather modeling, and virtual tourism, offering a more dynamic and realistic representation of natural landscapes than existing static datasets.