🤖 machine learning

LangPrecip: Language-Aware Multimodal Precipitation Nowcasting

The paper introduces LangPrecip, a language-aware multimodal framework that leverages meteorological text as semantic motion constraints within a Rectified Flow paradigm to improve short-term precipitation nowcasting, supported by a new 160k-scale dataset and demonstrating significant performance gains over state-of-the-art methods.

Original authors: Xudong Ling, Chaorong Li, Tianxi Huang, Qian Dong, Guiduo Duan

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xudong Ling, Chaorong Li, Tianxi Huang, Qian Dong, Guiduo Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict where a flock of birds will fly in the next hour. If you only look at a blurry, 4-second video of them moving, it's a guessing game. They might turn left, right, or split into two groups. Without more clues, your prediction is likely to be a fuzzy, washed-out mess because the video alone doesn't tell you why they are moving that way.

This is exactly the problem scientists face with short-term rain forecasting (called "nowcasting"). Current computer models look at radar images of rain clouds and try to guess where the rain will go next. But because they only have a few seconds of history, they often get confused, producing blurry predictions that miss dangerous storms or invent rain that isn't there.

The paper introduces LangPrecip, a new system that solves this by giving the computer a "narrator."

The Core Idea: Adding a "Sportscaster" to the Radar

Instead of just feeding the computer raw radar images, LangPrecip adds a second input: a natural language description of what the rain is doing.

Think of it like this:

Old Way (Vision-Only): You show a computer a video of a car driving and ask, "Where will it be in 10 minutes?" The computer guesses, but it might hallucinate the car driving off a cliff because the video didn't show a road.
LangPrecip Way: You show the video and tell the computer, "The car is driving steadily north on a highway." Now, the computer has a rule to follow. It knows the car won't suddenly fly or stop in mid-air.

In this paper, the "narrator" is an AI that watches the first few seconds of the radar and writes a sentence like: "The rain echoes are moving steadily leftward and slightly downward, holding their shape as they drift." The forecasting model then uses this sentence as a strict rulebook to guide its prediction, ensuring the rain moves in a physically realistic way.

The Two Big Innovations

1. The "Motion Script" (Language Guidance)
The researchers created a massive new dataset called LangPrecip-160K. It contains 160,000 pairs of radar videos and their corresponding "motion scripts."

Why it works: In the real world, rain doesn't just appear randomly; it follows wind and physics. By forcing the AI to read a description of the wind and movement, the model stops guessing wildly. It's like giving a student a math problem with the formula written down, rather than just asking them to guess the answer.
The Result: The paper shows this helps the model predict extreme rain (heavy storms) much better. Without the text, the model often "smooths out" the storm, making it look weak. With the text, it keeps the storm sharp and intense, which is crucial for warning people about floods.

2. The "Detail Restorer" (Wavelet Decoder)
Even with a good script, turning a compressed computer code back into a clear picture is hard. Standard AI models often produce blurry images, like a photo taken with a shaky camera.

The Solution: The team built a special "decoder" called the Wavelet Consistency Unfolding Decoder.
The Analogy: Imagine you have a high-resolution painting, but you squint your eyes and see only a blurry blob. A normal AI tries to guess the details, often getting them wrong. This new decoder is like a master restorer who knows the math of how the painting was blurred. It uses a technique called "wavelets" (a way of breaking images into patterns of sharp edges and smooth areas) to iteratively sharpen the image, forcing it to match the original radar data perfectly. It ensures the edges of the rain clouds stay crisp, not fuzzy.

What the Results Show

The team tested LangPrecip on real radar data from Sweden and the US (MRMS).

The Score: In predicting heavy rain (the kind that causes floods), LangPrecip improved accuracy by 60% compared to the best existing methods that only look at images.
The Visuals: When they looked at the predictions, the old models looked like watercolor paintings that were bleeding together. LangPrecip's predictions looked like sharp, clear photographs where the storm cells stayed distinct and moved in the right direction.

Summary

LangPrecip is a rain-forecasting system that doesn't just "see" the rain; it "reads" a description of how the rain is moving. By combining a visual radar feed with a text-based motion guide, and using a special math tool to keep the image sharp, it creates much more accurate and reliable short-term weather forecasts, especially for dangerous storms.

Note: The paper focuses strictly on improving the accuracy of radar-based rain prediction over short timeframes (up to 80 minutes). It does not claim to predict long-term climate change, diagnose medical conditions, or apply to other types of weather like tornadoes specifically, though the principles are related.

Technical Summary: LangPrecip: Language-Aware Multimodal Precipitation Nowcasting

1. Problem Statement

Short-term precipitation nowcasting is fundamentally an under-constrained problem. Limited historical observation windows (typically 4–8 frames) are insufficient to uniquely determine physically consistent motion trajectories, particularly for extreme events where identical radar sequences can lead to multiple plausible futures (e.g., translation, splitting, intensification, or dissipation).

Existing generative methods rely solely on visual features. While deterministic models (e.g., ConvLSTM, Transformers) often collapse multiple futures into over-smoothed forecasts that fail to capture extreme precipitation, recent stochastic generative approaches (e.g., GANs, Diffusion models) introduce diversity but lack explicit constraints on precipitation motion semantics. This results in ambiguous dynamics, blurred details, and unstable predictions. Furthermore, latent-space modeling introduces a severely ill-posed inverse problem during reconstruction, often yielding temporally inconsistent and spatially blurred outputs due to aggressive spatiotemporal compression.

2. Methodology

The authors propose LangPrecip, a language-guided multimodal precipitation nowcasting framework that reformulates forecasting as a semantically constrained trajectory generation problem under the Rectified Flow paradigm.

2.1. LangPrecip-160K Dataset

The authors introduce LangPrecip-160K, the first large-scale radar-text paired dataset containing 160,000 annotated sequences derived from Swedish radar data and the US MRMS system.

Annotation: Motion descriptions are generated using a Vision-Language Model (Qwen-VL) with meteorology-specific prompts and verified by trained meteorologists.
Content: These natural-language descriptions encode high-level semantic priors (advection direction, growth/decay patterns, structural evolution) without requiring precise numerical measurements, serving as explicit constraints to regularize under-determined radar dynamics.

2.2. Language-Aware Motion Prediction

The core generation model operates in a latent space encoded by a 2D VAE (downsampling factor of 8).

Architecture: A Rectified Flow model (based on a Diffusion Transformer backbone) predicts future latent states conditioned on historical radar frames and motion descriptions.
Conditioning: The model utilizes T5-XXL to encode motion descriptions. Crucially, the text is generated exclusively from the first four observed frames to ensure causal consistency, while the model is supervised on full 20-frame trajectories to learn long-range dynamics.
Mechanism: The language condition acts as a high-level prior, biasing the generative model toward physically plausible motion regimes while preserving uncertainty in fine-scale evolution (intensity fluctuations) learned from visual observations.

2.3. Wavelet Consistency Unfolding Decoder (WCUD)

To address the blurring and temporal inconsistencies caused by latent-to-pixel reconstruction, the authors design a novel decoder that integrates model-based optimization with deep feature learning.

Structure: The decoder consists of an Upsampling Branch (using Temporal Shift Modules for cross-frame alignment) and a Data Fidelity and Wavelet Prior Branch.
Optimization: The reconstruction is formulated as an optimization problem minimizing data fidelity error and a wavelet sparsity prior ( $\ell_1$ norm of wavelet coefficients).
Unfolding: Instead of iterative solving at inference, the optimization is "unfolded" into $T$ $T$ learnable blocks. Each block performs:
1. Data Consistency (DC): A gradient descent step (back-projection) to enforce fidelity to the observed low-resolution radar.
2. Wavelet Prior (WP): A proximal operator applying soft thresholding in the wavelet domain to suppress noise while retaining sharp precipitation boundaries.

3. Key Contributions

LangPrecip Framework: The first language-guided precipitation nowcasting framework that uses natural-language motion descriptions as explicit semantic constraints to reduce motion ambiguity in under-constrained short-window forecasts.
LangPrecip-160K Dataset: A large-scale, multimodal dataset pairing radar sequences with meteorologically validated motion descriptions, enabling explicit disambiguation of plausible futures beyond pixel-level observations.
Wavelet Consistency Unfolding Decoder (WCUD): A novel decoder architecture that enforces physical data fidelity and multi-scale sparsity through iterative optimization unfolding, significantly improving reconstruction sharpness and temporal coherence compared to standard VAE decoders.

4. Experimental Results

Experiments were conducted on Swedish and MRMS benchmarks, comparing LangPrecip against state-of-the-art vision-only methods (including optical-flow, GAN, diffusion, and Transformer-based models).

Extreme Event Prediction: LangPrecip achieves substantial improvements in heavy-rainfall detection. On the Swedish dataset, it yields a 60.9% relative gain in Critical Success Index (CSI) at the 6.3 mm/h threshold (0.046 to 0.074) at an 80-minute lead time. On the MRMS dataset, it achieves a 19.2% relative gain (0.026 to 0.031).
Spatial and Temporal Consistency: The model outperforms baselines in Fractions Skill Score (FSS) and Structural Similarity Index (SSIM), demonstrating better preservation of coherent system structures and propagation.
Motion Control: Ablation studies confirm that language guidance effectively regulates precipitation dynamics. When conditioned solely on text (without radar history), the model generates motion patterns that align semantically with the descriptions (e.g., specific propagation directions), validating the physical interpretability of the language priors.
Ablation: Removing language conditioning or the WCUD decoder leads to significant performance drops, particularly in extreme event detection (CSI) and reconstruction fidelity (PSNR/SSIM).

5. Significance and Claims

The paper claims that LangPrecip addresses the inherent ambiguity of precipitation nowcasting by leveraging linguistic semantics to constrain physical evolution. By reformulating the task as a semantically constrained trajectory generation problem, the framework:

Resolves Ambiguity: Effectively distinguishes between multiple plausible futures that vision-only models cannot disambiguate.
Enhances Extreme Event Detection: Significantly improves the detection of rare, high-intensity precipitation events, which are critical for disaster prevention but often missed by models regressing to climatological means.
Improves Physical Fidelity: The WCUD decoder mitigates the blurring artifacts typical of latent generative models, producing sharper and more physically consistent forecasts.

The authors note limitations, including the reliance on general-purpose vision-language models for annotation (which may miss domain-specific nuances) and the need for further validation across diverse climate regimes (e.g., tropical cyclones). Future work is suggested to incorporate structured meteorological signals (like NWP wind fields) alongside language and to explore hierarchically motion-aware architectures.