⚡ electrical engineering

Physics-Guided Multimodal Learning for Flexible Tropical Cyclone Intensity Forecasting

This paper proposes TCFNet, a physics-guided multimodal learning framework that integrates intensity variables to guide visual representation learning and handle missing data, thereby significantly improving the accuracy and adaptability of tropical cyclone intensity forecasting compared to existing methods.

Original authors: Kuai Dai, Hui Su, Chengxing Zhai, Xiaoyang Zhang, Zesheng Shi, Chenghao Yang

Published 2026-11-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kuai Dai, Hui Su, Chengxing Zhai, Xiaoyang Zhang, Zesheng Shi, Chenghao Yang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Tropical cyclones are among the most powerful and destructive forces on Earth, capable of unleashing extreme winds and torrential rain that can cripple transportation, cut off electricity, and endanger lives. For meteorologists, the most critical task is not just tracking where a storm will go, but predicting how strong it will become. This intensity is usually measured by the speed of the wind and the pressure at the storm's center; generally, the lower the pressure, the stronger the wind. However, forecasting these changes is notoriously difficult. The atmosphere is a complex, chaotic system where storms can suddenly intensify or weaken for reasons that are not always obvious. While computers have become better at simulating weather, they often struggle to capture the rapid, violent shifts in a cyclone's power, leaving communities with less warning time than they need.

In recent years, scientists have turned to artificial intelligence to help solve this puzzle, hoping that machines can spot patterns in data that humans might miss. A new study from researchers at the Hong Kong University of Science and Technology and other institutions introduces a fresh approach called TCFNet. Instead of relying on just one type of information, this system learns by looking at two different sources at once: the visual images of the storm taken from satellites and the numerical records of the storm's past behavior. The key innovation is that the system does not just mash these two data streams together; it uses the physical numbers to guide how the computer "sees" the images. This allows the model to understand that a specific cloud pattern might mean something different depending on the storm's current physical state, leading to more accurate predictions of how the storm will evolve.

The challenge the researchers faced was that satellite images and numerical data often tell slightly different stories. A picture of a swirling cloud might look very similar to a picture taken hours later, even if the storm has suddenly become much stronger or much weaker. Traditional computer models often treat these images as static pictures, missing the subtle clues that indicate a change in intensity. To fix this, the team designed a system that treats the storm's physical variables, like wind speed and pressure, as a guide for interpreting the visual data. Imagine a translator who not only hears the words being spoken but also understands the tone and context to get the true meaning; in this case, the physical data provides the context that helps the computer correctly interpret the visual patterns of the storm.

The researchers built their model to handle the messy reality of real-world data. In the field, satellite images and weather reports do not always arrive at the exact same time, and sometimes data is missing entirely due to technical glitches or delays. Older models often fail when the input is irregular or incomplete. TCFNet, however, is designed to be flexible. It can process observations that arrive at uneven intervals and can even make predictions for any specific time in the future, rather than being locked into a rigid schedule. This flexibility is crucial for practical use, as it allows forecasters to get updates even when the data stream is imperfect.

To test their idea, the researchers trained and tested the system on data from two major cyclone regions: the Western North Pacific and the South Pacific. They compared their new model against a wide range of existing methods, including standard weather forecasting schemes, simple statistical models, and other advanced artificial intelligence tools. The results showed that TCFNet consistently outperformed the others. It was particularly good at predicting the exact values of wind speed and pressure, but its greatest strength was in catching sudden changes. When a storm began to rapidly intensify or weaken, TCFNet was able to detect these shifts more quickly and accurately than the previous best methods.

The study also looked closely at how well the system could identify "rapid intensification," a dangerous phenomenon where a storm's wind speeds jump by a significant amount in a very short time. In these tests, the new model correctly identified these dangerous spikes more often than any other method tested, while making fewer false alarms. This suggests that by combining the visual evidence from space with the hard numbers from ground-based records, the system can see the storm more clearly than either method could alone. The researchers also found that the system remained robust even when they artificially removed large chunks of the input data, simulating the kind of missing information that often occurs during real disasters.

While the study demonstrates a clear improvement in forecasting accuracy, the researchers emphasize that this is a tool to assist human forecasters, not a replacement for them. The model works by learning from historical data, meaning it is as good as the records it has been trained on. The success of TCFNet highlights a broader shift in meteorology, where the future of forecasting lies in blending different types of data in smarter ways. By teaching computers to respect the physical laws of the atmosphere while learning from the visual evidence of the clouds, scientists are taking a significant step toward giving communities more time to prepare for the fury of tropical cyclones. The code for this new system has been made available to the public, inviting other researchers to build upon this foundation and further refine our ability to predict nature's most violent storms.

Technical Summary: Physics-Guided Multimodal Learning for Flexible Tropical Cyclone Intensity Forecasting

Problem Definition

Accurate forecasting of Tropical Cyclone (TC) intensity (characterized by maximum sustained wind and minimum sea-level pressure) is critical for disaster mitigation but remains challenging due to complex, nonlinear storm dynamics. While recent AI-based methods have shown promise, they face three primary limitations:

Lack of Physical Consistency: Existing multimodal approaches often fuse satellite imagery and track data using generic strategies without explicitly modeling the consistency between visual features (cloud patterns) and physical variables (intensity trends). Similar visual patterns can correspond to vastly different physical evolution states (e.g., intensification vs. weakening), confusing generic feature learners.
Rigid Temporal Constraints: Most methods require complete, regular historical observations, failing to adapt to practical scenarios where satellite scans are irregular, data is missing, or forecasts are needed at flexible lead times.
Limited Adaptability: Current models struggle to capture abrupt intensity changes and often rely on coarse-resolution reanalysis data or single-modality inputs that lack the necessary spatial or temporal resolution.

Methodology: TCFNet

The authors propose TCFNet, a physics-guided multimodal learning framework designed to integrate geostationary satellite imagery and best-track data. The architecture consists of three core modules:

1. Asynchronous History Encoding

To handle irregular sampling and the modality gap between high-dimensional images and low-dimensional track variables, TCFNet employs a dual-stream strategy using Time-Aware GRUs:

Visual Stream: Processes satellite imagery ( $X_v$ $X_{v}$ ). To ensure the encoder learns intensity-aware patterns rather than static appearance, the authors introduce PhyMoCo, a physics-guided pre-training encoder based on Momentum Contrast (MoCo).
- Innovation: Instead of sampling positive pairs based solely on temporal proximity, PhyMoCo uses a Trend-Constrained Positive Pair Sampling strategy. It aligns visual features in the latent space with intensity trends (derived from pressure gradients), ensuring that images with similar intensification/weakening trends are mapped closer together, regardless of temporal distance.
Physical State Stream: Processes best-track variables ( $X_p$ ) (e.g., wind speed, pressure). This stream bypasses complex feature extraction, directly encoding the low-dimensional state sequence.
Both streams incorporate time intervals ( $\Delta \tau$ ) as explicit inputs to the GRUs, making the representations robust to asynchronous and irregular sampling.

2. Continuous-Time State Propagation

Unlike discrete recurrent models constrained by fixed time steps, TCFNet models TC evolution as a continuous physical process governed by Ordinary Differential Equations (ODEs).

The latent states from the encoding phase serve as initial conditions.
A neural network $f_\phi$ explicitly conditions on time to define the derivative of the state: $dz(s)/ds = f_\phi(z(s), s)$ .
An adaptive-step ODE solver extrapolates these states to arbitrary target timestamps, enabling continuous-time forecasting and robustness to missing observations.

3. Visual-Conditioned Asymmetric Fusion

To effectively integrate the two modalities, TCFNet employs a fusion mechanism inspired by Feature-wise Linear Modulation (FiLM), reinterpreted for TC forecasting:

Concept: Visual features (cloud compactness, eye structure) are treated as environmental context that modulates the physical state trajectories.
Mechanism: The visual latent state generates scaling ( $\gamma$ ) and shifting ( $\beta$ ) parameters. These parameters are applied to the normalized physical state via an affine transformation: $z_{fused} = (1 + \gamma) \odot \hat{z}_p + \beta$ .
This asymmetric approach ensures that the physical dynamics are dynamically adjusted based on the visual context at each prediction step.

Key Contributions

Physics-Guided Framework: TCFNet is the first framework to explicitly use physical variables (intensity trends) to guide visual representation learning, addressing the semantic inconsistency between similar visual patterns and different physical states.
Flexible Temporal Modeling: By combining time-aware encoding with continuous ODE-based propagation, the model handles missing multimodal observations and supports forecasting at arbitrary, flexible lead times without retraining.
Asymmetric Fusion: The introduction of visual-conditioned fusion allows the model to leverage satellite imagery as a contextual modulator for physical state evolution, rather than treating both modalities as equal inputs.

Experimental Results

Experiments were conducted on two cyclone-active basins: the Western North Pacific (WNP) and the South Pacific (SP), using data from the DigitalTyphoon dataset.

Overall Performance: TCFNet outperformed state-of-the-art baselines, including unimodal models (LSTM, Transformer, PatchTST, Mamba), vision-only models (ResNet, CNNLSTM), and multimodal baselines (PIPE-3B, SatSolarCast).
- In the WNP basin, TCFNet achieved the lowest Pressure RMSE (3.03 hPa) and Wind RMSE (7.24 kt) at T+6, maintaining superior performance at T+24 (9.52 hPa and 13.60 kt).
Rapid Intensification (RI): TCFNet demonstrated superior capability in capturing abrupt intensity changes. For Rapid Intensification events in WNP, it achieved a Critical Success Index (CSI) of 45.26%, significantly outperforming the next best method (PIPE-3B at 38.81%).
Robustness to Missing Data: The model maintained functionality even when up to 75% of input observations were randomly dropped, though performance degraded gracefully with higher missing rates.
Ablation Studies:
- Removing the visual stream increased Pressure RMSE by ~21.7% at T+24, highlighting the importance of satellite imagery for long-term trends.
- The PhyMoCo pre-training with trend-constrained sampling was crucial; removing it increased Pressure RMSE by 5.27%.
- The visual-conditioned fusion outperformed standard Sum and Concat fusion strategies.

Significance and Claims

The paper claims that TCFNet represents a significant step forward in TC intensity forecasting by bridging the gap between data-driven learning and physical consistency.

Adaptability: The ability to handle irregular sampling and missing data makes the model more suitable for real-world operational scenarios where data continuity cannot be guaranteed.
Interpretability: By conditioning physical states on visual trends, the model better captures the underlying dynamics of storm evolution, particularly during critical phases like rapid intensification or weakening.
Generalizability: The framework's success across two distinct basins (WNP and SP) suggests that the physics-guided approach is transferable and not overfitted to a specific region.

The authors conclude that integrating physical constraints into multimodal learning is essential for improving the accuracy and reliability of TC intensity forecasts, particularly for capturing non-linear temporal changes that traditional statistical or generic deep learning models miss.