Transformer self-attention encoder-decoder with multimodal deep learning for response time series forecasting and digital twin support in wind structural health monitoring

This paper presents a novel transformer-based encoder-decoder model with multimodal deep learning that accurately forecasts wind-induced structural responses and serves as an adaptive digital twin for the Hardanger Bridge, enabling early detection of structural anomalies without relying on assumptions of environmental or behavioral stationarity.

The Gist

Imagine a massive suspension bridge, like the Hardanger Bridge in Norway, as a giant, living instrument. Every day, it sings a song made of vibrations caused by the wind, traffic, and its own weight. Usually, this song is predictable. But when the wind changes direction or speed, or if a part of the bridge gets a "cough" (damage), the song changes.

The problem for engineers is that the wind is chaotic. It's like trying to predict the exact shape of a cloud while a storm is blowing. Traditional methods try to build a perfect mathematical map of how the wind hits the bridge, but they often get confused when the weather changes or when the bridge starts acting strangely. They might scream "Fire!" (false alarm) when it's just a gust of wind, or they might miss a real fire because they were too busy looking at the clouds.

The Solution: A "Super-Listener" AI

This paper introduces a new kind of Artificial Intelligence (AI) called a Transformer. Think of this AI not as a calculator, but as a super-listener with a perfect memory.

Here is how it works, using simple analogies:

1. The Two Inputs: The Conductor and the Orchestra

Most old systems only listen to the orchestra (the bridge's vibrations). They try to guess what the music should sound like based on what they heard a second ago.

• The Old Way: "The violin played a high note, so the next note should be high." (But what if the wind suddenly changed the tempo?)
• The New Way (Multimodal): This AI listens to two things at once:
  1. The Conductor (The Wind): It watches the wind speed, direction, and turbulence.
  2. The Orchestra (The Bridge): It listens to the vibrations of the bridge.

By watching the conductor and listening to the orchestra simultaneously, the AI learns the true relationship between them. It understands, "Ah, when the wind blows from the North at 20 mph, the bridge usually sways like this."

2. The "Crystal Ball" (Forecasting)

The AI doesn't just record the past; it acts like a crystal ball. It looks at the last few seconds of wind and vibrations and predicts what the bridge will do in the next few seconds.

• The Magic: It doesn't need to know the physics of steel or aerodynamics. It just learns the pattern from the data. It's like a child learning to catch a ball; they don't need to know the formula for gravity, they just watch the ball and learn where it will go.

3. The "Digital Twin" (The Shadow Self)

The paper calls this a Digital Twin. Imagine the bridge has a perfect, invisible shadow twin living inside a computer.

• The real bridge is out in the storm.
• The AI predicts what the shadow twin should be doing based on the wind.
• The Alarm System: If the real bridge starts doing something the shadow twin didn't predict (like shaking violently when the wind is calm), the AI raises a red flag. "Hey! The bridge is doing something weird! It might be broken!"

4. Why is this better than the old way?

• No "Perfect Weather" Assumptions: Old models assumed the wind was steady and predictable. This AI handles messy, real-world storms perfectly.
• Fewer False Alarms: Because it knows the difference between "windy day" and "broken bridge," it stops crying wolf.
• Spotting the Invisible: It can detect tiny changes in the bridge's "song" that human ears (or old computers) would miss, acting as an early warning system for damage before it becomes a disaster.

The Real-World Test

The researchers tested this on the Hardanger Bridge, a real bridge in Norway. They fed the AI years of data from wind sensors and vibration sensors.

• The Result: The AI was incredibly accurate. It predicted the bridge's movements much better than the old methods, even when the wind was changing rapidly. It successfully learned the "normal" behavior of the bridge so well that any deviation stood out clearly.

The Big Picture

This technology is like giving infrastructure a smartwatch. Instead of waiting for a bridge to crack or collapse, we can now have a system that constantly monitors its "heartbeat," predicts its future movements, and alerts us the moment it gets sick. It's a step toward "self-healing" cities where our bridges and buildings can talk to us about their health.

Technical Summary

1. Problem Statement

Structural Health Monitoring (SHM) of wind-excited bridges faces significant challenges due to the non-stationary and uncertain nature of environmental conditions.

• The "Normality" Dilemma: Traditional SHM methods struggle to distinguish between changes in structural health (damage) and changes caused by varying environmental conditions (wind speed, turbulence, traffic). Defining "normal" vibration behavior is difficult when wind conditions are non-stationary.
• Limitations of Existing Models:
  • Physics-based models often require precise aerodynamic parameters and assume stationarity, which is rarely true in real-world scenarios.
  • Single-modal Deep Learning (using only acceleration data) fails to capture the causal relationship between wind excitation and structural response, leading to poor predictions when environmental conditions shift.
  • Long-term Dependencies: Many existing models fail to capture the long-term temporal dependencies inherent in structural dynamics under turbulent wind.

2. Methodology

The authors propose a Multimodal Deep One-Dimensional Transformer Neural Network designed for response forecasting and early-warning anomaly detection.

A. Architecture: Encoder-Decoder Transformer

The model utilizes a self-attention mechanism to jointly process two distinct data modalities:

1. Encoder (Wind Modality): Processes historical wind features (horizontal speed, direction, turbulence intensity, temperature). It embeds these features with positional encoding to create a memory tensor ($M$) representing the environmental context.
2. Decoder (Structural Modality): Takes historical structural acceleration data as input. It uses:
   • Masked Self-Attention: To ensure causality (predicting future based only on past).
   • Cross-Attention: To attend to the wind memory tensor ($M$) generated by the encoder. This allows the model to learn the causal influence of specific wind events on structural vibrations.
3. Forecasting Mechanism: The model operates autoregressively. It predicts the next time step, feeds that prediction back into the decoder, and iterates to forecast a horizon of $L_{pred}$ steps (1, 8, or 18 steps ahead).

B. Training Strategy

• Multimodal vs. Single-Modal: The study compares two scenarios:
  • Acceleration-only: The encoder is bypassed (dummy memory).
  • Multimodal: Uses both wind and acceleration history.
• Loss Function: Optimized using Mean Squared Error (MSE) between predicted and measured accelerations.
• Digital Twin Component: The model serves as a "digital twin" by learning the system's dynamic behavior without explicit physical modeling. Deviations between the forecasted response and actual measurements serve as early-warning indicators for structural changes.

C. Data Source and Preprocessing

• Dataset: Real-world data from the Hardanger Bridge (Norway), monitored by the Norwegian University of Science and Technology (NTNU).
• Sensors: Synchronized data from anemometers (wind) and tri-axial accelerometers (deck motion).
• Preprocessing: Includes noise reduction (Savitzky-Golay smoothing), outlier handling, detrending, and Z-score normalization.

3. Key Contributions

1. Novel Architecture: First application of a multimodal encoder-decoder Transformer specifically for wind-structure interaction forecasting, eliminating the need for explicit aerodynamic modeling.
2. Handling Non-Stationarity: The model does not assume wind stationarity or a fixed "normal" vibration behavior. It learns the dynamic relationship between changing wind inputs and structural outputs directly from data.
3. Cross-Modal Attention: Demonstrates that integrating wind data via cross-attention significantly improves the retention of modal energy and reduces prediction errors compared to acceleration-only models.
4. Digital Twin Framework: Establishes a lightweight, data-driven digital twin component capable of continuous learning and adaptive monitoring over the structure's lifecycle.

4. Key Results

The model was validated on the Hardanger Bridge dataset across three axes (Longitudinal $x$, Transverse $y$, Vertical $z$) and multiple forecasting horizons (1, 8, 18 steps).

• Time-Domain Performance:

  • Peak Error Reduction: The multimodal model significantly reduced peak error compared to the acceleration-only baseline. For the vertical ($z$) axis at 18 steps, peak error improved from −41.6% to −4.4%.
  • Energy Retention: The Root-Mean-Square Ratio (RMSR) for the $z$-axis improved from 0.60 (accel-only) to 0.96 (multimodal), indicating near-perfect retention of vibration energy.
  • Win Rate: The multimodal model achieved a win rate of 63–68% over the baseline across all axes and horizons regarding RMSE and MAE.

• Frequency-Domain Performance:

  • Modal Peak Error (MPE): At longer horizons (Step 18), the multimodal model reduced MPE by 28% (from 58.8% to 42.2%).
  • Band Energy Retention (BER): The multimodal model maintained BER values closer to unity (perfect retention) compared to the acceleration-only model, which suffered from significant attenuation.

• Risk and Anomaly Detection:

  • The multimodal model reduced tail-risk (probability of errors exceeding $3\sigma$). The probability of large residuals ($p_{>3\sigma}$) dropped significantly (e.g., from 0.6% to 0.0% in the $y$-axis at Step 18), indicating fewer false alarms and more reliable anomaly detection.

• Comparison with CNN: The Transformer architecture outperformed a baseline 1D Convolutional Neural Network (CNN) across all metrics, highlighting the superiority of self-attention for capturing long-range temporal dependencies in this context.

5. Significance and Implications

• Resilient Infrastructure Management: The framework provides a robust tool for monitoring bridges under variable environmental conditions, addressing a critical gap in current SHM practices where environmental variability often masks damage.
• Operational Efficiency: The model is computationally efficient (1D input) and suitable for real-time deployment on edge devices or centralized platforms.
• Scalability: It requires only local sensor data (no need for a full-system physics model), making it applicable to localized damage detection and adaptable to various infrastructure types beyond bridges.
• Future-Proofing: By functioning as a digital twin that can be continuously retrained with new data, the system adapts to aging structures and changing environmental patterns, supporting long-term lifecycle management.

Conclusion: The study demonstrates that a multimodal Transformer architecture is a superior approach for forecasting wind-excited structural responses. It effectively decouples environmental effects from structural health indicators, offering a powerful, data-driven solution for next-generation structural health monitoring and digital twin applications.