🤖 machine learning

Modular Foundation Models for Time-Series Perception in Digital Twins

This paper proposes a modular foundation model for time-series perception in Digital Twins that leverages self-supervised pretraining, a dynamic gating mechanism for encoder selection, and Transformer-based aggregation to create robust, transferable representations for diverse Prognostics and Health Management tasks.

Original authors: Quang Hung Pham, Ryad Zemouri, Martin Gagnon, Luc Vouligny

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quang Hung Pham, Ryad Zemouri, Martin Gagnon, Luc Vouligny

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a "Digital Twin"—a virtual, living copy of a massive, complex machine like a hydroelectric generator. To make this virtual copy work, you need a "perception system" that can look at a chaotic stream of data (vibrations, temperatures, electrical signals) and understand what the machine is doing.

The problem is that real-world machines are messy. They run at different speeds, in different weather, and with different levels of wear and tear. Traditional AI models are like specialists who only know how to fix one specific problem in one specific weather condition. If the machine changes, the model breaks.

This paper proposes a new solution: a Modular Foundation Model. Think of this not as a single expert, but as a super-smart "Swiss Army Knife" team designed to handle any time-series data.

Here is how it works, broken down into simple concepts:

1. The Team of Specialists (The Encoders)

Instead of training one giant brain to do everything, the authors built a "library" of smaller, specialized brains called Encoders.

How they learn: Each encoder was trained on its own unique dataset (like vibration data, heartbeats, or electricity usage) using a technique called Self-Supervised Learning.
The Analogy: Imagine a group of chefs. One chef only knows how to bake bread, another only knows how to grill fish, and another only knows how to make soup. They didn't learn by being told "make a sandwich"; they learned by practicing their specific craft over and over until they mastered the essence of their ingredient.
The Result: These chefs (encoders) are now frozen. They are experts at recognizing patterns in their specific type of data, but they don't know how to solve your specific problem yet.

2. The Smart Manager (The Gating Mechanism)

When you feed new data into the system (say, a new hydro-generator), you don't want to ask the "bread chef" to fix a "grill" problem. You need a manager to decide who to call.

The Process: A "Gating Mechanism" acts like a smart dispatcher. It looks at the incoming data and asks: "Does this look like the data the 'vibration chef' knows? Does it match the 'temperature chef'?"
The Selection: It picks the top few specialists whose past experience matches the current situation best. It ignores the rest. This saves energy and ensures the right expertise is applied.

3. The Translator and the Huddle (Projection & Aggregation)

The selected chefs speak different "languages" (their data is in different formats). You can't just throw their notes together.

The Translator: A projection layer translates each chef's notes into a common language (a shared "latent space").
The Huddle: A Transformer (a type of AI that is great at connecting dots) acts as a meeting room. It takes the translated notes from the selected chefs and has them "talk" to each other. It figures out how the vibration expert's insights combine with the temperature expert's insights to form a complete picture.

4. The Task-Specific Tool (The Head)

Once the team has agreed on what the data means, a lightweight "Head" attaches to the front to do the actual job.

The Jobs: This could be filling in missing data (Imputation), predicting the future (Forecasting), or spotting a problem with very little data (Few-shot learning).
The Magic: Because the "brain" (the team) is already trained, you only need to train this tiny "Head" with a small amount of new data. It's like having a master team of chefs who can instantly learn to make a new dish just by being shown the recipe once, rather than training a whole new kitchen from scratch.

What Did They Prove?

The authors tested this "Swiss Army Knife" team in two ways:

The Standard Test (ETT Benchmark): They used public datasets for electricity transformers. The model was good at filling in missing data and predicting the future, even when it had to learn with very little new information (Few-shot learning). It performed competitively against other top-tier models.
The Real-World Test (Hydro-Generator): They applied it to a real industrial problem: estimating the temperature of a spinning generator rotor without a physical sensor (a "Virtual Sensor").
- The Result: The model successfully predicted the temperature and even gave a "confidence interval" (a range of uncertainty). It worked well on the machine it was trained on. When they tried it on a different machine, the predictions were still good at tracking the general trend, though the confidence range widened (which is expected when moving to a new, slightly different environment).

The Bottom Line

This paper argues that instead of building a new, fragile AI model for every single industrial task, we should build a modular foundation. By having a pre-trained team of specialists and a smart manager to pick the right ones, we can create a perception system that is flexible, robust, and ready to adapt to new machines and new conditions without needing massive amounts of new data. It turns the "Digital Twin" from a static snapshot into a living, learning partner.

Technical Summary: Modular Foundation Models for Time-Series Perception in Digital Twins

1. Problem Statement

Engineering Digital Twins (DTs) and Prognostics and Health Management (PHM) systems require robust perception modules capable of extracting actionable information from heterogeneous, non-stationary time-series data. Existing approaches face three primary limitations:

Task-Specificity: Current methods are often designed for single tasks, making them difficult to transfer across different operating conditions or assets.
Data Hunger: Traditional supervised pipelines require extensive labeled data, which is scarce in industrial settings due to limited labeling and class imbalance.
Non-Stationarity: Industrial systems operate under varying regimes (load fluctuations, aging, environmental disturbances), causing traditional models to struggle with generalization.

The paper argues that building separate, task-specific models for every new scenario is costly and unsustainable. Instead, a unified perception layer is needed that can leverage pre-existing knowledge from heterogeneous sources while adapting to new, unseen domains with limited supervision.

2. Methodology

The authors propose a Modular Foundation Model for Time-Series Processing (FM-TSP). Unlike end-to-end architectures, this framework explicitly decouples representation learning, encoder selection, and task adaptation. The architecture consists of three main components:

A. Collection of Pretrained Encoders ( $E$ )

The foundation is a library of $M$ pretrained representation encoders ( $E = \{E_1, \dots, E_M\}$ ).

Training Strategy: Each encoder is trained independently using self-supervised learning (SSL) on heterogeneous datasets (e.g., vibration, acoustic, electrical, physiological signals).
Objectives: The pretraining utilizes a dual-objective approach:
1. Reconstruction: A lightweight decoder reconstructs input segments using Huber loss to ensure robustness against outliers.
2. Contrastive Learning: A contrastive objective (NT-Xent) enforces invariance to partial signal corruption (masking), regularizing the latent space.
Specialization: Encoders are trained on diverse data subsets and architectures (MLP, CNN, Transformer) to capture a wide range of temporal dynamics and physical regimes.

B. Gating and Selection Mechanism ( $G_{gate}$ )

To handle a new target dataset $Y$ , the model does not use all encoders. Instead, a gating mechanism dynamically selects a subset of the most relevant encoders ( $E^*$ ). This selection is based on two criteria:

Data Similarity: Computes the dissimilarity between the target dataset and the pretraining datasets using statistical and spectral descriptors (e.g., moments, quantiles, Fourier coefficients).
Structural Correlation: Evaluates whether an encoder preserves the local similarity structure (neighborhood relationships) of the target data when mapping it to its latent space.
This ensures conditional computation, activating only encoders whose learned representations align with the target domain.

C. Projection and Aggregation Module ( $G_{agg}$ )

The selected encoders produce latent representations in heterogeneous spaces.

Projection: Learnable linear projectors map these heterogeneous outputs into a shared $d$ -dimensional latent space. An alignment regularization term (Maximum Mean Discrepancy, MMD) is applied during early fine-tuning to ensure semantic consistency across encoders.
Aggregation: A Transformer-based self-attention module processes the projected representations as a sequence of tokens. This allows the model to capture cross-encoder interactions and dependencies, dynamically reweighting features based on their relevance to the specific downstream task.

D. Task Adaptation ( $T$ )

The framework supports multiple downstream tasks (imputation, forecasting, virtual sensing) via lightweight, task-specific heads. During adaptation, the pretrained encoders remain frozen; only the projectors, aggregation module, and task heads are updated. This enables efficient few-shot learning.

3. Key Contributions

Modular Foundation Architecture: Proposes a decoupled framework where representation learning is separated from task adaptation, allowing for the reuse of pretrained encoders across diverse industrial scenarios.
Two-Stage Gating Mechanism: Introduces a principled encoder selection strategy based on data similarity and structural correlation, moving beyond random or naive aggregation to ensure computational efficiency and relevance.
Cross-Encoder Aggregation: Demonstrates the utility of Transformer-based self-attention in fusing heterogeneous representations, outperforming simple concatenation or mean pooling.
Self-Supervised Pretraining: Validates a dual-objective SSL approach (reconstruction + contrastive) for learning robust, task-agnostic time-series representations from unlabeled industrial data.

4. Experimental Results

The framework was evaluated on the ETT (Electricity Transformer Temperature) benchmark and a real-world industrial case study.

Benchmark Evaluation (ETT Dataset)

The model was tested on three tasks: Imputation, Long-term Forecasting, and Few-shot Forecasting.

Imputation: FM-TSP achieved performance comparable to or better than state-of-the-art models (PatchTST, iTransformer, DLinear), benefiting from the direct link between the pretraining reconstruction objective and the imputation task.
Long-term Forecasting: The model remained competitive with leading baselines (DLinear, iTransformer), though it trailed slightly behind PatchTST. The authors attribute this to the fact that encoders were not pretrained on the specific target ETT data, highlighting a potential area for improvement via a more diverse encoder library.
Few-shot Forecasting: In data-scarce scenarios (using only 5% or 10% of training data), FM-TSP ranked among the top performers, demonstrating effective knowledge transfer from heterogeneous pretraining sources.

Industrial Case Study: Virtual Sensing for Rotor Temperature

A virtual sensor was developed to estimate hydro-generator rotor temperature using operational signals, addressing the difficulty of direct measurement.

Training: The model was trained on one turbine-generator unit (TGU) and tested on others.
Performance: The model successfully tracked the ground truth mean temperature.
Uncertainty Quantification: By predicting distribution parameters (mean and variance) via a Beta-loss likelihood, the model provided calibrated uncertainty estimates. On the training TGU, coverage was ~93.5% (close to the nominal 95%). On a different TGU (transfer learning), coverage dropped to 65% due to distribution shifts, but the model still effectively identified critical high-temperature events, which are crucial for safety monitoring.

5. Significance and Claims

The paper positions the FM-TSP as a scalable perception layer for industrial Digital Twins and hybrid model-data PHM systems. Its significance lies in:

Scalability: By freezing encoders and only training lightweight heads, the framework reduces the cost of deploying models for new tasks or assets.
Robustness to Data Scarcity: The ability to leverage self-supervised pretraining on heterogeneous data allows the system to function effectively in few-shot regimes where labeled data is limited.
Modularity: The architecture supports incremental development; new encoders can be added to the library to cover new operating regimes without retraining the entire system.
Hybrid Integration: The authors claim the framework serves as a data-driven component that can be naturally integrated with physics-based models to enhance interpretability and robustness in safety-critical applications.

The authors conclude that while the framework is competitive with state-of-the-art end-to-end models, its primary value is in providing a reusable, modular foundation that addresses the specific challenges of non-stationarity and data heterogeneity inherent in engineering Digital Twins.