Time-Series Foundation Model Embeddings for Remaining Useful Life Estimation

Technical Summary: Time-Series Foundation Model Embeddings for Remaining Useful Life Estimation

Problem Statement
Remaining Useful Life (RUL) prediction is a critical component of industrial predictive maintenance, aiming to forecast equipment failure to minimize downtime. Traditional data-driven approaches for RUL estimation typically rely on either extensive handcrafted feature engineering or end-to-end training of task-specific sequence models (e.g., LSTMs, GRUs, Transformers). These methods often face significant limitations: they require large volumes of labeled data, necessitate careful hyperparameter tuning for specific sensor types, and struggle to generalize across different datasets or operating conditions without resource-intensive retraining. While self-supervised and contrastive learning have been explored to reduce labeling requirements, they still generally require training on task-specific data.

Methodology
The authors propose a lightweight learning approach that leverages a frozen, pretrained Time-Series Foundation Model (TSFM) to extract temporal context features, which are then fed into a small, trainable regression head for RUL estimation.

• Backbone (Frozen): The core of the method utilizes Chronos-2, a probabilistic, decoder-only Transformer-based TSFM. The model is kept frozen (parameters not updated) and serves as a feature extractor. It processes multivariate sensor streams by tokenizing input time series into discrete bins.
• Data Preprocessing: Raw sensor data, often recorded at irregular timestamps, is linearly interpolated to a uniform time grid. Outliers are clipped to the 1st and 99th percentiles, and channels are normalized to zero mean and unit variance based solely on training statistics to prevent data leakage. RUL labels are constructed from maintenance logs by calculating the time remaining until the next recorded failure event, capped at 1000 days.
• Architecture:
  1. Context Extraction: A window of $L$ timesteps ($X_{t-L:t}$) from the multivariate sensor stream is fed into the frozen Chronos-2 backbone. The model extracts a sequence of hidden representations ($H$) capturing temporal dependencies within the context window.
  2. Regression Head: A lightweight Multi-Layer Perceptron (MLP) with two linear layers and ReLU activations is trained on top of the extracted embeddings to predict the RUL ($\hat{y}_t$). The output is constrained to be non-negative via a final ReLU.
• Training Protocol: Only the regression head parameters are optimized using Mean Squared Error (MSE) loss and the Adam optimizer. The backbone remains fixed, making the adaptation process computationally efficient.

Key Contributions and Results
The paper evaluates this approach on real-world industrial sensor data from two distinct device types (Device A and Device B) provided by Nokia Solutions and Networks.

• Performance Gains: The proposed method significantly outperforms a broad set of baselines, including non-sequential regressors (Linear Regression, Random Forest, Gradient Boosting) and sequential deep learning models (LSTM, GRU, Temporal CNN, Transformer encoders). On Device A with a 5-step context window, the method achieved a Mean Absolute Error (MAE) of 44, compared to 88 for the next best baseline (TCN), representing a substantial improvement.
• Impact of Context Length: A critical finding is the sensitivity of performance to context length ($L$). While traditional baselines showed only mild fluctuations as $L$ increased, the Chronos-2-based approach demonstrated significant performance gains as the context window expanded. Increasing $L$ from 5 to 80 steps reduced the MAE by approximately 2×, suggesting that TSFM representations effectively leverage long-range historical data that standard sequence models fail to utilize.
• Ablation Study: Experiments varying the regression head capacity revealed that even a simple linear head on top of frozen Chronos-2 embeddings outperformed the TCN baseline (MAE 60 vs. 88). This indicates that the pretrained foundation model provides the primary representational power, with the regression head adding necessary non-linearity for mapping embeddings to RUL values.

Significance and Claims
The authors position this work as a practical and data-efficient alternative for RUL estimation in industrial settings. By treating RUL prediction as a supervised regression problem conditioned on a frozen TSFM, the approach eliminates the need for extensive task-specific pretraining or large labeled datasets. The study demonstrates that foundation models, originally designed for forecasting, can be effectively repurposed for prognostics by extracting rich temporal context features. The results suggest that TSFMs offer a robust way to obtain temporally rich representations without the computational cost of training deep sequence models from scratch, while the ability to scale with longer context histories addresses a key limitation of previous methods. Future work outlined by the authors includes leveraging probabilistic outputs for uncertainty quantification and improving robustness to missing sensors and regime shifts.