Contrast All the Time: Learning Time Series Representation from Temporal Consistency

Technical Summary: CaTT (Contrast All The Time)

Problem Statement
Time series (TS) representation learning is critical for downstream tasks, yet large-scale unlabeled TS data remains underutilized due to the high cost of annotation. While self-supervised contrastive learning has succeeded in computer vision (CV) and natural language processing (NLP), its application to time series faces specific challenges. Existing TS contrastive methods often rely on data augmentations, which can introduce inductive biases, or they depend on heuristic pair selection (e.g., selecting specific positive/negative pairs from a batch). These selection strategies are often inefficient, fail to leverage the full temporal dimension of the data, and struggle with scalability in dynamic, real-world streaming environments. Furthermore, many approaches do not effectively utilize the inherent temporal consistency where adjacent time steps should naturally share similar representations.

Methodology
The authors propose CaTT (Contrast All The Time), a novel unsupervised contrastive learning framework designed to learn time series representations by leveraging the natural temporal structure of the data without relying on data augmentations or complex pair-selection heuristics.

• Core Concept: CaTT departs from conventional approaches by contrasting all time steps in parallel. Instead of selecting a single positive pair per anchor, it utilizes the full temporal dimension, treating adjacent time steps as implicit positive pairs and all other non-adjacent steps as negatives.
• MP-Xent Loss: The framework introduces a Multiple Positive Cross-Entropy (MP-Xent) loss. This is an adaptation of the classic N-pair loss (Sohn, 2020) and the NT-Xent loss (SimCLR).
  • For a batch of $N$ sequences with length $T$, CaTT forms an NT-tuple loss.
  • For a reference time step $z_{i,t}$, the adjacent steps ($z_{i,t-1}$ and $z_{i,t+1}$) serve as multiple positive pairs.
  • All other steps in the sequence (excluding the reference and its immediate neighbors) serve as negative pairs.
  • The loss function is formulated to maximize the similarity of the anchor to its multiple positives while minimizing similarity to all negatives, all computed efficiently via matrix operations to avoid the high computational complexity of naive slicing.
• Architecture: The method employs a simple 1D Convolutional Neural Network (CNN) as the encoder backbone. The input is a time series sequence $x$ of length $T$, which is projected into a representation space $z$. The training is end-to-end, requiring no separate identification step for positive pairs.
• Preprocessing: For specific datasets (e.g., medical signals), the authors apply Short-Time Fourier Transform (STFT) to sub-windows to generate temporally dependent instances, ensuring the assumption that consecutive instances share similar characteristics holds true.

Key Contributions

1. CaTT Framework: A new unsupervised contrastive learning framework for time series that integrates positive pair selection directly into the learning process using an NT-pair formulation, eliminating the need for external pair selection heuristics.
2. MP-Xent Loss: The adaptation of the N-pair loss to introduce multiple positive pairs (adjacent time steps) into the normalized temperature-scaled cross-entropy objective. This modification accelerates learning by considering more pairs per update and improves convergence.
3. Extensive Evaluation: The authors conducted experiments on 16 public datasets across four distinct downstream tasks:
   • Linear fine-tuning classification (frozen backbone).
   • Semi-supervised classification.
   • Time series forecasting (short-term and long-term).
   • Anomaly detection.

Results
The paper reports that CaTT produces superior embeddings compared to state-of-the-art baselines (including TS2Vec, InfoTS, CoST, SimMTM, and MF-CLR) across all evaluated tasks:

• Classification: On the HARTH, SLEEPEEG, and ECG datasets, CaTT achieved the highest accuracy, F1 scores, precision, and recall. Notably, it achieved these results with significantly lower training times (e.g., 6.12s vs. 2322.75s for MF-CLR on HARTH).
• Semi-Supervised Learning: CaTT demonstrated robustness with limited labeled data (5%, 10%, 20%), consistently outperforming baselines in accuracy.
• Forecasting: On ETT and Weather datasets, CaTT delivered competitive Mean Squared Error (MSE) and Mean Absolute Error (MAE) results, often outperforming or matching specialized forecasting models like MF-CLR and CoST, while maintaining superior pretraining efficiency.
• Anomaly Detection: In the TSB-AD benchmark, CaTT showed competitive performance in detection metrics (VUS-PR) and was the fastest method among those tested.
• Efficiency: A primary finding is that CaTT is significantly faster to train than existing contrastive approaches, making it suitable for large-scale applications.

Significance and Claims
The paper claims that CaTT represents a shift in how temporal consistency is utilized in contrastive learning. By moving away from data augmentation and selective pair sampling toward a "contrast all the time" approach, the method learns directly from the natural structure of temporal data. The authors assert that:

• The use of multiple positive pairs (adjacent time steps) within an NT-tuple loss is crucial for learning semantically meaningful representations and ensuring training stability (single-positive approaches often resulted in NaN losses).
• The approach is highly efficient, offering a balance of high performance and low computational cost that is particularly well-suited for real-world, dynamic time series applications where inference speed and scalability are critical.
• The method is versatile, performing effectively across classification, forecasting, and anomaly detection tasks without requiring task-specific architectural changes.

The source code for the method is publicly available, facilitating reproducibility and further research.