Disentangled Mode-Specific Representations for Tensor Time Series via Contrastive Learning

This paper proposes MoST, a novel contrastive learning framework that disentangles multi-mode tensor time series into mode-specific and mode-invariant representations to significantly improve classification and forecasting performance across various domains.

The Gist

Imagine you are trying to understand a massive, chaotic library. But this isn't just a library of books; it's a living, breathing library where every book changes its story every second, and every book is written in a different language, by a different author, about a different topic.

This is what a Tensor Time Series (TTS) is. It's data that has three or more dimensions happening at once:

1. Time: Things change over seconds, days, or years.
2. Location: Things happen in different places (like New York vs. Tokyo).
3. Category: Things happen in different groups (like "Search for Apple" vs. "Search for Amazon").

The Problem:
Most computer scientists try to read this library by squashing all the books into one giant, messy pile. They look at the whole thing at once. The problem is, the "Apple" stories in New York have a very different rhythm than the "Amazon" stories in Tokyo, even though they both happen at the same time. When you mash them all together, the computer gets confused and misses the unique patterns of each group.

The Solution: MoST (Mode-Specific Representations)
The authors of this paper built a new tool called MoST. Think of MoST as a super-smart librarian who refuses to mash the books together. Instead, MoST uses a special technique called "Tensor Slicing."

The Analogy: The "Slice and Dice" Chef

Imagine the data is a giant, multi-layered cake.

• Traditional methods try to eat the whole cake in one bite. They get a mix of chocolate, vanilla, and strawberry, but they can't taste the distinct flavor of the strawberry layer.
• MoST acts like a chef who carefully slices the cake horizontally and vertically.
  • Slice 1: All the "Location" layers (New York, Tokyo, etc.).
  • Slice 2: All the "Category" layers (Apple, Amazon, etc.).

MoST takes these slices and feeds them into separate, specialized kitchens (encoders).

1. The "Location" Kitchen: Learns how New York behaves differently from Tokyo. It learns the local habits.
2. The "Category" Kitchen: Learns how "Apple" searches differ from "Amazon" searches. It learns the topic habits.

The Secret Sauce: Contrastive Learning (The "Twin" Game)

How does MoST know it's doing a good job? It plays a game of "Spot the Difference" (which the paper calls Contrastive Learning).

• The "Twin" Game (Mode Loss): MoST looks at the "Location" slice and the "Category" slice at the exact same moment in time. It asks: "Even though these are different slices, they come from the same event. Do they share a common heartbeat?"
  • Example: If it's Christmas, both the "Location" slice and the "Category" slice should show a spike in activity. MoST learns to recognize this shared rhythm (the "invariant" feature).
• The "Chameleon" Game (Instance Loss): MoST takes a slice, cuts out a random piece of it (like cropping a photo), and asks: "Is this piece still the same story, even if I hide part of it?"
  • This teaches the computer to understand the unique details of that specific slice without getting confused by the noise.

By playing these games, MoST learns two things at once:

1. What makes each group unique (The specific flavor of the strawberry layer).
2. What connects them all together (The fact that they are all part of the same cake).

Why Does This Matter?

The paper tested MoST on real-world data, like:

• Google Search Trends: Predicting what people will search for next.
• Air Quality: Forecasting pollution levels in different cities.
• Bike Sharing: Predicting where people will ride bikes.

The Result:
MoST was much better at predicting the future and classifying data than previous methods. Why? Because it didn't try to force a square peg into a round hole. It respected the unique structure of the data.

In a nutshell:
If traditional AI is like listening to a choir where everyone is shouting at once, MoST is like a conductor who asks each section (strings, brass, woodwinds) to play their part separately, listens to how they harmonize, and then creates a perfect symphony. It separates the noise from the signal, allowing us to understand complex, multi-dimensional data for the first time.

Technical Summary

1. Problem Definition

Tensor Time Series (TTS) are multi-dimensional data structures where time is one dimension, and other dimensions (modes) represent different attributes (e.g., location, query, sensor type). Examples include search engine logs (Location $\times$ Query $\times$ Time) or environmental monitoring systems.

The core challenge in learning representations from TTS is twofold:

1. Intra-mode Dependencies: Variables within the same non-temporal mode (e.g., the relationship between different locations) have specific interactions.
2. Temporal Dependencies: All modes share common temporal patterns (e.g., seasonality, trends) because they originate from the same underlying system.

Existing methods fail to address this effectively:

• Traditional Tensor Decomposition: Focuses on reconstruction accuracy but often ignores temporal dynamics or treats all modes equally, missing specific intra-mode structures.
• Standard Time Series Contrastive Learning (e.g., TS2Vec): Designed for 2D matrices (variables $\times$ time), they flatten the tensor structure, thereby losing the rich multi-modal relationships inherent in TTS.

2. Methodology: MoST

The authors propose MoST (Mode-Specific representations for Tensor time series), a self-supervised learning framework that disentangles mode-specific features from mode-invariant features using Contrastive Learning (CL).

A. Model Architecture

The architecture consists of three main components (illustrated in Fig. 2a of the paper):

1. Tensor Slicing:

   • Instead of processing the whole tensor at once, the input TTS is sliced along each non-temporal mode.
   • For a 3rd-order TTS (Mode 1 $\times$ Mode 2 $\times$ Time), the model generates two sets of slices:
     • Mode-1 Slices: Sets of matrices where Mode 2 is fixed (capturing Mode 1 dependencies).
     • Mode-2 Slices: Sets of matrices where Mode 1 is fixed (capturing Mode 2 dependencies).
   • This reduces complexity and isolates intra-mode relationships.

2. Slice Feature Encoder:

   • Each slice is fed independently into a Slice Feature Encoder (Mode-Independence approach).
   • Embedding: Linear projection maps raw signals to a latent space, augmented with temporal embeddings to preserve time order.
   • Causal Convolution: A stack of causal convolutional blocks captures long-range temporal dependencies at multiple scales.
   • Pooling: An average-pooling layer summarizes the latent representation across time scales for each slice.

3. Aggregator:

   • The representations of all slices within a specific mode are aggregated (via pooling) to form a Mode-Specific Representation ($V^{(d1)}$ and $V^{(d2)}$).
   • The final output is the concatenation of these disentangled representations: $V = [V^{(d1)}; V^{(d2)}]$.

B. Contrastive Learning Framework

MoST employs a custom contrastive loss function composed of two parts to learn both specific and invariant features:

1. Instance Loss ($L_I$):

   • Goal: Learn mode-specific features.
   • Mechanism: Uses random cropping to create two augmented views of the same tensor. Representations of the same timestamp from different augmentations are treated as positive pairs. Representations from different time series samples are negative pairs.
   • This forces the model to learn robust features specific to the data instance, invariant to local perturbations.

2. Mode Loss ($L_M$):

   • Goal: Learn mode-invariant features (temporal dependencies common across modes).
   • Mechanism: Treats the Mode-1 representation ($V^{(d1)}$) and Mode-2 representation ($V^{(d2)}$) of the same time series at the same timestamp as a positive pair.
   • It treats representations from other time series samples (even if they share the same timestamp) as negative pairs.
   • This encourages the model to align the temporal dynamics (e.g., seasonality) across different modes while keeping their specific structural features distinct.

Total Loss: $L = L_I + \alpha(L_M^{(d1)} + L_M^{(d2)})$, where $\alpha$ is a hyperparameter.

3. Key Contributions

1. Novel Representation Learning for TTS: MoST is the first work to provide TTS representations specifically via Contrastive Learning, addressing the unique structure of tensor data.
2. Disentangled Learning Strategy: By introducing Tensor Slicing and a dual-loss contrastive framework, MoST successfully separates intra-mode dependencies (specific to a mode) from temporal dependencies (invariant across modes).
3. Comprehensive Evaluation: The method was validated on 11 real-world datasets covering diverse domains (search trends, environmental monitoring, mobility, and motion sensors), demonstrating superior performance in both classification and forecasting tasks.

4. Experimental Results

The authors compared MoST against state-of-the-art baselines:

• Baselines: CoST (disentangled time series), TS2Vec (general time series), TS-TCC, ATD (tensor decomposition + CL), and various forecasting models (Informer, LaST, SSMF).

Key Findings:

• Classification: MoST achieved the highest accuracy on motion sensor datasets (Daily and Realdisp), outperforming TS2Vec and ATD. The ablation study showed that ignoring mode dependencies (e.g., using Channel-Dependence) significantly degraded performance.
• Forecasting: MoST consistently outperformed all baselines across 9 datasets (Google Trends, KnowAir, NYC-CB) with varying forecast horizons (2 to 32 weeks).
  • MoST reduced MSE significantly compared to tensor decomposition methods (NET3, SSMF) and standard CL methods (CoST, TS2Vec).
  • The ablation study confirmed that both the tensor slicing approach and the Mode Loss are critical; removing Mode Loss hurt classification accuracy, while removing the slicing approach hurt forecasting significantly.
• Ablation Study:
  • Tensor Slicing: Essential. Randomizing the tensor or treating channels independently (CI) resulted in lower accuracy.
  • Architecture: Temporal embeddings and Causal Convolutions were proven necessary for capturing long-term dependencies.
  • Loss Functions: The combination of Instance and Mode losses was superior to using MSE loss alone (end-to-end training).

5. Significance

• Generalizability: Unlike previous tensor methods that are often task-specific or rely on specific physical models (e.g., Lotka-Volterra), MoST is a general-purpose framework applicable to any TTS data.
• Interpretability: The disentangled nature of the representations allows for better analysis of how different modes interact versus how they share temporal trends.
• Performance: By explicitly modeling the tensor structure rather than flattening it, MoST unlocks richer feature representations, leading to state-of-the-art results in downstream tasks.

In conclusion, MoST represents a significant advancement in time series analysis by bridging the gap between tensor decomposition and modern contrastive learning, offering a robust solution for complex, multi-attribute temporal data.