Technical Summary: Mantis: Lightweight Foundation Model for Time Series Classification

Problem Statement

While foundation models have revolutionized vision and language domains, their application to time series classification (TSC) remains under-explored. Existing literature predominantly focuses on time series forecasting, often utilizing pre-training paradigms (e.g., masked reconstruction or autoregressive modeling) that are inherently better suited for imputation or prediction rather than discriminative tasks. Although time series classification is a fundamental problem across diverse domains such as human activity recognition, healthcare, finance, and neuroscience, there is a scarcity of foundation models specifically designed for this task. Furthermore, existing approaches often rely on fine-tuning, whereas there is a need for robust zero-shot feature extraction capabilities that can generalize without extensive labeled data or parameter updates.

Methodology

The authors introduce Mantis, a lightweight, transformer-based foundation model pre-trained exclusively on synthetic data via self-supervised contrastive learning. The methodology is built upon three core technical pillars:

1. Time-Series Multi-View Tokenizer

To effectively adapt the Transformer architecture to time series, Mantis employs a dedicated Token Generator Unit that maps raw sequences to a fixed number of tokens (32) to control attention costs. This unit processes the input through three parallel branches:

• Signal Branch: Applies instance normalization, followed by convolution and mean pooling to extract patch-level features.
• Differential Branch: Applies the same patching procedure to the first-order temporal difference of the signal to encourage stationarity and reduce long-term trend influence.
• Statistics Branch: Splits the raw (unnormalized) signal into non-overlapping patches and encodes the mean and standard deviation of each patch using a Multi-Scaled Scalar Encoder.
  These features are concatenated and linearly projected into token embeddings.

2. Synthetic Contrastive Pre-Training

Unlike many foundation models pre-trained on real-world datasets (which may overlap with evaluation benchmarks), Mantis is pre-trained exclusively on synthetic data generated by CauKer.

• Objective: The model is trained using a self-supervised contrastive InfoNCE loss. The goal is to learn representations where augmentations of the same time series are close (positive pairs) while representations of different series are far apart (negative pairs).
• Augmentation: The primary augmentation strategy is Random Crop Resize (RCR), which randomly crops a segment of the time series and resizes it back to the original length. This encourages invariance to temporal stretching and compression, which is crucial for classification but less suitable for forecasting.
• Rationale: Synthetic data provides high diversity and sample efficiency, ensuring the model learns generalizable representations without data leakage from standard benchmarks like UCR or UEA.

3. Enhanced Test-Time Methodology

To bridge the performance gap between foundation models and specialized approaches, the authors propose an inference-time pipeline that leverages the frozen encoder:

• Layer-wise Representations: Instead of relying solely on the final Transformer layer, the model utilizes intermediate-layer representations. Experiments show that intermediate layers often encode more discriminative features for classification tasks, particularly as pre-training data scales increase.
• Output-Token Aggregation: The authors propose concatenating the [CLS] token with the mean of the non-classification tokens, capturing complementary information often discarded in standard discriminative Transformers.
• Self-Ensembling (SE): The input time series (and its first-order difference) are interpolated to multiple sequence lengths. Since the number of tokens is fixed, varying the interpolation length changes the receptive field and overlap between tokens, generating complementary embeddings that are concatenated.
• Cross-Model Embedding Fusion: Mantis embeddings are combined with those from other foundation models (e.g., vision-based models like TiViT-H or TiConvNext) to leverage complementary aspects of time series data.

Key Contributions

1. Mantis Architecture: A lightweight Transformer-based foundation model specifically tailored for time series classification, featuring a novel multi-view tokenization strategy.
2. Synthetic Pre-Training: Demonstrating that pre-training exclusively on synthetic data via contrastive learning yields strong zero-shot generalization, outperforming models pre-trained on real-world data of similar size.
3. Test-Time Optimization: Introducing a suite of inference-time techniques (intermediate layer selection, token aggregation, self-ensembling, and cross-model fusion) that significantly boost zero-shot performance without requiring fine-tuning.
4. State-of-the-Art Performance: Establishing new state-of-the-art results across multiple benchmarks (UCR, UEA, HAR, EEG) in the frozen-encoder regime.

Experimental Results

Extensive experiments were conducted on four diverse dataset collections:

• UCR Archive (128 univariate datasets): Mantis and its variations (SE-Mantis, fused models) consistently outperformed existing foundation models (e.g., MOMENT, Chronos2, TiRex) and specialized baselines (e.g., TS2Vec, Catch22+). Mantis achieved an average accuracy of 0.8195 on the full UCR benchmark, surpassing the next best foundation model.
• UEA Archive (27 multivariate datasets): Mantis significantly outperformed other models by more than 2%, achieving 0.7420 accuracy.
• Human Activity Recognition (HAR) & EEG: Mantis demonstrated strong performance on domain-specific benchmarks, including inertial measurement unit data and brain-computer interface signals, often outperforming vision-based adaptations and other TSFMs.
• Efficiency: Mantis is highly efficient, with only 2.2 million parameters (after pruning) and inference times comparable to statistical baselines like Catch22+, making it suitable for compute-limited environments.

Significance and Claims

The paper claims that Mantis establishes a new state-of-the-art for zero-shot time series classification, demonstrating that:

• Task-Specific Pre-training Matters: Models pre-trained with objectives aligned to classification (contrastive loss) and architectures tailored for TSC outperform general-purpose forecasting models.
• Synthetic Data is Viable: Pre-training on synthetic data is a highly sample-efficient strategy that avoids data leakage and enforces robust out-of-distribution generalization.
• Intermediate Layers are Critical: The "scaling laws" of pre-training in time series classification are unlocked at test time by leveraging intermediate-layer representations rather than just the final layer.
• Lightweight Deployment: High performance can be achieved with lightweight models that do not require heavy fine-tuning, facilitating deployment in resource-constrained settings.

The authors emphasize that their work bridges the gap between the success of foundation models in other modalities and the specific needs of time series classification, providing a robust, lightweight, and generalizable solution for diverse application domains.