TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysis

TSPulse is a family of ultra-lightweight, pre-trained time-series models that utilize a novel disentanglement framework to learn complementary temporal, spectral, and semantic views, achieving state-of-the-art zero-shot and fine-tuned performance across diverse diagnostic tasks while outperforming models 10–100 times larger.

The Gist

Imagine you are a doctor trying to diagnose a patient. You have a stethoscope (listening to the heart), an X-ray (seeing the bones), and a blood test (checking chemistry). Each tool gives you a different, crucial piece of the puzzle. If you only looked at the X-ray, you might miss a heart rhythm issue. If you only listened to the heart, you might miss a broken bone.

For a long time, AI models analyzing time-series data (like stock prices, weather, or machine sensors) tried to cram all this information into one giant, messy "brain." They would mix the heartbeat, the bone structure, and the blood chemistry into a single, tangled knot of data. This made the AI huge, slow, and bad at diagnosing specific problems without retraining.

Enter TSPulse: The "Swiss Army Knife" Doctor.

The paper introduces TSPulse, a new AI model that changes the game. Here is how it works, broken down into simple concepts:

1. The Problem: The "Tangled Knot"

Existing AI models are like a student who tries to memorize a whole textbook by reading every word at once. They get the general idea but struggle to find specific facts quickly. They are also giants—some are 100 times larger than TSPulse, requiring massive supercomputers to run. This makes them useless for small devices (like a smart thermostat or a factory sensor) that need instant answers.

2. The Solution: "Disentangled" Glasses

TSPulse is different because it wears three pairs of special glasses simultaneously, rather than trying to see everything with one blurry eye:

• The Time Glasses: It looks at the data exactly as it happens second-by-second (great for spotting sudden spikes or glitches).
• The Frequency Glasses: It looks at the "rhythm" or "beat" of the data (great for spotting repeating patterns or cycles).
• The Semantic Glasses: It looks at the "big picture" meaning (great for understanding the overall story, like "this machine is overheating").

Instead of mixing these views into a knot, TSPulse keeps them disentangled (separate but organized). This means when a task comes in, the AI can instantly grab the exact pair of glasses it needs without wading through the others.

3. The Training: Learning from "Broken" Data

To teach TSPulse, the researchers didn't just show it perfect data. They used a clever trick called Hybrid Masking.

• Imagine you are teaching a child to read. Instead of just covering up whole words (block masking), you cover up whole words and random letters inside words (hybrid masking).
• This forces the child to learn how to guess missing letters and missing words.
• Because TSPulse is trained on this messy, "broken" data, it becomes incredibly robust. It can fill in missing sensor data or spot anomalies even when the data is noisy or incomplete, which is exactly what happens in the real world.

4. The Size: A Tiny Brain, A Giant Impact

TSPulse is ultra-lightweight. It has only 1 million parameters.

• Analogy: If other top AI models are like a Cruise Ship (huge, powerful, but slow and expensive to run), TSPulse is a Speedboat.
• Despite being tiny, it is faster and often more accurate than the Cruise Ships. It can run on a standard laptop or even a CPU without needing a powerful graphics card (GPU). This means it can be deployed anywhere, from a cloud server to a tiny microchip on a factory machine.

5. The Results: Winning Everywhere

The paper tested TSPulse on four major tasks, and it crushed the competition:

• Anomaly Detection (Spotting the weird stuff): It found errors in machine data 20% better than the best existing models.
• Imputation (Filling in the blanks): When data is missing, it guessed the missing values 50% better than others.
• Classification (Sorting things): It sorted different types of time-series data 5–16% better.
• Similarity Search (Finding look-alikes): It found patterns that looked similar 25% better, even if the data was shifted in time or had noise.

The Bottom Line

TSPulse is a breakthrough because it proves you don't need a massive, bloated AI to do great work. By organizing its knowledge into clear, separate categories (time, frequency, and meaning) and training it on messy, realistic data, it creates a model that is:

1. Fast: Runs instantly on cheap hardware.
2. Smart: Understands data from multiple angles.
3. Ready: Can be used immediately (Zero-Shot) without needing to be retrained for every new job.

It's the difference between carrying a library of books to solve a problem versus having a smart, organized assistant who knows exactly which page to open, instantly.

Technical Summary

1. Problem Statement

Time-series (TS) diagnostic tasks (anomaly detection, imputation, classification, similarity search) require understanding signals across multiple representation spaces (time vs. frequency) and abstraction levels (local patterns vs. global semantics). Existing pre-trained time-series models face three critical limitations:

• Entangled Representations: Current models often compress heterogeneous signals into a single large embedding, making it difficult for downstream tasks to selectively access specific information (e.g., spectral vs. temporal features).
• Scalability and Latency: State-of-the-art (SOTA) models are often massive (100M–300M+ parameters), hindering real-time, low-latency deployment, especially in CPU-only or resource-constrained environments.
• Masking Bias: Most models rely on fixed-length block masking during pre-training. This fails to generalize to real-world missingness patterns, which are often irregular (point-level or partial patch-level), leading to poor performance in imputation tasks.

2. Methodology: TSPulse Architecture

TSPulse is a family of ultra-light pre-trained models (~1M parameters) designed to learn disentangled representations across time, frequency, and semantic spaces.

A. Core Architecture

• Backbone: Replaces heavy Transformers with TSMixer, a lightweight MLP-Mixer architecture that performs feature mixing across patches and channels with significantly reduced compute.
• Dual-Space Input: The model processes input in both the Time Domain and Frequency Domain (via FFT).
  • Input: Multivariate time series $X$ is masked and normalized (RevIN).
  • FFT Extraction: The masked time-series is directly fed into an FFT to generate frequency-domain features, ensuring the same data is masked in both domains to prevent information leakage.
• Disentangled Embeddings: The model produces three distinct types of embeddings via multi-output heads:
  1. Temporal Embeddings: Fine-grained time analysis.
  2. Spectral (FFT) Embeddings: Frequency-aware fidelity.
  3. Semantic Embeddings: High-level structural abstractions (derived from "Register Tokens").

B. Pre-Training Framework

• Hybrid Masking Strategy: Unlike prior work using fixed block masking, TSPulse employs a hybrid masking scheme. It randomly masks both full patches (blocks) and partial patches (points) within a sample. This mimics real-world irregular missingness, reducing overfitting and improving robustness.
• Multi-Objective Loss: The model is trained to reconstruct the input in multiple ways:
  • Full Reconstruction: Reconstructs the time-domain signal and the frequency spectrum.
  • Semantic Reconstruction: Predicts a global frequency signature (log-magnitude spectrum) using the register tokens, capturing high-level patterns.
  • Auxiliary Prediction: A lightweight next-point prediction head injects temporal cues into semantic embeddings without full forecasting overhead.

C. Downstream Workflows & Post-Hoc Fusers

TSPulse supports rapid adaptation via lightweight fusers tailored to specific tasks:

• Classification (TSLens): A module that replaces standard pooling with a learned mechanism to selectively attend to and weight features across fine-grained patches and high-level register tokens.
• Anomaly Detection (Multi-Head Triangulation): Leverages the three distinct views (Time, FFT, Prediction). Anomaly scores are computed from deviations in each view. A "triangulation" strategy (e.g., selecting the best head or ensembling) adapts to specific anomaly types (spikes, periodicity breaks, or trend shifts).
• Imputation: Directly utilizes the hybrid masking pre-training to handle irregular missing data in a zero-shot manner.
• Similarity Search: Uses the robust Semantic Embeddings (Register Tokens) which are invariant to time shifts, magnitude changes, and noise.

3. Key Contributions

1. Compact & Versatile Models: Introduction of a 1M parameter pre-trained model family that supports rapid zero-shot transfer and fast fine-tuning for four diagnostic tasks.
2. Architectural Novelties:
   • Disentangled Masked Reconstruction: Explicitly learning temporal, spectral, and semantic embeddings within a unified framework.
   • Post-Hoc Fusers: Task-specific lightweight modules (MHT for AD, TSLens for Classification) that exploit these disentangled views.
   • Hybrid Masking: A pre-training strategy that mitigates bias toward fixed-length gaps, improving generalization to irregular real-world data.
3. Emergent Properties: Sensitivity analyses confirm that the learned embeddings are truly disentangled:
   • Temporal embeddings are sensitive to phase shifts (good for precise timing).
   • Spectral embeddings are invariant to phase but capture frequency.
   • Semantic embeddings are robust to missing data, noise, and magnitude scaling (ideal for retrieval).
4. Efficiency: The model runs efficiently on CPUs, enabling GPU-free deployment with near-instant inference.

4. Experimental Results

TSPulse was evaluated on over 75 datasets across four tasks, outperforming models 10–100x larger:

• Anomaly Detection (TSB-AD Leaderboard):
  • Achieved +20% improvement on the VUS-PR metric over SOTA.
  • Ranked #1 in both univariate and multivariate settings, outperforming models like MOMENT and Chronos despite being ~100x smaller.
• Classification (UEA Archive):
  • Achieved 5–16% higher accuracy than pre-trained baselines (VQShape, UniTS, Moment) and 5–12% over supervised baselines.
• Imputation:
  • +50% gain in zero-shot performance over MOMENT and UniTS under hybrid (irregular) masking.
  • Demonstrated that hybrid pre-training is critical; removing it caused a 79% performance drop in irregular missingness scenarios.
• Similarity Search:
  • +25% improvement in family-level match and +40% in fine-grained match over MOMENT.
  • Embeddings are 2x smaller and enable 10–100x faster CPU inference compared to baselines.

5. Significance

TSPulse represents a paradigm shift in time-series foundation models by prioritizing efficiency and disentanglement over sheer scale.

• Deployment Ready: It proves that ultra-light models (1M params) can match or exceed the performance of massive foundation models, making advanced time-series analysis accessible for edge devices, CPU-only servers, and real-time applications.
• Robustness: The hybrid masking strategy addresses a fundamental flaw in existing pre-training methods, making models significantly more robust to the irregular missing data common in industrial and IoT settings.
• Interpretability & Flexibility: By disentangling representations, TSPulse allows practitioners to select the most appropriate view (time, frequency, or semantic) for a specific task, offering a level of control and adaptability previously unavailable in monolithic embeddings.

The code and models are publicly available at Hugging Face (ibm-granite/granite-timeseries-tspulse-r1).