InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement

Technical Summary: InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement

Problem Statement
The paper addresses two inherent limitations in existing time series classification (TSC) methods, which predominantly follow a discriminative modeling paradigm that maps input sequences to one-hot encoded class labels. First, these methods struggle to effectively exploit contextual features, which are often provided in textual form (e.g., patient demographics in medical diagnosis). Second, the use of one-hot label distributions fails to capture the semantic relationships among classes (e.g., the semantic similarity between "walking" and "jogging" in human activity recognition). Furthermore, while initial attempts to use language models (LMs) for TSC can model explicit contextual features, LMs generally lack the inductive bias to capture implicit features—such as latent temporal dynamics and structural patterns—that are not directly observable from raw time series or explicit contextual inputs.

Methodology
The authors propose a two-stage framework, evolving from InstructTime to InstructTime++, which reformulates TSC as a multimodal generative task.

1. InstructTime (Base Framework):

   • Discrete Prompt Construction: To bridge the modality gap between continuous numerical sequences and discrete textual tokens, the method employs a time series discretization module based on Vector-Quantized (VQ) networks. Raw time series are partitioned into patches, encoded, and mapped to discrete temporal tokens from a learned codebook.
   • Multimodal Input Integration: The framework treats continuous numerical sequences (as discrete tokens), discrete textual features (contextual information), and task-specific instructions as multimodal inputs. Class labels are represented as textual outputs.
   • Alignment and Training: An alignment projection layer (MLP) and a generative self-supervised pre-training strategy are used to align cross-modal representations. The model is then fine-tuned with task-specific instructions to enhance multimodal reasoning capabilities.

2. InstructTime++ (Extended Framework):

   • Implicit Feature Extraction: To compensate for the limited inductive bias of LMs regarding temporal structures, InstructTime++ introduces specialized toolkits to mine implicit features from multiple views of the raw data. This is instantiated from two complementary perspectives:
     • Statistical Feature Extraction: A rule-driven module computes deterministic statistical properties (e.g., mean, variance, entropy, trend, periodicity) to characterize distributional and structural complexity.
     • Visual Feature Extraction: A vision-language model (VLM)-based image captioning approach extracts perception-driven structural patterns by grounding time series visualizations.
   • Textual Grounding: The discovered implicit features are translated into textual descriptions. These descriptions are integrated into the prompt template alongside explicit contextual features and task instructions, allowing the LM to reason over both explicit and implicit information within a unified textual context.

Key Contributions

• Paradigm Shift: Reformulates TSC from discriminative mapping to generative multimodal reasoning, enabling the natural incorporation of contextual features and the modeling of semantic class relationships.
• Modality Bridging: Introduces a VQ-based discretization strategy and alignment projection to alleviate inconsistency between numerical and textual modalities.
• Implicit Feature Enhancement: Proposes InstructTime++, which augments the generative framework with implicit feature modeling via statistical and visual toolkits, addressing the lack of temporal inductive bias in standard LMs.
• Unified Framework: Demonstrates that implicit features can be seamlessly integrated into the generative pipeline by translating them into textual descriptions, maintaining modality consistency without modifying the underlying LM architecture.

Results
Extensive experiments were conducted on multiple benchmark datasets, including EEG, ECG, Human Activity Recognition (HAR), Fault Detection (FD), Right Whale Calls (RWC), Epilepsy (EP), and Spoken Arabic Digits (SAD).

• Performance: InstructTime++ (specifically the adapted variant, InstructTime++ Adapt) achieved superior performance across most datasets compared to baselines including Transformer variants, CNNs, self-supervised models (TimeMAE, TS-TCC), and GPT-based classifiers.
• Robustness: The method demonstrated consistent performance across diverse temporal structures, outperforming reconstruction-driven or purely discriminative methods on datasets with high semantic ambiguity (e.g., EEG, ECG).
• Ablation Studies:
  • Cross-domain pre-training significantly improved generalization and semantic alignment.
  • Instruction text was crucial for modality alignment; removing it led to marked performance degradation.
  • Both visual and statistical implicit features contributed complementary information; removing either component resulted in performance drops, with visual features being more critical for complex temporal patterns (EEG, HAR) and statistical features for domains with strong numerical regularities (ECG, RWC).
  • Medium-sized backbones (e.g., Qwen3-0.6B) offered a favorable trade-off between performance and efficiency, outperforming both smaller and larger models in many settings.

Significance
The paper claims that InstructTime++ paves the way for "agentic time series classification," where models can autonomously leverage external tools and iterative reasoning. By effectively modeling both explicit contextual cues and implicit temporal/structural features within a unified generative framework, the approach reduces the representation gap between numerical sequences and linguistic semantics, offering a robust and state-of-the-art solution for time series classification. The framework is described as task-agnostic, with potential extensions to forecasting and anomaly detection.