Technical Summary: Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data

Problem Definition

Large Language Models (LLMs) are increasingly deployed in domains such as healthcare, finance, and transportation, where time-series data is fundamental. However, current LLMs lack the ability to perform reasoning that simultaneously integrates time-series data and corresponding textual content. Existing approaches often convert time-series into text formats or embed them into model weights, which frequently compromises the model's inherent natural language understanding (NLU) and reasoning capabilities. Furthermore, the field suffers from a scarcity of multimodal training data and a lack of large-scale benchmarks for evaluating time-series reasoning.

Methodology

The authors propose Chat-TS, a framework designed to enable LLMs to reason over time-series and textual data without degrading their core language capabilities. The methodology consists of three primary components:

1. Vocabulary Expansion via Discrete Tokenization

Instead of using connectors to map time-series representations to text embeddings, Chat-TS expands the LLM's vocabulary to include time-series tokens.

• Tokenizer: A lightweight discrete tokenizer is introduced to convert continuous numerical time-series values into discrete tokens. It quantizes normalized time-series ranges $[-s, s]$ into $K-1$ bins (with $K=8192$) and uses a special token to mark channel ends.
• Advantage: This approach allows for near-perfect reconstruction of time-series data (particularly for sequences under 1,000 points) without requiring the training of complex encoder-decoder architectures that might suffer from out-of-distribution issues.

2. Training Strategy

The framework employs a two-phase training strategy to preserve NLU capabilities while acquiring time-series reasoning skills:

• Phase 1 (Pre-training): The model is pre-trained on time-series tokens. Two initialization methods are explored: mean initialization (setting new embeddings to the average of existing text tokens) and time-series pre-training (unfreezing only the embedding and final linear layers).
• Phase 2 (Instruction Tuning): A dual-dataset strategy is used. The model is fine-tuned on a combination of:
  • TS-Instruct: A multimodal dataset pairing time-series data with text instructions.
  • Open-Orca: A large corpus of pure natural language instruction data.
    This interleaving ensures the model retains its general language reasoning strengths while learning to follow instructions involving time-series.

3. Dataset Curation

To address data scarcity, the authors contribute three new datasets:

• TS Instruct Training Dataset: A diverse multimodal dataset generated using GPT-4o-mini, pairing real-world time-series (from LOTSA and Time-Series Classification Archive) with synthetic conversational instructions covering reasoning, classification, decision-making, and mathematical analysis.
• TS Instruct QA Gold Benchmark: A human-validated set of 1,056 multiple-choice questions designed to evaluate multimodal reasoning capabilities.
• TS Instruct Quantitative Probing Set: A subset for quantitative evaluation including math and decision-making tasks.

Key Contributions

1. New Datasets: The release of the TS Instruct Training Dataset and the TS Instruct QA Gold Benchmark fills a critical gap in the literature regarding multimodal time-series training and evaluation.
2. Architecture: A novel approach that integrates time-series tokens directly into the LLM vocabulary, eliminating the need for intermediary connectors and preserving the model's original text-generation and reasoning capabilities.
3. Performance: The proposed method achieves state-of-the-art performance in multimodal reasoning tasks while maintaining strong natural language proficiency.

Experimental Results

Experiments were conducted using the Llama 3.1-8B model as a base.

• Time-Series Reasoning: On the TS Instruct QA Gold Benchmark, Chat-TS achieved a score of 67.22%, outperforming the base Llama 3.1-8B (54.22%) and other baselines like Phi-3-medium-4k (64.64%). This represents an average improvement of approximately 13% over existing state-of-the-art baselines.
• Generalization: When tested on the MCQ2TS dataset (a synthetic dataset with counter-factual reasoning tasks distinct from the training data), Chat-TS improved from 36.5% (base model) to 47.6%, demonstrating that performance gains are not due to dataset contamination.
• Preservation of NLU: Ablation studies on MMLU-Pro, Big-Bench-Hard, and GPQA benchmarks showed that all Chat-TS variants maintained performance within ±2% of the base Llama 3.1-8B model. This confirms that integrating time-series reasoning does not degrade the model's natural language capabilities.
• Ablation Analysis: Models trained solely on time-series data (without interleaved text) struggled with instruction following. The best performance was achieved by models trained with a mix of Open-Orca text data and TS-Instruct multimodal data (PreOrcaTS), highlighting the necessity of interleaving modalities.

Significance and Limitations

The paper claims that Chat-TS establishes a strong baseline for multimodal reasoning, demonstrating that LLMs can be effectively augmented for time-series analysis without sacrificing their foundational language skills. The work provides a unified framework for handling text and time-series, supported by open-source models, datasets, and code.

The authors acknowledge specific limitations:

• Time-Series Generation: The models currently struggle with accurate time-series forecasting and generation, limiting applications in areas like weather or financial prediction.
• Zero-Shot Classification: Performance on zero-shot time-series classification is weak, often resulting in guessing or repetition.
• Scaling: The experiments were limited to an 8B parameter model for accessibility; the authors suggest that scaling up data volume and model size would likely yield further improvements.

The paper concludes that while Chat-TS advances the state of the art in time-series reasoning, future work must address generation capabilities and classification robustness to fully realize the potential of multimodal LLMs in time-series domains.