Thoth: Mid-Training Bridges LLMs to Time Series Understanding

This paper introduces Thoth, the first family of mid-trained Large Language Models that leverages the high-quality "Book-of-Thoth" corpus to bridge natural language and time series data, significantly enhancing temporal reasoning and performance on knowledge-intensive benchmarks compared to existing models.

The Gist

Imagine you have a brilliant, world-class detective (a Large Language Model, or LLM) who has read every book, newspaper, and website on Earth. This detective is amazing at solving mysteries, writing stories, and understanding human conversation.

However, there's one thing this detective is terrible at: reading the rhythm of time.

If you show them a graph of stock prices, a patient's heart rate, or the temperature in a city over a year, they might see a bunch of squiggly lines. They can't "feel" the pattern. They don't understand that a sudden spike means a crisis, or that a slow, steady rise means a trend. They are like a music critic who has read every music theory book but has never actually heard a song.

This paper introduces Thoth, a new kind of AI detective designed to fix this problem.

Here is the story of how they did it, explained simply:

1. The Problem: The "Language" Gap

Most AI models are trained on text (words). Time series data (numbers changing over time) is a different language entirely.

• The Old Way: Researchers tried to teach the detective specific tricks for specific cases. "If you see a stock drop, say 'sell'." "If you see a fever, say 'call a doctor'."
• The Flaw: This is like teaching a student to memorize answers for a specific test. If the test changes slightly, they fail. They don't truly understand the concept of time.

2. The Solution: "Mid-Training" (The Bridge)

The authors realized that instead of just teaching the detective specific tricks at the end, they needed to give them a specialized education in the middle of their training.

Think of it like this:

• Pre-Training: The detective reads the whole library (general knowledge).
• Mid-Training (The Thoth Bridge): Before they start solving specific cases, they spend a year working as a meteorologist and a stockbroker. They study thousands of weather charts and financial graphs, learning to describe them in words and predict them from words.
• Post-Training: Now, when they go back to being a general detective, they have a superpower: they can look at any timeline and instantly understand the story it tells.

3. The Secret Ingredient: The "Book of Thoth"

To teach the AI this new language, the researchers built a massive textbook called Book-of-Thoth.

• What's in it? It's not just numbers. It's a two-way dictionary.
  • Time-to-Text: They took thousands of graphs and asked an AI to write a story about them. "This line goes up slowly, then crashes hard."
  • Text-to-Time: They took a description like "A stock that rises steadily for a week and then drops" and generated the exact graph to match it.
• Why is this cool? It forces the AI to learn that a "sudden drop" in numbers is the same concept as a "crash" in words. It bridges the gap between the visual world of data and the logical world of language.

4. The New Benchmark: "KnoTS"

To prove their detective is actually smarter, they created a new test called KnoTS (Knowledge-intensive Time Series).

• Old Tests: "Here is a graph. Is it going up or down?" (Too easy).
• KnoTS: "Here is a graph of CO2 levels in soil. We know that rain blocks gas. The graph shows a weird dip after a storm. Why did this happen?"
• The Result: Thoth didn't just guess; it used its new "mid-training" to combine the graph with real-world knowledge (like how rain affects soil) to solve the mystery.

5. The Results: A Super Detective

When they tested Thoth:

• It beat the biggest, smartest AI models currently available (like GPT-4 and Gemini) at understanding time-based data.
• It needed very little extra training to become an expert at specific jobs (like predicting energy usage or detecting heart anomalies).
• It didn't forget how to speak human language; it just added a new superpower on top of its existing skills.

The Big Picture

This paper is about teaching AI to feel the flow of time.

Before, AI was like a person who could read a map but couldn't drive a car. With Thoth and the Book-of-Thoth, they have given the AI a driver's license. It can now look at the road (the data), understand the traffic patterns (the trends), and make smart decisions about where to go next.

This is a huge step forward for using AI in the real world, where almost everything—from your bank account to your health—is a story written in numbers over time.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have achieved remarkable success in general reasoning and natural language processing, they struggle to understand and reason about time series data. Real-world decision-making in domains like finance, healthcare, and transportation relies heavily on temporal dynamics.

• The Gap: LLMs are fundamentally built on language modeling and lack explicit mechanisms to capture fine-grained temporal dependencies and complex dynamics inherent in time series.
• Limitations of Current Approaches:
  • Supervised Fine-Tuning (SFT): Existing methods rely on task-specific labeled datasets (e.g., forecasting or classification). These approaches suffer from poor generalization across diverse temporal patterns and domains, require substantial labeled data, and often lead to "catastrophic forgetting" of general reasoning capabilities.
  • Lack of Intermediate Stage: There is a missing intermediate training phase between large-scale pre-training and task-specific post-training that can align general-purpose LLMs with time series data without sacrificing their foundational capabilities.

2. Methodology

The authors propose Thoth, the first family of mid-trained LLMs designed for general-purpose time series understanding. The methodology consists of three core components:

A. Book-of-Thoth: A Mid-Training Corpus

To enable task- and domain-agnostic alignment between time series and natural language, the authors constructed Book-of-Thoth, a large-scale, time-series-centric corpus (26.6M tokens). It is designed to avoid the narrow focus of existing datasets by featuring two core bidirectional tasks:

1. Time Series-to-Text (TS-to-Text):
   • Generation: Synthetic time series are generated using KernelSynth (a Gaussian-process-based method) to ensure diverse temporal patterns (trends, seasonality, noise).
   • Annotation: GPT-5.2 generates natural language descriptions from multiple perspectives. These include Structured descriptions (detailed analysis of trends, seasonality, peaks) and Unstructured descriptions (concise captions).
2. Text-to-Time Series (Text-to-TS):
   • The process is reversed: structured textual descriptions are paired with their corresponding time series to train the model to generate numerical sequences from text.
3. Knowledge Integration: The corpus includes open-source time series knowledge from authoritative textbooks (e.g., Forecasting: Principles and Practice) to ground the model in theoretical concepts.
4. Mitigating Forgetting: To prevent "vertical forgetting" (loss of general capabilities), the training corpus mixes a small proportion of general-purpose pre-training data (C4, No Robots) with the time-series-specific data.

B. Thoth Model Architecture

• Base Models: Built upon the Qwen3 architecture (specifically Qwen3-30B-A3B-Instruct and Qwen3-8B).
• Training Strategy: Thoth undergoes mid-training (full-parameter fine-tuning) on Book-of-Thoth. This stage acts as a "warm-up," expanding the model's knowledge coverage with temporal data while preserving its general reasoning skills.
• Technical Specs: Trained using DeepSpeed ZeRO-3, AdamW optimizer, cosine learning rate schedule, and a maximum sequence length of 4096 tokens.

C. KnoTS: A New Benchmark

To evaluate advanced reasoning capabilities, the authors introduce KnoTS (Knowledge-intensive Time Series QA).

• Design: Unlike previous benchmarks that rely on template-driven synthetic data, KnoTS focuses on joint reasoning over temporal patterns and domain-specific knowledge (e.g., Finance, Healthcare, Environment).
• Tasks: It includes reasoning tasks (inferring patterns based on domain rules) and decision-making tasks (predicting outcomes based on data + context).
• Scale: 300 carefully curated QA pairs with human-verified factual accuracy.

3. Key Contributions

1. Mid-Training Paradigm for Time Series: Proposes mid-training as a critical intermediate stage to bridge general LLMs and time series understanding, addressing the limitations of direct SFT.
2. Book-of-Thoth Corpus: Automatically constructs a large-scale, bidirectional (TS-to-Text and Text-to-TS) corpus that aligns temporal data with natural language, featuring diverse structural and semantic descriptions.
3. Thoth Model Family: Releases the first mid-trained LLMs with general-purpose time series understanding, demonstrating strong few-shot and fine-tuned capabilities.
4. KnoTS Benchmark: Introduces a rigorous benchmark for evaluating knowledge-intensive time series reasoning, moving beyond simple pattern recognition to complex, context-aware decision-making.

4. Experimental Results

Extensive experiments were conducted on ChatTime, Time-MQA, and the new KnoTS benchmark.

• Performance on Public Benchmarks:
  • Thoth-30B-A3B achieves performance comparable to, and in some cases exceeding, the massive Qwen3-235B model on time series QA tasks.
  • Thoth-8B significantly outperforms its base model (Qwen3-8B) and matches or exceeds open-source models with ~30B parameters.
  • Thoth models consistently outperform task-specific time series models (e.g., ChatTS, Time-MQA baselines) which often suffer from cross-task imbalance.
• KnoTS Performance:
  • Thoth-30B-A3B demonstrates superior reasoning capabilities, outperforming powerful proprietary models (Grok-4.1-Fast, GPT-4o-mini) and VLMs.
  • It successfully integrates domain knowledge (e.g., "barometric pump" effects in soil gas monitoring) with time series data to solve complex reasoning problems.
• Few-Shot and Fine-Tuning Efficiency:
  • Data Scarcity: When fine-tuned on only 5% of downstream data, Thoth significantly outperforms baselines.
  • Stability: Unlike base models which suffer from capability trade-offs (losing open-ended reasoning when fine-tuned on specific tasks), Thoth maintains stable performance improvements across diverse tasks.
• Ablation Studies:
  • Removing the TS-to-Text component causes the most significant performance drop, highlighting the importance of understanding temporal patterns via language.
  • Text-to-TS provides additional value, though less than TS-to-Text, by reinforcing the alignment between text and numerical generation.
• Scaling: Performance improves steadily as the size of the mid-training dataset increases, confirming the scalability of the approach.

5. Significance and Impact

• Bridging the Gap: Thoth provides a practical solution to the "time series gap" in LLMs, enabling them to reason over temporally grounded information without requiring massive task-specific datasets.
• Generalization: By using mid-training, the model acquires transferable temporal representations, allowing it to generalize across diverse domains (finance, health, environment) rather than being locked into a single task.
• Efficiency: The approach demonstrates that high-quality mid-training can yield models that outperform much larger base models and specialized fine-tuned models, offering a more efficient path to specialized AI.
• Future Direction: The paper establishes mid-training as a vital "warm-up" stage for downstream time series tasks, suggesting that future research should focus on refining this intermediate stage and exploring reinforcement learning for post-training alignment.

In conclusion, Thoth represents a paradigm shift from task-specific fine-tuning to domain-aligned mid-training, successfully equipping general-purpose LLMs with the foundational grasp of temporal patterns necessary for real-world decision-making.