🤖 machine learning

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

Chronicle is a compact, 324M-parameter decoder-only transformer trained from scratch on both natural language and time series within a unified architecture, achieving competitive performance in both domains and superior multimodal forecasting capabilities without relying on post-hoc adaptation of pretrained language models.

Original authors: Paul Quinlan, Jeremy Levasseur, Qingguo Li, Xiaodan Zhu

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Quinlan, Jeremy Levasseur, Qingguo Li, Xiaodan Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two very different types of experts in a room.

The first expert is a Time Series Analyst. They are a wizard at looking at numbers that change over time—like stock prices, weather patterns, or heart rate monitors. They can predict what happens next just by looking at the shape of the lines. However, they are mute; they can't read the notes, news reports, or descriptions that often accompany those numbers.

The second expert is a Language Scholar. They are brilliant at reading books, understanding stories, and answering questions. But if you hand them a spreadsheet of raw numbers, they are completely lost. They don't speak the language of "data."

For a long time, scientists tried to make these two experts work together by taking the Language Scholar, giving them a crash course in numbers, and hoping they could understand both. The paper argues this is like trying to teach a fish to climb a tree by forcing it to wear boots. The Language Scholar's brain was built for words, not for the rhythm of time, so the results were clunky and inefficient.

Enter "Chronicle."

The authors of this paper built a completely new kind of expert from scratch. Instead of taking an existing language expert and trying to teach them math, they built a single, compact brain (a 324-million-parameter model) that learned both languages simultaneously from day one.

Here is how they did it, using some simple analogies:

1. The "Shared Classroom" Approach

Most previous models tried to bolt a time-series module onto a language model. It's like building a house and then trying to add a swimming pool by gluing it onto the roof.

Chronicle is different. It's like building a single, shared classroom where the students (the data) are either reading a book (text) or watching a clock tick (time series).

The Curriculum: For most of the training, the students study in separate groups. 92% of the time, they read books. 8% of the time, they study the clock.
The Magic: Even though they study separately, they sit at the same desks and use the same notebooks (the "shared parameters"). By the time they graduate, the "Language" part of the brain has learned the rhythm of time, and the "Time" part has learned the nuance of language, all without ever needing to be glued together.

2. The "Patchwork" Quilt

To teach the model about time, the authors didn't feed it raw numbers one by one. Instead, they cut the time series into little squares, like patches of a quilt.

Imagine a long strip of fabric representing a year of temperature data.
They cut this into 32-day chunks (patches).
Each patch is turned into a single "token" (a word-like unit) that the model can understand.
This allows the model to see the "shape" of the data (is it going up? is it spiking?) just as easily as it sees a sentence.

3. The "Two-Stage" Training

The training happened in two phases:

Stage 1 (The Basics): The model learned to read books and predict the next patch of the quilt separately. It became very good at both tasks on its own.
Stage 2 (The Conversation): They introduced a small amount of "mixed" data. Imagine showing the model a picture of a storm cloud (the quilt patch) and the sentence "It is raining" (the text) at the same time. This taught the model how the two modalities relate to each other, allowing it to understand that a specific pattern in the numbers means "rain."

What Did They Prove?

The paper claims that this new "Chronicle" model is a powerhouse for three reasons:

It's a Bilingual Master: It understands text just as well as other small, specialized language models (like Gemma-3 or LLaMA) of the same size. It didn't lose its ability to read just because it learned math.
It's a Time-Telling Pro: It predicts future numbers better than almost every other model that only looks at numbers, even though it spends most of its training time reading text.
It's the Best at Mixing Them: When you give it both text and numbers together (like a news report about a stock crash plus the stock chart), it predicts the future better than any previous model that tried to combine the two.

The Bottom Line

The paper concludes that you don't need to force a language model to learn time series, or a time-series model to learn language. Instead, you can build a unified foundation that learns both from the ground up.

Think of it like a universal translator who didn't just learn a second language by translating word-for-word, but by growing up speaking both languages at the same time. They understand the feeling and context of both, making them far more accurate than someone who just learned the rules of grammar for a second language later in life.

Note: The paper focuses strictly on the model's ability to understand, classify, and forecast data. It does not claim the model is currently being used in hospitals, financial trading floors, or weather stations, nor does it predict future clinical applications. It is a foundational research step proving that this specific way of building AI works better than the old way.

Technical Summary: Chronicle

Problem Statement

Real-world time series data is inherently multimodal, accompanied by textual metadata, descriptions, news, and reports. However, existing time series foundation models (TSFMs) process numerical sequences in isolation, lacking mechanisms to incorporate this textual context. Conversely, multimodal models that attempt to bridge language and time series suffer from three systematic limitations:

Post-hoc Adaptation: They all start from a pretrained language model (LLM) and adapt it to time series. Consequently, their internal representations are shaped solely by text, forcing the time series modality to fit a representational space not designed for temporal data.
Evaluation Gaps: These models are rarely evaluated against state-of-the-art unimodal foundation models in either domain (dedicated TSFMs or dedicated LLMs), leaving the necessity of joint training unproven.
Architectural Bloat: Many target conversational reasoning and require massive backbones (8–14B parameters), conflating the goal of learning general-purpose temporal representations with building reasoning agents.

The central question addressed is whether a compact model can learn high-quality representations for both text and time series from scratch without catastrophic interference, and whether joint pretraining is necessary compared to adapting pretrained LLMs.

Methodology

The authors present Chronicle, a 324M-parameter decoder-only transformer trained from scratch on natural language and time series within a single unified architecture.

Architecture

Shared Backbone: Both modalities share the same 16-layer transformer blocks, attention mechanism, and residual stream. Modality-specific components are restricted to input and output interfaces.
Input Representation:
- Text: Tokenized via a 131k-entry BPE vocabulary with tied embeddings to the output head.
- Time Series: Following PatchTST-FM and Chronos-2, series are standardized (using visible values only) and transformed via inverse hyperbolic sine to suppress outliers. They are segmented into non-overlapping patches ( $P=32$ ). Each patch is represented by a 4P-dimensional feature vector concatenating a time ramp, normalized values, a validity mask, and a channel ramp (for multivariate inputs). These are projected to the shared embedding dimension ( $d=1024$ ).
Output Heads:
- Text: A tied language model head with logit soft-capping, optimized via cross-entropy ( $L_{CE}$ ).
- Time Series: A quantile head predicting 21 quantile levels ( $\tau \in [0.05, 0.95]$ ), optimized via masked quantile loss ( $L_{QL}$ ).
Training Objective: The total loss is a weighted sum $L = w_{text}L_{CE} + w_{TS}L_{QL}$ , with weights $w_{text}=1.0$ and $w_{TS}=2.5$ to balance gradient contributions given the scale difference between losses.

Training Strategy

Chronicle employs a two-stage pretraining process:

Stage 1 (Unimodal Pretraining): The model is trained on a 92/8 mix of text-only and time-series-only batches. This ensures the backbone learns robust representations for each modality independently. The text data comes from FineWeb-Edu and Dolmino-mix; time series data comes from GiftEvalPretrain augmented with KernelSynth.
Stage 2 (Cross-Modal Alignment): The context length is extended to 4096. A small fraction (5%) of time series tokens are replaced with interleaved text and time series sequences (from ChatTS and Merrill et al.) to establish explicit cross-modal correspondences. The bulk of the training remains unimodal to preserve modality-specific capabilities.

Key Contributions

Joint Pretraining from Scratch: Chronicle is the first model to learn text and time series end-to-end from random initialization within a single shared transformer backbone, rather than adapting a pretrained LLM.
Rigorous Cross-Domain Evaluation: The model is benchmarked against dedicated foundation models in both domains (scale-matched LLMs for NLU and dedicated TSFMs for forecasting/classification), addressing a longstanding gap in multimodal time series literature.
Strong Frozen-Backbone Performance: Chronicle achieves state-of-the-art results on multimodal tasks without per-dataset retraining of the backbone, demonstrating that a compact model can serve as a general-purpose representation engine.

Results

Language Understanding (NLU)

Evaluated on 19 NLU tasks (DCLM suite), Chronicle (Stage 2) achieves an average accuracy of 0.406, matching Gemma-3-270M-PT (0.406) and outperforming GPT-2 (0.324). Notably, it achieves this despite seeing only ~138B text tokens, roughly 43–87 times fewer than its scale-matched text-only competitors. This validates that devoting ~8% of compute to time series does not cause catastrophic interference.

Zero-Shot Probabilistic Forecasting (GIFT-Eval)

On the GIFT-Eval benchmark (97 tasks), Stage 1 (pure unimodal training) achieves a MASE of 0.978 and CRPS of 0.690, outperforming several supervised baselines (e.g., N-BEATS, DLinear) and statistical methods, though trailing dedicated TSFMs like PatchTST-FM-r1. Stage 2 shows a slight performance drop (MASE 1.053) due to the alignment stage, highlighting a trade-off between pure forecasting performance and multimodal transfer capability.

Downstream Multimodal Tasks

Multimodal Classification (TimeCAP): Chronicle outperforms all MM-TSFlib and FM Fusion baselines. Stage 2 with LoRA ( $r=64$ ) achieves the best macro-F1 (0.613) and AUC (0.757).
Multimodal Forecasting (Time-MMD): Chronicle Stage 2 (Linear Probe) achieves the best overall Normalized MAE (0.514) and average rank (2.56), surpassing the strongest baselines (e.g., BERT+TNet at 0.621).
Time Series Classification (UCR/UEA): On 24 datasets, Chronicle Stage 1 sets a new bar for frozen-embedding classification with 0.736 accuracy and 0.712 F1, outperforming dedicated TSFMs like Moirai-2 (0.714) and TimesFM (0.611).

Technical Nuances

Token Repetition: For short time series tasks (TimeCAP), repeating the time series token block ( $r=64$ ) significantly improves performance by rebalancing the text-to-TS token ratio in the shared sequence.
Channel Awareness: Explicitly encoding channel identity in multivariate inputs improves classification performance on average compared to mean-channel pooling.

Significance and Claims

The paper claims that Chronicle demonstrates that text and time series can share a transformer backbone without catastrophic interference, challenging the prevailing assumption that multimodal time series models must adapt a pretrained LLM.

The authors argue that joint pretraining from scratch is a more direct path to general-purpose temporal representations. While a gap remains between Chronicle and dedicated TSFMs in pure forecasting (attributed to compute allocation and the causal next-patch objective), the model successfully unifies language understanding and temporal representation in a compact 324M parameter architecture. The results suggest that the benefits of multimodality are not merely a function of adding text to a frozen LLM, but can emerge from a unified training paradigm where both modalities shape the representations from the beginning.