Technical Summary: TempoWave – Multi-Wavelet Number Embeddings for Time Series Forecasting

1. Problem Statement

The integration of Large Language Models (LLMs) into time series forecasting faces a fundamental architectural mismatch. LLMs are optimized for discrete token prediction, whereas time series forecasting requires the precise modeling of continuous numerical values. Standard language-oriented tokenization methods (e.g., breaking "2026" into "20" and "26") fragment numbers into sub-tokens that are not tied to their magnitude. This fragmentation breaks ordinal relations, obscures the continuity intrinsic to temporal processes, and introduces spurious similarities between numerically distant values. Consequently, the translation layer between real-valued sequences and discrete tokens becomes a principal bottleneck, often leading to unreliable numerical behavior even when the model captures high-level temporal patterns.

Existing attempts to bridge this gap—such as specialized foundation models, agentic systems coupling LLMs with tools, or input adaptations like patching and quantization—often trade away numerical faithfulness, blur fine-grained variations, or rely on external components that reduce end-to-end differentiability. There remains a persistent gap in representing continuous numerical values within the standard transformer input space in a principled, information-preserving manner.

2. Methodology: TempoWave

The authors propose TempoWave (Multi-Wavelet Number Embedding), a plug-and-play temporal wavelet digit interface designed to map scalar observations into dense vector embeddings constructed from multi-wavelet, multi-scale coefficients.

Core Mechanism

• Digit-Level Tokenization: Instead of tokenizing entire numbers, the input time series values are rendered into fixed-precision strings (e.g., $V.FFFF$). Each digit ($d \in \{0, \dots, 9\}$) is treated as an individual token.
• Wavelet-Based Embedding Construction:
  • Signal Representation: Each digit $d$ is normalized to $\tilde{d} \in [0, 1]$ and represented as a discrete impulse (Kronecker delta) on a fixed grid.
  • Multi-Wavelet Coefficients: The digit signal is projected onto a predefined family of wavelets $\Psi$ (e.g., Haar, db4, Morlet) across multiple scales $S$. The wavelet coefficient for a digit $d$ at wavelet $\psi_i$ and scale $s_j$ is computed as the inner product of the digit signal and the sampled wavelet: $W_{\psi_i, s_j}(d) = \langle f_d, \psi_{i, s_j} \rangle$.
  • Embedding Generation: These coefficients are concatenated into a feature vector $\phi(d)$ and mapped to the LLM's embedding dimension $D$ via a fixed alignment function $g(\cdot)$. This produces a digit embedding $E(d) \in \mathbb{R}^D$.
• Integration: The standard token embeddings for digit positions in the input sequence are overridden by these computed $E(d)$ vectors, while text and context tokens retain standard embeddings. The LLM backbone remains unchanged and is trained via Supervised Fine-Tuning (SFT) to predict future numeric tokens.

Design Principles

• Structural Faithfulness: By encoding digits through structured multi-scale patterns rather than raw scalar magnitudes, TempoWave ensures that numerical identity is determined by relative patterns. This makes the embeddings robust to normalization layers (e.g., LayerNorm, RMSNorm) commonly used in transformers, which typically rescale and re-center embeddings.
• Monotonic Neighborhood Consistency: The multi-wavelet construction induces a finite digit codebook with improved separability. Theoretical analysis suggests that concatenating coefficients across wavelet-scale pairs increases pairwise distances between digit embeddings, ensuring robust nearest-neighbor recoverability and preserving numerical ordering.

3. Key Contributions

1. A Wavelet-Based Numeric Interface: TempoWave enables the direct use of standard LLM backbones for numerical forecasting without relying on language tokenization of numbers, effectively bridging the modality gap.
2. Faithful and Stable Representations: The method establishes properties related to numerical faithfulness and stability, including improved separability across values and robustness to common normalization operations.
3. Comprehensive Evaluation with Diagnostics: The authors provide extensive experiments across diverse benchmarks and introduce diagnostic analyses (e.g., probing monotonic neighborhood consistency) to explain how the embedding interface reshapes numerical neighborhoods within the model.

4. Experimental Results

The authors evaluated TempoWave on five context-enriched forecasting benchmarks:

• Datasets: AUL (Australia electricity), BIT (Bitcoin), MSPG (Melbourne solar power), PTF (Paris traffic flow), and LEU (London electricity usage). These datasets pair historical time series with textual or event-based context (e.g., news, weather, dates).
• Baselines: Comparisons included standard LLM tokenization, alternative embedding interfaces (e.g., Fourier-based FoNE), specialized time-series models (DLinear, Autoformer, TimesNet), and time-series foundation models (Chronos, Moirai).
• Performance:
  • TempoWave achieved a new state-of-the-art (SOTA) on 7 out of 10 reported metrics (RMSE and MAE) across the five datasets.
  • It ranked in the top-2 for all metrics.
  • Relative to the previous best method, TempoWave yielded an average 7.0% relative improvement in MAE, with significant gains on LEU (14.4%) and AUL (11.2%).
  • It demonstrated superior robustness on non-stationary, event-driven datasets (e.g., Bitcoin) compared to Fourier-based interfaces.
• Ablation Studies:
  • Context: Incorporating rich contextual information (Catch22 features, situational context) consistently improved performance, with the full-context setting yielding the best results.
  • Embedding Alignment: Analysis of token ID difference distributions showed that TempoWave produces a smooth, unimodal distribution centered around zero, indicating strong local proximity and alignment with underlying time series dynamics. In contrast, baselines exhibited flatter distributions or anomalous spikes, suggesting weaker alignment or overfitting.

5. Significance and Claims

The paper posits that the numeric interface is a key bottleneck in LLM-based time series forecasting. By introducing TempoWave, the authors demonstrate that principled multi-resolution embeddings can better couple an LLM's contextual reasoning capabilities with precise numerical forecasting.

The significance of this work lies in:

• Preserving Numerical Structure: It offers a method to represent continuous values that respects ordinal relations and multi-scale temporal structures (local fluctuations and global trends) within the discrete transformer architecture.
• Robustness: It addresses the fragility of numerical reasoning in LLMs under normalization and distribution shifts.
• Generalizability: It shows that standard LLM backbones, when equipped with a numerically grounded interface, can outperform specialized time-series models, particularly in scenarios requiring the integration of heterogeneous textual signals (context-aware forecasting).

The authors conclude that while challenges remain, TempoWave provides a pathway to more faithful and reliable LLM-based forecasting, suggesting that future work could explore its applicability to non-contextual pipelines relying on discretization.