From Tokenizer Bias to Backbone Capability: A Controlled Study of LLMs for Time Series Forecasting

This paper investigates the inherent forecasting capabilities of large language models (LLMs) by controlling for tokenizer bias through large-scale pre-training, revealing that while LLM backbones show some promise, they still struggle to consistently outperform models specifically trained on large-scale time series data.

The Gist

Imagine you have a brilliant, world-famous chef (the Large Language Model or LLM) who has spent years mastering the art of cooking complex French cuisine. Now, you want to use this chef to bake a simple, perfect loaf of bread (a Time Series Forecast, like predicting tomorrow's stock price or weather).

The current trend in the tech world is to say, "Let's just give this French chef some bread dough and see what happens!" But there's a catch: the chef doesn't speak "dough." So, researchers built a translator (the Tokenizer) to turn the dough into French words, and a reverse translator (the Detokenizer) to turn the chef's French instructions back into bread.

This paper, titled "From Tokenizer Bias to Backbone Capability," is a controlled experiment to answer one big question: Is the chef actually doing the cooking, or is the translator just doing all the work?

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Over-Adapting" Translator

The authors noticed that in previous studies, researchers tested these chefs on very small batches of dough (small datasets).

• What happened: The translators (Tokenizer/Detokenizer) were so good at adapting to that specific small batch of dough that they memorized the recipe perfectly. They learned to translate "dough" into "French" so well that even if you swapped the French chef for a random person, the bread still turned out great.
• The Flaw: Because the translators were so specialized to the small data, they masked the chef's actual talent. It looked like the chef was a genius, but really, the translators were just doing all the heavy lifting. The researchers call this "Tokenizer Bias."

2. The Experiment: The Three "Chef" Scenarios

To find out the truth, the researchers set up a fair test using three different scenarios, all using the exact same kitchen setup (architecture) but different training histories:

• Scenario A (The Real Chef): They took the famous French chef (pre-trained LLM) and kept them frozen (didn't let them learn new recipes). They only trained the translators on a massive amount of dough data (100 million samples) so the translators wouldn't be biased toward a small batch.
  • Result: The bread was okay, but not amazing. The French chef didn't seem to add much magic to the bread.
• Scenario B (The Blank Slate Chef): They took a random person with no cooking experience (randomly initialized weights) and trained them on the same massive dough data, but kept the translators frozen (using the ones from Scenario A).
  • Result: Surprisingly, this random person performed almost as well as the famous French chef.
• Scenario C (The Full Team): They trained the random person and the translators together from scratch on the massive data.
  • Result: This team did the best. But the key takeaway is that the "famous chef" (the pre-trained LLM) didn't seem to provide a significant advantage over the random person who just learned the specific task.

3. The Big Revelations

The study uncovered a few surprising truths:

• The "Vocabulary" Mismatch: The researchers tried to force the dough to look like words the chef already knew (matching time series to the LLM's vocabulary). It was like trying to explain baking a loaf of bread using only words about painting a portrait. It didn't help; in fact, it made the results worse. The chef's "language brain" doesn't naturally understand "time patterns."
• Size Doesn't Matter (Much): They tested bigger, smarter chefs (larger LLMs like LLaMA-8B). You'd think a bigger brain would be better at predicting the future, but no. The bigger chefs performed slightly worse or the same as the smaller ones. Their massive knowledge of human language didn't translate to predicting weather or traffic.
• The "Small Data" Trap: When you test on small datasets, the translators can cheat by memorizing the data. You need to test on "Zero-Shot" scenarios (predicting a brand new type of dough the chef has never seen) to see if the chef actually has the skill. Even then, the pre-trained knowledge didn't help much.

The Bottom Line

The paper concludes that using a pre-trained Large Language Model as the "brain" for time series forecasting is currently overhyped.

Think of it like this: You wouldn't hire a Nobel Prize-winning physicist to fix your car engine just because they are smart. They might be brilliant, but they haven't learned the specific mechanics of engines. Similarly, LLMs are brilliant at understanding human language, but they haven't learned the specific "mechanics" of time series data.

The researchers found that a model trained specifically on time series data (even a simpler one) often outperforms a massive, pre-trained language model. The "magic" of the LLM isn't transferring to this new job because the job requires a different kind of intelligence.

In short: Don't just throw a giant language model at a time series problem and hope for the best. The translators are doing the work, not the giant brain. To get good results, you need a model specifically trained on the data, not just a generalist with a big vocabulary.

Technical Summary

1. Problem Statement

The paper addresses a critical controversy in the field of time series forecasting: Do pre-trained Large Language Models (LLMs) actually possess intrinsic predictive capabilities for time series data, or are their reported successes merely artifacts of specific architectural designs and dataset biases?

Existing research typically follows a pipeline where time series are split into patches, mapped to an LLM's token space via a Tokenizer, processed by a frozen or fine-tuned LLM backbone, and reconstructed via a Detokenizer. The authors identify a fundamental flaw in current evaluation methods:

• Tokenizer-Detokenizer Bias: When trained on small datasets, the Tokenizer and Detokenizer (typically simple MLPs) overfit to the specific data distribution.
• Masking Effect: This tight coupling allows the Tokenizer-Detokenizer pair to learn the dataset's patterns effectively on their own, masking the true (or lack of) contribution of the frozen LLM backbone.
• Evaluation Flaw: Consequently, zero-shot and few-shot evaluations on small datasets are biased. If the Tokenizer-Detokenizer is trained on Dataset A and tested on Dataset B, performance depends heavily on the similarity between A and B, rather than the LLM's generalization ability.

2. Methodology

To isolate the intrinsic capability of the LLM backbone, the authors propose a Controlled Evaluation Framework using three models with identical architectures (a 12-layer GPT-2 Decoder-Only transformer) but distinct pre-training strategies. All models use a large-scale pre-training dataset (UTS-2G, ~100M samples) to ensure unbiased Tokenizer-Detokenizer pairs.

The three models are:

1. Train-TD (Train Tokenizer-Detokenizer):
   • Backbone: Frozen, pre-trained GPT-2 weights (text knowledge).
   • Strategy: The Tokenizer and Detokenizer are trained on the large-scale time series dataset.
   • Goal: To test if the original text-pre-trained LLM backbone can generalize to time series when paired with unbiased adapters.
2. Train-B (Train Backbone):
   • Backbone: Randomly initialized Transformer layers (no pre-trained knowledge).
   • Strategy: The Tokenizer and Detokenizer are reused from Train-TD (frozen), while the Backbone is trained from scratch on the large-scale time series dataset.
   • Goal: To measure the performance of a time-series-specific backbone without text pre-training, controlling for the adapter bias.
3. Train-BTD (Train Backbone and Tokenizer-Detokenizer):
   • Backbone: Randomly initialized.
   • Strategy: All parameters (Tokenizer, Detokenizer, and Backbone) are trained jointly from scratch on the large-scale dataset.
   • Goal: To serve as an upper-bound baseline representing a model fully optimized for time series without relying on language priors.

Experimental Setup:

• Datasets: Pre-trained on UTS-2G; evaluated on seven real-world multivariate datasets (ETTh1/2, ETTm1/2, Traffic, Electricity, Exchange Rate).
• Tasks: Zero-shot (no fine-tuning on target data) and Few-shot (10% fine-tuning).
• Baselines: Compared against state-of-the-art time series foundation models (Timer, Moirai, Chronos) and a Diagonal Initialized GPT (DI-GPT) baseline.

3. Key Contributions

1. Identification of "Tokenizer-Detokenizer Bias": The authors demonstrate that previous studies failed to decouple the adapters from the backbone. On small datasets, the adapters overfit, making it impossible to determine if the LLM backbone contributes to performance.
2. Controlled Pre-training Paradigm: By training on a massive dataset (100M samples) before evaluation, they created unbiased Tokenizer-Detokenizer pairs, allowing for a fair assessment of the backbone's zero-shot and few-shot capabilities.
3. Quantification of LLM Utility: The study provides empirical evidence quantifying exactly how much time-series data is needed to match the performance of a frozen LLM backbone.
4. Analysis of Vocabulary Alignment: The paper investigates whether aligning time series tokens with the LLM's vocabulary improves performance, finding that it actually degrades zero-shot capabilities.

4. Key Results

A. The "Bias" Effect

• Small Dataset Failure: When trained on small datasets, models with frozen backbones (Train-TD) perform similarly to models with random backbones (DI-GPT). This confirms that the Tokenizer-Detokenizer adapts to the data, rendering the backbone irrelevant in small-data regimes.
• Scale Sensitivity: LLM-based models show high relative errors as dataset size increases, whereas specialized models (like TSMixer) remain stable. This suggests LLM-based models rely on the adapters rather than the backbone for learning complex patterns.

B. Zero-Shot Performance

• Train-BTD > Train-B > Train-TD: The model trained from scratch on time series (Train-BTD) significantly outperformed the model using the frozen GPT-2 backbone (Train-TD).
• Text Knowledge is Useless: Replacing the frozen text-pre-trained backbone with a randomly initialized one trained on time series (Train-B) yielded better results than keeping the text backbone (Train-TD). This implies textual pre-training knowledge does not transfer effectively to time series forecasting.
• Comparison with SOTA: Even the best LLM-based approach (Train-BTD) generally underperformed compared to foundation models specifically designed for time series (Timer, Moirai, Chronos), especially on complex, high-dimensional datasets like Traffic and Electricity.

C. Vocabulary and Fine-tuning Insights

• Vocabulary Misalignment: Attempts to force time series tokens into the GPT-2 vocabulary space (via cross-attention) significantly reduced zero-shot performance. The models rely on out-of-distribution generalization rather than vocabulary matching.
• Fine-tuning: For models with time-series-specific knowledge (Train-BTD), fine-tuning the Tokenizer-Detokenizer was more effective than fine-tuning the backbone. For the text-based model (Train-TD), fine-tuning the backbone was necessary but yielded limited gains.
• Data Efficiency: A randomly initialized Transformer backbone trained on only 15–50 million time series samples achieved performance comparable to the frozen GPT-2 backbone. This suggests that the massive parameter count of LLMs is not being utilized efficiently for this task.
• Model Scale: Increasing the LLM size (from GPT-2 128M to Qwen-1.8B and LLaMA3-8B) did not improve zero-shot performance; in many cases, larger models performed slightly worse.

5. Significance and Conclusion

The paper fundamentally challenges the prevailing assumption that pre-trained LLMs are a "silver bullet" for time series forecasting.

• Debunking the Hype: The study concludes that the success of LLMs in time series is often an illusion caused by Tokenizer-Detokenizer overfitting on small datasets.
• Limited Intrinsic Capability: The intrinsic predictive capability of frozen LLM backbones for time series is limited. Textual pre-training does not provide a significant advantage over training a Transformer from scratch on time series data.
• Future Direction: The authors suggest that rather than forcing LLMs into time series tasks, the community should focus on developing specialized foundation models trained on massive, diverse time series datasets (like Timer or Moirai) or optimizing architectures specifically for temporal dependencies, rather than relying on language priors.

In summary, the paper calls for a shift from "LLM-centric" approaches to "Time-Series-Centric" approaches, emphasizing the need for unbiased evaluation protocols to accurately measure model capabilities.