💬 NLP

Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models

This paper proposes an adaptive token compression framework for time series language models that leverages the distinct spectral properties of time series data and the diminishing influence of prompts across layers to achieve significant inference acceleration and performance improvements.

Original authors: Jialin Gan, Xin Qiu, Guangzhe Chen, Xue Wang

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jialin Gan, Xin Qiu, Guangzhe Chen, Xue Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly overworked, assistant (the Large Language Model or LLM) who is trying to understand two very different things at the same time: a long, complex weather report (the Time Series) and a set of instructions written on a sticky note (the Prompt).

Usually, this assistant treats every single word in the weather report and every single word in the instructions as equally important. They process them all at the same speed, one by one. The paper argues that this is a waste of time and energy.

Here is the simple breakdown of what the authors discovered and built:

1. The Problem: "One Size Fits All" is Wasteful

The authors realized that the two types of information behave very differently:

The Weather Report (Time Series): It's full of patterns. Some parts are just background noise or repetitive loops (like the sun rising every day). Other parts are the critical "plot twists" (like a sudden storm). Treating the repetitive loops with the same intensity as the storm is inefficient.
The Sticky Note (Prompt): The instructions are crucial at the very beginning to tell the assistant what to do. But as the assistant starts thinking and analyzing the data, they don't need to keep reading the whole sticky note over and over again. The instructions get "absorbed" quickly, and keeping them in the working memory for the rest of the process is just clutter.

2. The Solution: "TokenDecouple"

The team built a new system called TokenDecouple that handles these two inputs differently, like a smart traffic manager directing two different types of cars.

Part A: Compressing the Weather Report (Frequency-Domain Merging)

Instead of looking at the weather report second-by-second (time-domain), the system looks at it like a music equalizer (frequency-domain).

The Analogy: Imagine a song. Most of the time, it's just a steady drumbeat (redundant). Occasionally, there's a solo guitar riff (critical information).
The Fix: The system scans the "equalizer." It sees that many "notes" (tokens) are just repeating the same drumbeat. It groups these repetitive notes together and merges them into a single, representative note. It keeps the "guitar solos" (the critical frequencies) separate and untouched.
The Result: The assistant has to read a much shorter, cleaner version of the weather report, but it hasn't lost the important details.

Part B: Shrinking the Sticky Note (Pyramidal Decay)

The system also realized that the instructions don't need to stay big forever.

The Analogy: Think of the instructions as a large map you use to start a journey. Once you know the destination and the route, you don't need to keep the whole giant map open on the table; you just need a tiny corner of it to remember the turn.
The Fix: As the assistant goes deeper into its thinking process (through the layers of the model), the system aggressively shrinks the sticky note.
- Layer 1: Full map (100% of instructions).
- Layer 2: Quarter of the map (25%).
- Layer 3: A tiny corner (6%).
- Deep Layers: Almost nothing (less than 1%).
The Result: The assistant stops wasting brainpower re-reading instructions it has already understood.

3. The Outcome: Faster and Smarter

By doing these two things separately (decoupling them), the system achieved some impressive results:

Speed: It made the assistant 7.68 times faster in some cases.
Accuracy: Surprisingly, it didn't just get faster; it actually got better at tasks in about 78% of the tests.
Tasks: They tested this on predicting the future (forecasting), finding mistakes (anomaly detection), filling in missing data (imputation), and sorting things into categories (classification).

Summary

The paper is essentially saying: "Stop treating every piece of data the same."

For numbers (time series), group the boring, repetitive parts together so you only focus on the interesting patterns.
For text (prompts), read the instructions once, learn them, and then let them fade away as you do the actual work.

This approach allows the AI to be much more efficient without losing its smarts, making it possible to run these powerful models on faster, cheaper hardware.

Technical Summary: Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models

Problem Statement

Large Language Models (LLMs) have been increasingly applied to time series (TS) analysis by modeling numerical observations and textual prompts through a shared token interface. However, existing approaches treat TS tokens and prompt tokens uniformly, ignoring their fundamental structural differences. This uniform processing leads to inefficiency for two primary reasons:

Redundant TS Information: TS tokens often exhibit highly uneven spectral contributions. Large groups of tokens share redundant frequency patterns, while only a small subset carries critical temporal evidence. Standard time-domain merging fails to separate these functional components effectively.
Inefficient Prompt Retention: Text prompts provide high-level guidance but their influence attenuates rapidly as model depth increases. Retaining full prompt tokens across all layers incurs substantial computational overhead without proportional performance gains, as textual information is typically absorbed and transferred to TS tokens within the first few shallow layers.

Methodology: TokenDecouple

The authors propose TokenDecouple, a unified framework that decouples the compression strategies for TS tokens and prompt tokens based on their distinct information structures.

1. Frequency-Domain TS Token Merging

Instead of merging tokens based on time-domain proximity, the method operates in the frequency domain to identify and remove spectral redundancy.

Spectral Analysis: The input TS tokens are transformed via Discrete Fourier Transform (DFT). The method identifies that real-world TS signals concentrate energy in specific frequency bands, unlike white noise which has a uniform spectrum.
Active Token Selection: For each frequency component, the algorithm selects a minimal subset of tokens whose cumulative spectral magnitude accounts for at least 95% of the total energy.
Adaptive Merging Budget: The merging granularity is determined by spectral identifiability, estimated using Fisher Information. Tokens that enable sharper peaks or low-noise periodic patterns contribute more to the model's understanding. An adaptive budget is calculated to determine how many tokens to merge for each frequency band.
Local Affinity Graph: Within the selected active set, tokens are grouped into local clusters based on spectral similarity. A weighted average is computed for each group to generate merged representatives, which are then transformed back to the time domain via Inverse DFT (IDFT).

2. Pyramidal Prompt Token Compression

The framework introduces a "pyramidal decay" strategy for prompt tokens, acknowledging that their utility diminishes with model depth.

Layer-wise Decay: The number of prompt tokens is reduced geometrically at each layer (specifically, $P_{l+1} = \lceil 0.25 P_l \rceil$ ).
Residual Preservation: To prevent information loss during compression, the method computes a global residual summary (via MeanPool) of the prompt tokens at each layer and adds it to the compressed representation passed to the next layer.
Result: By the fourth layer, the number of prompt tokens becomes negligible (dropping to ~0.4% of the original), drastically reducing attention computation costs in deeper layers.

Key Contributions

Frequency-Domain Perspective: The paper provides the first systematic analysis of TS tokens from a frequency-domain perspective, revealing that different frequency components contribute unevenly to token-level information, justifying frequency-based merging.
Pyramidal Decay Effect: The authors uncover that prompt token influence decays rapidly with model depth, demonstrating that full prompt retention is unnecessary and that aggressive compression in early layers is safe.
TokenDecouple Framework: They propose a decoupled compression framework that simultaneously merges TS tokens via frequency-domain structure and compresses prompt tokens via pyramidal decay. This approach achieves significant efficiency gains while maintaining or improving performance.

Experimental Results

The framework was evaluated across four advanced LLM-based TS models (OFA, TimeLLM, CALF, S2IP) and 27 datasets covering seven tasks: imputation, classification, anomaly detection, forecasting, and three levels of reasoning (simple deterministic, complex deterministic, and probabilistic).

Performance: TokenDecouple outperformed the original models in 78% of evaluated settings. In cases where performance dropped, the degradation was minimal (e.g., maximum MAE increase of 0.008 in imputation, 1.3% accuracy drop in classification).
Efficiency: The method achieved up to 7.68× inference acceleration (specifically for the TimeLLM model). Across all models and tasks, average speedups ranged from 3.43× to 7.68×.
Token Reduction: Even for the most challenging tasks, at least 50% of TS tokens could be merged. Prompt token overhead was reduced by approximately 89% in 12-layer models and 94% in 24-layer models.
Comparison: TokenDecouple significantly outperformed baseline token merging methods (global, local, and dynamic merging) in both performance retention and inference speedup.

Significance and Claims

The paper claims that TokenDecouple offers a scalable solution for TS foundation models by addressing the specific inefficiencies of treating time series and text uniformly. The authors emphasize that their approach is not arbitrary compression but a principled reduction of redundancy based on spectral structure and layer-wise information flow.

The work highlights that:

Efficiency-Performance Trade-off: It is possible to drastically reduce computational costs (inference latency and memory) without sacrificing, and often while enhancing, model performance.
Task Robustness: The method remains effective across diverse temporal dynamics, including varying levels of stationarity, seasonality, and noise, suggesting it is not overly dependent on specific dataset statistical patterns.
Design Principle: The findings advocate for "token-aware" design in TS-language integration, where compression strategies are tailored to the specific information structure of the data modality (frequency-domain for TS, depth-dependent for prompts).

The authors modestly note limitations, stating that the study does not systematically evaluate applicability under highly non-stationary or irregularly sampled conditions, nor does it examine interactions with other efficiency techniques like quantization or structured pruning. They recommend the method be used as an efficiency-enhancing component rather than an autonomous decision-making system for safety-critical applications.