💬 NLP

Prompting Underestimates LLM Capability for Time Series Classification

This paper demonstrates that large language models possess significantly stronger time series classification capabilities than zero-shot prompting suggests, as revealed by linear probes on internal representations that achieve performance comparable to specialized models.

Original authors: Dan Schumacher, Erfan Nourbakhsh, Rocky Slavin, Anthony Rios

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dan Schumacher, Erfan Nourbakhsh, Rocky Slavin, Anthony Rios

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Genius in a Glass House"

Imagine you have a brilliant, world-class detective (the Large Language Model or LLM) who can solve complex crimes. However, you are only allowed to ask them questions through a tiny, crackling walkie-talkie (the Prompt).

When you ask the detective, "Who stole the cookie?" through the walkie-talkie, they might mumble, "Uh, maybe the cat?" and get it wrong. You conclude: "This detective isn't very good at solving crimes."

But what if the detective actually saw the whole crime scene, noticed the muddy footprints, and knew exactly who did it? The problem wasn't the detective's brain; it was the walkie-talkie. The detective just couldn't speak the answer clearly through that specific channel.

This paper argues that LLMs are actually great at understanding time series data (like stock markets, heartbeats, or weather patterns), but the way we usually test them (by asking them to "chat" about the data) makes them look stupid.

The Problem: The "Walkie-Talkie" Test

Time Series Data is just a list of numbers that happen over time. Think of it like a heartbeat monitor: beep... beep... beep... skip... beep.

Researchers have been trying to use AI to read these heartbeats. They usually do this by feeding the numbers into the AI and saying, "Here is a heartbeat. Is it healthy or sick?"

The Result: The AI often fails. It guesses randomly.
The Old Conclusion: "AI is bad at understanding time. It doesn't get the pattern."

The New Discovery: The authors of this paper decided to peek behind the curtain. Instead of asking the AI to speak the answer, they looked directly inside the AI's "brain" (its internal math) to see what it was thinking.

The Solution: The "X-Ray Vision" Test

The researchers used a technique called Linear Probing.

Prompting (The Walkie-Talkie): You ask the AI to talk. It struggles to translate its complex internal thoughts into words.
Probing (The X-Ray): You don't ask the AI to talk. Instead, you take a snapshot of its internal "thoughts" (hidden representations) and run them through a simple, super-fast calculator (a linear classifier) to see if the answer is already there.

The Analogy:
Imagine the AI is a library filled with books.

Prompting is like asking a librarian to find a book and read the title out loud. If the librarian is shy or the library is noisy, they might get the title wrong.
Probing is like walking into the library and looking at the books on the shelf. You can see immediately that the book is there, even if the librarian can't say the title.

The Shocking Result:
When they used the "X-Ray" (Probing), the AI's performance skyrocketed.

Prompting Score: ~15% (Basically guessing).
Probing Score: ~65% (Very good, often beating specialized time-series experts!).

This proves the AI does understand the patterns. It just can't say them out loud when asked in a chat format.

Key Findings (The "Aha!" Moments)

1. The "Early Bird" Discovery

The researchers looked at which part of the AI's brain held the answers. They found that the AI figures out the pattern very early in its processing (in the first few layers), almost immediately after seeing the data.

Metaphor: It's like a chef who tastes a soup and instantly knows it needs salt. The "knowing" happens in the first second. The rest of the cooking process is just trying to figure out how to explain why it needs salt.

2. Pictures Help, But Numbers Work Too

They tested the AI with just numbers (text) and with pictures of the data (graphs).

Surprise: Even without pictures, just looking at the numbers, the AI's internal "X-ray" showed it understood the data.
Bonus: Giving the AI a picture of the graph (like a line chart) helped it understand even better, but the numbers alone were enough to prove the AI had the knowledge.

3. The "Random Guess" Baseline

To make sure they weren't tricking themselves, they compared the AI to a "Random Brain" (a model with random weights).

The Random Brain failed miserably.
The Real AI succeeded.
Conclusion: The AI isn't just memorizing the answers; it actually learned the structure of time.

Why Does This Matter?

For a long time, people thought, "AI is great at writing poems, but it's terrible at math and science data."

This paper says: "No, you're just asking the wrong question."

If you want an AI to classify a heartbeat or predict a stock trend, don't ask it to "chat" about it. Instead, use its internal "brain waves" (probing) to make the decision. The AI is a hidden genius for time series data; we just need to stop using the walkie-talkie and start using the X-ray machine.

Summary in One Sentence

Large Language Models are actually experts at reading time-based data, but they look stupid when we force them to answer in chat format; if we look directly at their internal thoughts, they perform brilliantly.

1. Problem Statement

Recent evaluations suggest that Large Language Models (LLMs) perform poorly on time-series classification tasks, often achieving near-random accuracy when using zero-shot prompting. This has led to the hypothesis that LLMs lack the ability to encode meaningful temporal structures.

The authors argue that this conclusion is flawed because it conflates model representational capacity (what the model "knows" internally) with decoding interface limitations (how well the model can express that knowledge via text generation). The paper posits that current prompt-based evaluations systematically underestimate the true capabilities of LLMs for time-series tasks due to the difficulty of extracting discriminative information through natural language generation rather than direct access to internal states.

2. Methodology

The study employs a comparative framework to disentangle prompting performance from internal representation quality.

A. Experimental Setup

Models: The authors evaluated several state-of-the-art models, including:
- Vision-Language Models (vLLMs): Llama-3.2-11B-Vision, Mistral-Small-3.1-24B, Qwen2.5-VL-32B.
- Text-Only LLMs: Gemma-2-9B, GPT-oss.
Datasets: Six diverse time-series datasets were used, covering domains like healthcare (EMG, TEE), human activity recognition (HAR, HAD), finance/energy (CTU), and environmental science (RWC).
Input Modalities:
1. Text (d): Numerical values serialized into digit-based tokens (e.g., 1.2 $\to$ 1 2) to avoid Byte-Pair Encoding (BPE) fragmentation.
2. Visual (v): Time series rendered as line plots or spectrograms.
3. Multimodal (d + v): Combination of text and visual inputs.

B. Evaluation Paradigms

The core of the methodology involves comparing two distinct evaluation methods on the same internal representations:

Prompt-Based Classification: The model is prompted to generate a class label directly (e.g., "Classify this series as A or B"). Performance is measured by the generated text.
Linear Probing: Instead of generating text, the authors extract hidden state embeddings ( $h^{(\ell)}$ $h^{(ℓ)}$ ) from specific layers ( $\ell$ $ℓ$ ) of the frozen LLM. A lightweight linear classifier (logistic regression) is trained on these embeddings to predict the class label.
- Controls: To ensure the probe isn't just memorizing, they used a Random-Probe baseline (training a probe on a model with random weights) and compared against standard time-series baselines (Attend, TS2Vec, Moment, OneFitsAll).

C. Ablation Studies

The authors tested the impact of:

Chain-of-Thought (CoT): Does reasoning improve representation or prompting?
In-Context Learning (Few-Shot): Does providing examples help?
Prompt Stability: Sensitivity to prompt wording variations and sampling consistency.

3. Key Contributions

Disentanglement of Capacity vs. Interface: This is the first work to explicitly separate prompt-level performance from representational capacity in time-series classification. They demonstrate that while LLMs fail to generate correct answers via prompting, their internal states contain highly discriminative temporal information.
Superiority of Probing over Prompting: The study shows that simple linear probes over frozen LLM hidden states consistently match or outperform complex, fine-tuned time-series pipelines and prompt-based approaches.
Layer-wise Analysis: The authors identify that class-discriminative temporal information emerges early in the transformer stack (often within the first 5–12 layers) and is amplified by visual inputs, suggesting LLMs act as effective sequence encoders before higher-level reasoning stages.

4. Key Results

A. Performance Gap (Prompting vs. Probing)

There is a massive discrepancy between what LLMs can "say" and what they "know":

Average F1 Scores:
- Prompting: Llama (0.15), Mistral (0.26), Qwen (0.24). These are near random chance.
- Probing: Llama (0.61), Mistral (0.65), Qwen (0.67).
Specific Case (EMG Dataset):
- Prompting Llama: 0.175 F1.
- Probing Llama: 1.00 F1 (Perfect separation).
- This indicates the model perfectly encodes the signal but fails to retrieve it via text generation.

B. Comparison with State-of-the-Art (SOTA)

Probing Performance: The linear probes on vLLMs achieved average F1 scores (0.61–0.67) that are competitive with, and in some cases exceed, specialized time-series foundation models like Moment (0.76) and OneFitsAll (0.61).
Visual Modality: Including visual inputs (v) significantly boosted probing performance, particularly for Mistral and Qwen, suggesting vLLMs leverage visual pattern recognition effectively for time series.

C. Prompt Stability and Sensitivity

Prompt-based evaluation is highly unstable. Small changes in prompt wording resulted in an average performance variance ( $\Delta$ ) of 0.088 in Macro F1.
Repeated sampling (Pass@20) only improved accuracy by ~0.048, indicating that the failure is not just due to sampling noise but a fundamental inability of the generation interface to access the temporal logic.

D. Text-Only LLMs

The findings extend to text-only models (Gemma, GPT-oss), where probing (F1 ~0.46–0.60) significantly outperformed prompting (F1 ~0.20), confirming that the limitation is the prompting interface, not the multimodal architecture.

5. Significance and Conclusion

Reframing LLM Capabilities:
The paper concludes that the perceived failure of LLMs in time-series classification is an artifact of the evaluation interface, not a lack of representational capacity. LLMs encode rich, discriminative temporal structures in their early layers, but current prompting strategies cannot reliably extract this information through text generation.

Implications:

Feature Extraction: Pretrained LLMs (and vLLMs) should be viewed as robust, general-purpose feature extractors for non-linguistic modalities. Simple linear probes on their hidden states can replace complex, task-specific fine-tuning pipelines.
Evaluation Standards: Future evaluations of LLMs on non-NLP tasks must look beyond zero-shot prompting. Relying solely on generation metrics leads to a systematic underestimation of model capabilities.
Future Directions: The gap suggests a need for better decoding mechanisms (e.g., attention-based extraction, better prompt engineering, or hybrid architectures) that can bridge the gap between internal representation and external output.

In summary, the paper demonstrates that LLMs "know" more about time series than they can "say," and that linear probing reveals a latent capability that prompt-based evaluations completely miss.