🤖 AI

Inferring Events from Time Series using Language Models

This paper introduces a new benchmark and automated task generation method to evaluate Large Language Models' ability to infer natural language events from time series data, demonstrating that even minimal context yields surprising success and that combining distillation with Reinforcement Learning can enable smaller models to match the performance of large proprietary reasoning models.

Original authors: Mingtian Tan, Mike A. Merrill, Zack Gottesman, Tim Althoff, David Evans, Tom Hartvigsen

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingtian Tan, Mike A. Merrill, Zack Gottesman, Tim Althoff, David Evans, Tom Hartvigsen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a sports game on TV, but the commentator has gone silent. You can only see a single number on the screen: Team A's chance of winning. It goes up and down like a heartbeat.

When the number jumps up, you know something good happened for Team A (like a basket) or something bad happened for Team B (like a foul).
When the number crashes down, the opposite occurred.

This paper asks a fascinating question: Can a super-smart computer (a Large Language Model) look at that wiggling line of numbers and guess exactly what happened in the game, even if it can't see the players?

Here is the story of how they tested this, using simple analogies.

1. The "Blind Detective" Game

The researchers built a giant puzzle game called GAMETIME. They took thousands of real basketball and football games.

The Clue: They hid the play-by-play commentary (the events) and only showed the "Win Probability" line.
The Task: They gave the computer four possible stories of what happened during a specific time.
- Story A: "Team A scored a 3-pointer."
- Story B: "Team B missed a shot."
- Story C: "A player got a timeout."
- Story D: "The referee threw a flag."
The Goal: The computer had to pick the story that matched the wiggling line of numbers.

2. The Results: Who Got It Right?

They tested 18 different "brains" (AI models) to see who could solve the mystery best.

The "Reasoning" Champions: Some AIs are built specifically to think hard before answering (like a student who takes a deep breath and solves a math problem step-by-step). These models, like OpenAI's o1 and DeepSeek-R1, were the detectives. They got about 83% of the answers right. They could look at a sudden spike in the win probability and say, "Ah, that must be a big score!"
The "Fast Talkers": Other models are great at chatting but don't always pause to think deeply. They struggled more, getting around 40% right.
The "Small Brain" Miracle: The most surprising part? They took a tiny, cheap model (Qwen 1.5B) that was initially terrible at this (scoring near random guessing). They gave it a "tutor" (a process called distillation) to learn how to think, and then let it practice with a reward system (Reinforcement Learning).
- The Analogy: Imagine a student who can barely read. You give them a textbook from a genius, then let them take practice tests where they get a gold star for every correct step. Suddenly, that small student starts beating the big, expensive models! They went from the worst to nearly the best.

3. The "No Cheating" Rule

To make sure the computers weren't just cheating by memorizing player names (e.g., knowing that "LeBron James" usually scores), the researchers scrubbed the data.

Instead of "LeBron," they wrote "Player A."
Instead of "Lakers," they wrote "Team B."
The Result: Even without knowing who the players were, the smartest models could still figure out the events just by looking at the numbers. This proves they were actually reasoning about cause and effect, not just guessing based on famous names.

4. Does It Work Outside Sports?

The researchers asked: "Can this work for other things, like stock prices or weather?"
They tested the models on cryptocurrency prices, flu rates, and gas prices.

The Finding: Yes! Even when they removed all the numbers from the news headlines (so the AI couldn't just match a price to a headline), the smart models could still guess the right story about what happened. They learned the pattern of how events change numbers, not just the specific numbers themselves.

5. The "Overthinking" Trap

The paper also found a funny flaw in how these computers think.

Too Little Thinking: Some models guessed too fast. They saw a number go up and immediately picked a "good" event without checking if it made sense. This is like "rushing" through a test.
Too Much Thinking: Other models got stuck in a loop. They would think, "Wait, maybe this happened... no, wait, maybe that..." and talk to themselves for so long that they got confused and picked the wrong answer.
The Sweet Spot: The best performance happened when the model thought for just the right amount of time—about 1,400 words of internal reasoning.

The Bottom Line

This paper shows that AI is getting really good at being a translator between numbers and stories. It can look at a graph of data and tell you the narrative behind it.

Most importantly, they proved that you don't need a massive, expensive super-computer to do this. If you take a small, cheap model and teach it how to think (using the "tutor" and "practice test" method), it can become a master detective, almost as good as the most expensive models on the market.

Technical Summary: Inferring Events from Time Series using Language Models

Problem Definition

The paper addresses a gap in current Large Language Model (LLM) research regarding the ability to reason about natural language events associated with numerical time series data. While prior work has focused on using text to assist in time series forecasting or analyzing time series in isolation (e.g., trend analysis, anomaly detection), few benchmarks exist for inferring the sequence of events that caused or correlated with observed variations in a time series. The authors define the task as inferring unobserved event descriptions given a sequence of timestamped real-valued data (e.g., win probabilities) and partial context, where the model must select the event sequence most consistent with the numerical fluctuations.

Methodology

Dataset and Benchmark Generation

The authors introduce GAMETIME, a dataset comprising 4,200 games from the NBA and NFL, totaling 1.7 million data points.

Data Source: The numerical time series consists of win probabilities (estimated by ESPN's proprietary models) which fluctuate in response to in-game events. The event descriptions are derived from play-by-play data (e.g., scoring, fouls, turnovers).
Task Formulation: The evaluation is framed as a multiple-choice question. The model is provided with a segment of the time series (win probabilities) and must select the correct sequence of events from four options.
Contamination Control: To prevent memorization, the dataset uses generic labels (e.g., "Team A," "Player from Team A") instead of real names. For open-domain extensions (cryptocurrency, health, energy), numerical values in event descriptions are replaced with symbols (e.g., $\alpha$ ) to prevent models from matching events based on specific dates or prices.
Ablation Studies: The authors test the impact of providing additional context (timestamps, scores, partial events) and perform ablations by removing time series data, removing real names, or shuffling event orders to test logical consistency.

Experimental Setup

Models Evaluated: 18 LLMs were tested, including proprietary models (GPT-4o, OpenAI's o1), open-weight models (LLaMA 3.1, Qwen 2.5, Phi, Mixtral), and reasoning-focused distilled models (DeepSeek-R1 variants).
Training Strategy: The authors investigate post-training optimization for smaller models (specifically Qwen2.5 1.5B). This involves a two-stage process:
1. Warm-up: Knowledge distillation from a stronger reasoning model (DS-R1-distill-Qwen-32B) using correctly reasoned Q&A pairs.
2. Reinforcement Learning (RL): Further optimization using Group Relative Policy Optimization (GRPO) to refine reasoning trajectories.
Reasoning Prompts: The study evaluates the impact of Chain-of-Thought (CoT) prompting, providing models with examples of step-by-step reasoning.

Key Results

Baseline Performance

Reasoning Models Outperform: Models explicitly designed for reasoning (e.g., OpenAI's o1, DeepSeek-R1 distilled variants) significantly outperform standard generative models. OpenAI's o1 achieved the highest accuracy (83% on NBA tasks, 75.5% on NFL), followed by DS-R1-distill-Qwen-32B (68% NBA, 43% NFL).
CoT Impact: Chain-of-Thought prompting improved accuracy by an average of 4.5% for basketball tasks and 9.6% for football tasks, though it slightly increased the rate of invalid format responses.
Context Sensitivity: Providing additional context (timestamps, scores) generally improved performance, particularly for the NFL task. Conversely, removing the time series data or real entity names caused performance to drop to near-random levels, confirming that models rely on the correlation between numerical data and events rather than entity memorization.

Post-Training Optimization

Small Model Enhancement: The most significant finding regarding model capability is the success of post-training on the small Qwen2.5 1.5B model.
- Baseline: The untrained 1.5B model performed poorly (11 correct out of 200 on NBA).
- Distillation + RL: After warm-up with distilled reasoning data and GRPO training, the 1.5B model's accuracy rose to 151/200 (75.5%) on the NBA task.
- Comparison: This post-trained small model surpassed the much larger DS-R1-distill-Qwen-32B (136/200) and approached the performance of OpenAI's o1 (166/200).

Generalizability

The approach was extended to four other domains: cryptocurrency (Bitcoin), trade (IMEX), health (Influenza rates), and energy (Gasoline prices).

LLMs demonstrated moderate reasoning capabilities in these open-domain settings even when numerical values in event descriptions were filtered out.
GPT-4o consistently achieved over 50% accuracy across domains, while reasoning-focused open models (Qwen2.5 72B, DS-R1) performed comparably.

Significance and Contributions

The paper claims three primary contributions:

Methodology: A programmatic method for generating tasks that assess an LLM's ability to reason about events associated with time series data, moving beyond simple forecasting or trend analysis.
Benchmark: The introduction of GAMETIME, a continuously updating, real-world dataset with 1.7 million paired time-series and event points, designed to avoid the contamination issues of static benchmarks.
Optimization Insights: The demonstration that post-training optimization (distillation followed by RL) can drastically enhance the event-inference capabilities of small language models, enabling a 1.5B parameter model to outperform larger proprietary models on this specific reasoning task.

The authors conclude that while current models show promising reasoning capabilities, the task remains challenging, particularly for non-reasoning models and in domains where the causal link between events and time series is less direct than in sports. The work establishes a foundation for evaluating and improving multimodal reasoning over numerical sequences and natural language events.