💬 NLP

Measuring Iterative Temporal Reasoning with Time Puzzles

This paper introduces "Time Puzzles," a benchmark for evaluating iterative temporal reasoning with tools, revealing that even top LLMs struggle with date inference without explicit constraints and that current tool use remains unreliable for such tasks.

Original authors: Zhengxiang Wang, Zeyu Dong

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhengxiang Wang, Zeyu Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for a stolen diamond, you are looking for a specific date on the calendar.

This paper introduces a new game called "Time Puzzles" designed to test how well modern AI (Large Language Models) can play detective when they are allowed to use tools like a web search engine.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Static" vs. "Real World" Gap

Think of most AI tests like a closed-book exam. The AI is given a question and must answer it using only what it memorized during its training.

The Reality: In the real world, AI doesn't just sit in a library. It has a smartphone. It can Google things, check a calendar app, or look up a historical event.
The Issue: Current tests don't check if the AI can use its smartphone effectively. They only check if it remembers things. The authors wanted to see: Can the AI use a search engine to solve a puzzle step-by-step, just like a human would?

2. The Solution: "Time Puzzles"

The authors created a new type of riddle. Imagine a puzzle where you have to find a specific day, but you are given clues like a detective's case file:

Clue 1: "It's the same year Hanns Blaschke was the mayor of Vienna." (You need to search this fact).
Clue 2: "It's the 6th month of the Chinese Lunar calendar." (You need to convert this to a Gregorian month).
Clue 3: "It's the second-to-last Sunday of that month."
Clue 4: "It's after the 5th of the month."

To solve this, the AI can't just "guess." It has to:

Search for when Blaschke was mayor (finding the year: 1945).
Translate the Lunar month to July.
Check a calendar for 1945 to find the second-to-last Sunday (July 22).
Verify that July 22 is after the 5th.

This is Iterative Reasoning: A chain of small steps where one answer leads to the next, often requiring a tool (like a search engine) at every step.

3. The Experiment: Testing 13 AI Models

The researchers built 600 of these puzzles and tested them on 13 different AI models (including the newest ones like GPT-5). They tested them in two ways:

The "No-Phone" Test: The AI had to solve it using only its memory.
The "With-Phone" Test: The AI was allowed to use a web search tool.

4. The Surprising Results

Even though these puzzles seem easy for a human with a Google search, the AI struggled mightily.

The "No-Phone" Struggle: Even the smartest AI (GPT-5) only got about 55% of the answers right without tools. It's like a genius student who failed the test because they refused to use their calculator.
The "Tool" Gap: When the AI was allowed to use the web search, it got better, but not as good as it should have been.
- The Twist: When the researchers rewrote the puzzles to give the AI the exact dates instead of asking it to search for them (e.g., changing "Mayor of Vienna" to "The year 1945"), the AI's performance skyrocketed.
- The Conclusion: The AI isn't bad at math or logic. It's bad at connecting the dots. It struggles to take a fact it found on the web ("Blaschke was mayor in 1945") and seamlessly plug that number into the next step of the puzzle. It often gets lost in the middle of the process.

5. The "Code Interpreter" Fail

The researchers also tried giving the AI a "calculator" (Code Interpreter) to do the math. Surprisingly, this sometimes made things worse. The AI got confused trying to use the tool, suggesting that simply handing a tool to an AI doesn't guarantee it knows how to use it correctly.

The Big Takeaway

Think of the current AI as a very smart intern.

If you give them a clear, step-by-step instruction with all the numbers filled in, they are great.
But if you say, "Go find out who was mayor in 1945, then figure out what day of the week that was, and tell me the date," they often get lost. They might find the mayor, forget the year, or mix up the calendar.

In short: We have built AIs that are great at remembering facts, but they are still learning how to be effective detectives who can use tools to solve complex, multi-step problems in the real world. The "Time Puzzles" are a new ruler to measure exactly how far they have to go.

1. Problem Statement

Current Large Language Models (LLMs) are increasingly deployed with external tools (e.g., web search, code interpreters) to retrieve factual information and verify reasoning steps. However, existing benchmarks for temporal reasoning primarily evaluate models in static, single-shot settings without tool usage. These benchmarks fail to capture the iterative, tool-augmented workflows used in real-world applications like scheduling, planning, and historical analysis.

There is a critical gap in understanding how well LLMs can:

Integrate factual lookups (e.g., "When was X the mayor?") with calendar logic.
Perform iterative reasoning (proposing, refining, and verifying dates) rather than relying on rote memorization.
Handle constraints that require cross-referencing multiple temporal anchors (e.g., combining a historical event with a specific day-of-week rule).

2. Methodology: Time Puzzles

The authors introduce Time Puzzles, a novel, constraint-based date inference task designed to evaluate iterative temporal reasoning with tools.

Task Formulation

Goal: Given a set of $N$ natural-language temporal constraints, the model must identify all valid Gregorian dates that satisfy the intersection of all constraints.
Constraints: The constraints are a mix of:
- Factual Anchors: Historical events, political tenures, or cultural facts (e.g., "The year Hanns Blaschke was mayor of Vienna").
- Calendar-Structural Relations: Cross-cultural calendar rules, weekdays, seasons, lunar months, and day-of-month relations (e.g., "It is the 2nd to last Sunday of the month").
Output: A set of valid dates (1 to 6 solutions allowed).

Data Generation

Algorithmic Generation: Puzzles are generated programmatically to ensure controllability and scalability.
Constraint Taxonomy: The system uses a curated taxonomy of 50 easy-to-search historical facts and various calendar logic types (Year, Month, Day levels).
Dataset Construction:
- Implicit Dataset: Contains natural language constraints requiring tool use (e.g., "It is the year X was born").
- Explicit Dataset: The same puzzles where factual anchors are rewritten as explicit dates (e.g., "It is the year 1945"), removing the need for factual lookup.
- Size: 600 puzzles per dataset, with solution counts evenly distributed (1 to 6 solutions).
Quality Control: Manual verification by authors and fact-checking via web search ensured 100% logical consistency. A human baseline accuracy was estimated at 80%.

Experimental Setup

Models: Evaluated 13 diverse LLMs (including GPT-5, GPT-4.1, DeepSeek-V3.2, Qwen3, and Ministral-3) released in 2025.
Settings:
- Tool-less: Zero-shot Chain-of-Thought (CoT) prompting.
- Tool-Augmented: Enabled Web Search and/or Code Interpreter (CI).
- Metrics: Exact Match (EM), Jaccard Index (JI), and F1 score. EM is the primary metric.

3. Key Results

Performance Without Tools

Even the most advanced model, GPT-5, achieved only 55.3% Exact Match (EM) accuracy on the implicit dataset without tools.
All other models scored below 37% EM.
Observation: Models struggle to retrieve and integrate simple, easily searchable facts with calendar logic, often failing to perform the necessary iterative verification.

Impact of Tool Use (Web Search)

Web search consistently improved performance across all models.
However, a significant reasoning gap remains: Models performed substantially better when the same puzzles were presented with explicit dates (removing the need for lookup) compared to when they had to use web search to find the dates.
- Example: GPT-5 reached ~85% accuracy with explicit constraints but only ~55% with implicit constraints + web search.
Conclusion: The bottleneck is not just knowledge retrieval, but the integration of retrieved facts into multi-step constraint reasoning.

Impact of Code Interpreter

Enabling Code Interpreter (+CI) did not close the gap.
For GPT-5, +CI actually degraded performance on implicit constraints, both with and without web search.
GPT-4.1 showed slight improvement with +CI + Web Search but still lagged behind the explicit constraint baseline.

Model Architecture Insights

Reasoning vs. Instruction: Reasoning-tuned models (e.g., GPT-5, DeepSeek-V3.2-R) generally outperformed instruction-tuned variants.
Scale vs. Efficiency: Larger models generally performed better, but reasoning efficiency mattered. Some smaller reasoning models (e.g., Qwen3-4B-Think) outperformed larger instruction models.
Token Usage: High-performing reasoning models often used fewer output tokens, suggesting success relies on structured, sustained reasoning rather than exhaustive generation.

4. Key Contributions

Time Puzzles Benchmark: A new, algorithmically generated dataset specifically designed to evaluate iterative, tool-augmented temporal reasoning, filling a gap left by static benchmarks.
Diagnostic of Tool Integration: The study reveals that while LLMs can retrieve facts, they struggle to reliably integrate those facts into complex, multi-step logical constraints. The "reasoning gap" persists even with tool access.
Empirical Evidence on Tool Efficacy: Demonstrates that simply adding web search does not guarantee success; the ability to rewrite implicit constraints into explicit logical steps is a critical missing capability in current models.
Open Resources: The authors open-sourced the code, dataset generation algorithms, and the puzzles themselves.

5. Significance and Implications

Real-World Relevance: The findings highlight a critical limitation in current AI agents. In real-world scenarios (e.g., legal analysis, event planning), systems must combine external knowledge with rigid logical constraints. Current models often fail to maintain consistency across these steps.
Future Directions: The results suggest that future improvements in LLMs should focus not just on retrieval accuracy, but on reasoning architectures that can effectively manage state, verify intermediate steps, and synthesize retrieved data into coherent logical conclusions.
Benchmarking Standard: Time Puzzles offers a cost-effective, discriminative, and scalable diagnostic tool for evaluating the "agentic" capabilities of future LLMs.

Limitations Noted by Authors

Synthetic Nature: Constraints are template-based and may lack the ambiguity or conflicting evidence found in messy real-world data.
Scope: The study focuses on English and off-the-shelf tools; it does not test sophisticated custom tool-use policies.
Solution-Less Puzzles: The current dataset excludes puzzles with zero solutions to focus on finding valid dates, though future work could explore detecting contradictions.