🤖 AI

Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding

This paper introduces TIDE, a new expert-verified benchmark for evaluating large language models on temporally evolving document understanding, revealing that current models struggle significantly with version resolution and often prioritize confident parametric knowledge over authoritative, time-specific text.

Original authors: Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury, Md Adnan Arefeen, Farig Sadeque, Md. Faizul Bari, Swakkhar Shatabda

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury, Md Adnan Arefeen, Farig Sadeque, Md. Faizul Bari, Swakkhar Shatabda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the right rule for a game, but the rulebook is a living thing that changes every day. In the world of artificial intelligence, there are two main ways knowledge is usually handled. The first is like a giant encyclopedia: if a fact changes, the old one is simply erased and replaced with the new one. If you ask about the capital of a country that changed its name, the encyclopedia just shows you the new name. The second way, which is much trickier, is like a stack of legal contracts or a video game with patches. Here, an old rule doesn't disappear; it stays perfectly correct for the time it was active, while a new rule takes over later. If you ask, "What was the speed limit in 2015?" you need the 2015 rule, not the 2025 one. If the computer gets this wrong, it's not just a silly mistake; it could be like telling someone to pay the wrong tax or follow a medical guideline that was canceled years ago. This is the challenge of "version resolution": figuring out exactly which version of a rule was in charge on a specific date.

A team of researchers has built a new test called TIDE (Temporal Information Drift in Evolving Documents) to see if today's smartest AI computers can handle this tricky time-traveling logic. They didn't just make up fake questions; they dug into 644 real, official documents from the Bangladesh government, covering customs laws, tax codes, and regulations from 1969 all the way to 2025. These documents are messy, mixing English and Bengali, and they contain a web of amendments where one rule cancels another, which then gets canceled by a third. The researchers created 3,050 questions based on these documents, asking the AI to act like a time-traveling lawyer. They tested nine of the most powerful AI models available, giving them three different ways to find answers: relying only on what they memorized during training (like a closed-book exam), reading the exact right documents provided to them, or searching a database to find the right pages.

The results were a bit of a wake-up call. Even the best AI models, when given the perfect documents to read, only got about 68.5% of the answers right. That means they were still failing more than one out of every three times, even when the answer was right in front of their "eyes." The study found that these models are surprisingly good at finding the right text when they know exactly where to look, but they are terrible at realizing when a text they are reading is actually the wrong version for the date in question. For example, if you give an AI a document that says "The tax is 10%" but the question asks about a time when the tax was 5%, the AI often confidently ignores the document and sticks with its memory or the wrong text. In fact, when the researchers tried to trick the AI with a confident but wrong statement, the models were very good at flagging that something was wrong (detecting errors about 79% of the time), but they were terrible at pinpointing exactly what was wrong (identifying the specific error only about 19% of the time). This means they often sense a problem but confidently blame the wrong detail.

The paper suggests that simply giving AI more text to read or better search tools isn't enough to solve this problem. The models struggle most with tasks that require them to piece together a timeline, like sorting events in order or calculating what the rule was "three years later." They tend to assume that rules only get stricter or change in one direction, missing the fact that laws often get repealed or reverted to old versions. While the researchers showed that having the right documents helps, it doesn't guarantee success. The study concludes that "version resolution" remains a major, unsolved hurdle for AI. Until these models can truly understand the flow of time in legal documents, they might be too risky to trust with real-world tasks like tax filing or customs clearance, where getting the date wrong could lead to serious financial or legal trouble.

Technical Summary: TIDE – Benchmarking Large Language Models on Temporally Evolving Document Understanding

1. Problem Definition: The Challenge of Version Resolution

Current Large Language Models (LLMs) struggle with evolving documents, such as laws, tax codes, and software documentation, where rules are amended, replaced, or reverted over time. Unlike encyclopedic knowledge (e.g., Wikipedia), where old facts are simply overwritten, evolving documents maintain a persistent historical record: an earlier version remains legally correct for its specific validity period, while a new amendment explicitly states what it replaces and when it takes effect.

The core challenge identified is version resolution: the ability to identify the specific version of a rule that is in force on a queried date. Existing temporal QA benchmarks (e.g., TempReason, MenatQA, evolveQA) treat time merely as an annotation or test closed-book knowledge on superseded values. They fail to test the critical capability of resolving which amendment governs a specific date, particularly when the source text itself contains the logic for replacement and reversion. Furthermore, existing benchmarks often lack the complexity of real-world regulatory documents, which are frequently code-mixed, heterogeneous in layout, and utilize dual calendar systems.

2. Methodology: The TIDE Benchmark

To address these gaps, the authors introduce TIDE (Temporal Information Drift in Evolving Documents), an expert-verified benchmark designed to evaluate LLMs on the version resolution of authentic, evolving legal texts.

2.1 Data Sourcing and Construction

Corpus: The benchmark is built on 644 official customs instruments issued by the Government of Bangladesh between 1969 and 2025. This includes 3 Acts of Parliament, 20 General Orders (GOs), 609 Statutory Regulatory Orders (SROs), and 12 Rules.
Complexity: The documents are deeply code-mixed (Bangla–English), contain embedded tariff tables, and 97% of them state dates in both the Bangla and Gregorian calendars.
Parsing Pipeline:
1. Extraction: A general schema extracts clauses, hierarchy, tables, and metadata without text loss, preserving the original mixed-script structure.
2. Entity-Concept Extraction: Using an LLM, the system extracts canonical entity-concept pairs (e.g., "Customs Duty" and its rate changes) to normalize naming across documents.
3. Thread Construction: Entities are clustered and linked into evolving concept threads. A thread represents a regulatory topic as a time-ordered list of instruments and the values they set, explicitly capturing amendments, repeals, and reverts.
4. QA Generation: Questions are generated against these threads, ensuring every question is grounded in the history of the rule rather than a single snapshot.

2.2 Task Types

TIDE comprises 3,050 QA pairs across eight distinct task types that isolate different aspects of temporal reasoning:

Temporal MCQ: Identifying the correct instrument/date among lexically close distractors.
Event Sorting: Reconstructing the chronological order of shuffled events without explicit dates.
Perturbation Detection: Verifying if a statement is accurate and correcting specific altered fields.
Scenario Short Answer: Extracting values from realistic scenarios without direct clause quotes.
Temporal Scenario Short Answer: Reasoning over multiple instruments to find the value in force at a specific date.
Relative-Time QA: Computing a target date from an anchor and offset (e.g., "three years later") to find the governing rule.
Open-Ended: Explaining the rationale behind policy changes.
Context Misalignment: Determining the correct answer when the prompt contains a misleading, superseded preamble.

2.3 Evaluation Protocol

The authors evaluate nine recent LLMs (six open-weight, three proprietary) under three knowledge-access settings:

Parametric (Closed-Book): The model answers from internal memory only.
Gold-Context (ICL): The model is provided with the exact gold documents used to generate the question.
Retrieval-Augmented (RAG): The model receives the top-10 retrieved clauses from a semantic search index.

Scoring Mechanism:
A unique LLM-as-a-Council approach is used, employing three heterogeneous judges (Claude Opus 4.6, Gemini 3.5 Flash, GPT-5.1). Crucially, the scoring includes a hard date gate: an answer receives zero credit if the date is incorrect, even if the meaning is correct. This separates "correct meaning" from "correct time."

3. Key Results

The evaluation of nine models reveals that version resolution remains an open problem, with the best macro-averaged accuracy reaching only 68.5% (GPT-5 under ICL).

Knowledge Access Matters: Performance consistently improves from Parametric $\to$ RAG $\to$ ICL. However, even with gold context, models fail to reach perfect accuracy, indicating that reading the correct text is necessary but not sufficient.
The Version Resolution Gap: Models are significantly better at finding correct versions than rejecting incorrect ones.
- Context Misalignment: This is the hardest task. The best score achieved by any model in any setting is 27.62% (Kimi under ICL), while other models and settings perform significantly lower. Models tend to follow a confident but false premise over the supplied authoritative text.
- Relative-Time QA: Strict accuracy is low across the board. The highest parametric score is 12.37% (GPT-OSS), while GPT-5 achieves 11.08% and Claude 3.77%. Under ICL, Claude leads with 68.03%, but the task remains difficult for most models.
- Event Sorting: Performance collapses in RAG (dropping from ~66% in ICL to ~31%) because semantic retrieval often misses parts of the timeline required to reconstruct the order.
Detection vs. Identification: In perturbation detection, models are good at flagging that a statement is wrong (high detection) but poor at identifying what was altered (low identification) without grounding. Grounding significantly improves identification.
Model Behavior:
- Parametric: Models often decline to answer or hallucinate dates.
- Grounded: Models often hallucinate dates that contradict the provided text or fail to resolve conflicts between a misleading context and the source.
- Task Difficulty: Task type predicts performance more than model identity. Simple recognition tasks (Temporal MCQ) are easy, while tasks requiring multi-document reasoning (Event Sorting, Context Misalignment) are extremely difficult.

4. Contributions

Formalization of Evolving Documents: The paper formalizes the challenge of version resolution, distinguishing it from simple fact updates.
TIDE Benchmark: The release of a high-quality, expert-verified benchmark of 3,050 QA pairs over 644 real-world, code-mixed legal instruments.
Unified Evaluation Protocol: A novel protocol evaluating models across parametric, gold-context, and retrieval settings with a strict date-gated scoring mechanism.
Empirical Insights: The study demonstrates that current LLMs, even with access to source text, fail to reliably resolve versions, often trusting confident false premises over authoritative evidence.

5. Significance and Limitations

Significance:
The paper argues that version resolution is not solved by retrieval or long-context access alone. The persistent failures on version-sensitive tasks highlight a critical gap in LLM capabilities for high-stakes domains like law, tax, and compliance. The benchmark provides a foundation for developing models that can reliably reason over evolving documents, separating the ability to read text from the ability to resolve temporal conflicts.

Limitations:

Domain Specificity: The benchmark is currently limited to Bangladesh customs law and a specific language pair/calendar system.
Frozen Models: Evaluation is conducted on frozen models via prompting; it does not clarify if fine-tuning or continued pretraining would resolve these issues.
Retrieval Baseline: The RAG results use a standard semantic retriever without temporal re-ranking, meaning RAG scores represent a lower bound.
Parametric Uncertainty: Parametric scores rely on unverifiable assumptions about training data cutoffs.
Temporal Drift: As the corpus is set to 2025, the benchmark will require periodic updates to remain current.

The authors conclude that while reading the correct text is a prerequisite, the ability to identify the version in force and reject superseded information remains a significant, unsolved challenge for current LLMs.