Technical Summary: LexKairos

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in various legal tasks such as judgment prediction and case retrieval, existing legal AI benchmarks largely overlook the temporal dimension. Legal practice is inherently time-conditioned: statutes have specific effective periods and retroactivity rules, and legal claims depend on the chronology of factual events and procedural deadlines. Current evaluations focus on semantic and logical aspects ("what" and "how") but fail to assess the "when" that governs the validity and enforceability of legal rights. Furthermore, prior work on legal temporal capabilities has been limited to isolated competencies like event ordering or timeline extraction, lacking a unified framework that integrates statutory temporal knowledge with factual case chronologies.

Methodology

To address this gap, the authors propose LexKairos, a comprehensive benchmark designed to evaluate the temporal capabilities of LLMs within the Chinese legal context. The benchmark is structured around three complementary dimensions of "Legal Time":

1. Statutory Temporal Knowledge (Statute-centric): Evaluates the model's ability to recall temporal metadata and rules associated with laws.
   • Sub-tasks: Effective Date Inquiry, Effective Version Identification, and Effective Status Verification.
   • Data: Automatically synthesized from statutory metadata using a rule-based pipeline.
2. Case Temporal Modeling (Fact-centric): Evaluates the capacity to process and structure timelines within case facts.
   • Sub-tasks: Event Relation Discerning (based on Allen's interval relations), Temporal Distance Calculation, Global Event Ordering, and Bounded Event Retrieval.
   • Data: Constructed from verified civil cases using LLM-assisted event extraction and SQL-based question generation.
3. Statute-Case Temporal Reasoning (Hybrid Integration): Evaluates the ability to apply statutory temporal rules to concrete case scenarios.
   • Sub-tasks: Peremptory Period Reasoning (absolute deadlines) and Action Limitation Reasoning (statutes of limitation).
   • Data: Curated through a multi-stage hybrid pipeline involving LLM extraction, rule-based filtering, and expert legal review.

The benchmark comprises nine sub-tasks derived from real-world Chinese judicial cases and statutes. The authors conducted systematic evaluations on eight LLMs (including GPT-5.4, Gemini-3-Flash, DeepSeek-R1, Qwen3.5, and legal-specific models like LegalOne-8B) under three inference settings:

• Vanilla: Direct prompting without explicit reasoning steps.
• Chain-of-Thought (CoT): Explicit "Let's think step by step" prompting for models without native reasoning modes.
• Thinking Mode: Enabling built-in reasoning capabilities for models that support them.

Key Contributions

1. LexKairos Benchmark: The first dedicated benchmark for systematically evaluating legal temporal capabilities in the Chinese legal domain, covering three dimensions across nine sub-tasks.
2. Systematic Evaluation: A comprehensive assessment of eight state-of-the-art LLMs, revealing performance disparities across different temporal capabilities and inference settings.
3. Error Analysis: A granular analysis of failure modes, particularly in the Effective Version Identification sub-task (1-2), identifying "tag only" errors (incorrect version identifiers) as the dominant failure mode. The study highlights distinct cross-model error profiles (e.g., omission vs. hallucination) and how thinking modes shift these distributions.
4. Efficiency Insights: An investigation into the trade-offs between unconstrained thinking modes and task-specific legal prompting, demonstrating that structured prompts can achieve competitive performance with significantly reduced token usage (up to 3.4× shorter) and fewer truncation failures.

Results

• Overall Performance: Closed-source models generally outperform open-source models. Gemini-3-Flash (with thinking mode) achieved the highest overall score (84.57), followed by GPT-5.4 (82.75).
• Capability Disparities: Models consistently performed better on Case Chronology Analysis (Task 2) than on Statutory Lifecycle Tracking (Task 1) or Procedural Time Limit Reasoning (Task 3). This suggests LLMs are more proficient at modeling temporal information embedded in case facts than at recalling statutory metadata or applying complex procedural rules.
• Impact of Reasoning Modes: Enabling Thinking Mode yielded consistent performance gains, particularly for models with weaker vanilla performance (e.g., LegalOne-8B saw a +106.61% relative gain). However, even the best-performing models exhibited notable limitations on tasks requiring precise statutory metadata recall or complex procedural reasoning.
• Error Patterns: In the Effective Version Identification sub-task (1-2), "tag only" errors accounted for 85.20% of failures on average. Under vanilla inference, GPT-5.4 was dominated by "missing tag" errors (omission), while LegalOne-8B was dominated by "hallucinated tag" errors (fabrication). Activating thinking mode shifted these distributions but did not eliminate the underlying precision deficit.
• Truncation Issues: For models like Qwen3.5-9B, unconstrained thinking modes led to significant output truncation (up to 22.78% of responses in Task 3), drastically reducing accuracy. Task-specific legal prompting mitigated this by producing more concise outputs.

Significance and Claims

The paper claims that legal temporal knowledge and reasoning remain open challenges for current LLMs. Despite the availability of advanced reasoning modes, models struggle with the precise recall of statutory version identifiers and the complex integration of statutory rules with factual chronologies. The authors emphasize that while thinking modes improve performance, they do not resolve the fundamental precision deficits in legal version identifier recall.

The significance of LexKairos lies in its ability to expose these specific temporal weaknesses, moving beyond general legal reasoning to probe the "when" of legal analysis. The work suggests that future legal AI systems require stronger temporally aware reasoning capabilities, potentially through structured prompting or specialized training, rather than relying solely on unconstrained reasoning chains which may suffer from token budget constraints and hallucination. The authors note limitations in their work, including the relatively modest size of the Procedural Time Limit Reasoning dataset due to the labor-intensive nature of expert annotation and the current restriction of Global Event Ordering to day-level granularities.