Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citations

Technical Summary: Temporal Decay of Co-Citation Predictability in Legal Retrieval

Problem Statement
Legal information retrieval systems frequently rely on co-citation structures—the patterns of which statutory articles are cited together in court decisions—as a stable retrieval signal. The implicit assumption is that if articles A and B are frequently co-cited in historical decisions, they will remain relevant to similar future cases. This paper challenges the stationarity of this assumption. It investigates whether co-citation predictability remains stable over time or decays, potentially rendering graph-based retrieval methods obsolete without continual updating. The study focuses on the Ukrainian legal system, utilizing a massive corpus of court decisions to measure the temporal stability of statutory retrieval signals over a 20-year period (2007–2026).

Methodology
The authors constructed UA-StatuteRetrieval, a benchmark derived from the Ukrainian Unified State Register of Court Decisions (ЄДРСР). The dataset comprises 101 million court decisions and 396 million validated codex-article citations extracted from 13 legal codices.

• Evaluation Protocol: The study employs a leave-one-out citation prediction protocol over a full bipartite citation graph (cases vs. articles). For each decision, specific cited articles are masked, and the system attempts to recover them using the remaining citations. This serves as a proxy for statute retrieval, measuring the stability of the co-citation signal rather than the end-user task of retrieving statutes from case facts.
• Scoring Functions: The primary metrics are Mean Reciprocal Rank (MRR) and Hit@K, evaluated using:
  • Adamic-Adar (AA) and Common Neighbors (CN): Graph-based link prediction methods adapted from social network analysis.
  • Degree Baseline: A popularity-based retrieval method.
  • BM25: A text-based baseline using masked case text (stripped of citations) as a query against full statutory texts.
• Ablation Controls: To distinguish genuine temporal decay from compositional shifts (e.g., new articles entering the corpus), the authors implemented:
  1. Fixed-Article Ablation: Evaluating performance on a shared vocabulary of articles present in both 2012 and 2024.
  2. Temporal Train/Test Split: Building the co-citation matrix from the first 50% of cases (by document ID) and evaluating on the remaining 50% to prevent data leakage.
• Mechanism Analysis: The study utilizes multilingual E5-large embeddings to measure "embedding drift." By comparing the semantic centroids of case snippets citing specific articles in 2012 versus 2024, the authors quantify the semantic shift in how articles are applied.

Key Results
The central finding is that co-citation predictability decays significantly over time, confirming that legal citation structures are non-stationary.

1. Magnitude of Decay:

   • On a fixed set of articles, Adamic-Adar MRR declined by 33% (from 0.43 to 0.29) over the study period.
   • Under a strict temporal train/test split (no data leakage), the decline was even more severe at 47% (from 0.51 to 0.27).
   • The decline is monotonic after 2012, with no periods of recovery.

2. Non-Uniform Degradation:

   • Frequency: "Hub" articles (>100K citations) resist decay, maintaining MRR > 0.5. However, mid-frequency articles (1K–10K citations), which constitute the practical retrieval frontier, lose approximately 50% of their predictability (MRR dropping from ~0.15 to ~0.07).
   • Domain: Criminal Procedure maintains stable co-citation patterns (MRR ~0.40). In contrast, Civil Law degrades rapidly (MRR dropping from 0.35 to 0.15), coinciding with the 2017 judicial reform. Administrative law also shows significant degradation, particularly post-2022.

3. Text-Based Baseline Performance:

   • BM25 text retrieval performs significantly worse than co-citation methods in absolute terms (MRR 0.047 vs. 0.324 in 2024) but decays even faster (31% decline vs. 21% for co-citation). This challenges the assumption that text-based methods are inherently more temporally robust.

4. Mechanistic Explanation:

   • Embedding drift analysis reveals a 4.3% semantic shift in how articles are cited over 12 years.
   • Civil Procedure exhibits the highest semantic drift (0.046), directly mirroring its steepest retrieval degradation. This suggests that as the legal context and interpretive practice of an article evolve, both structural (co-citation) and lexical (BM25) signals weaken.

Significance and Claims
The paper claims to provide the first longitudinal benchmark demonstrating that legal retrieval signals are not static. Its significance lies in three areas:

• System Design Implications: Graph-based retrieval systems cannot be treated as static indexes. The observed decay rate (~4% MRR/year during 2012–2018) implies that such systems lose meaningful performance within 3–4 years without retraining. The authors suggest that hybrid systems combining graph and text modalities, or those utilizing continual retraining, are necessary to address the "mid-frequency" retrieval challenge where signals are weakest.
• Benchmarking Standards: The study reveals that standard leave-one-out evaluation protocols underestimate real-world degradation by approximately 5 percentage points due to data leakage. The authors advocate for temporal train/test splits as the default protocol for future legal retrieval benchmarks.
• Theoretical Contribution: By linking semantic embedding drift to retrieval performance decay, the paper offers a mechanistic explanation for why legal citation networks degrade: the evolving semantic context of legal articles renders historical co-citation patterns less predictive of future relevance.

The authors release the UA-StatuteRetrieval benchmark, including per-article difficulty scores, 20 annual snapshots, and embedding drift measurements, to enable the community to directly measure the temporal robustness of future retrieval approaches. The paper explicitly notes limitations, including that the task is citation prediction (a proxy) rather than full statute retrieval, and that results are specific to the Ukrainian civil-law system.