💬 NLP

Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions

This study reveals that legal language in Ukrainian court decisions undergoes significant temporal concept drift across geopolitical epochs, causing severe performance degradation in standard models when applied forward in time, while demonstrating that chronological continual learning and legal-domain pretraining can effectively mitigate this drift and prevent catastrophic forgetting.

Original authors: Volodymyr Ovcharov

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Volodymyr Ovcharov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student to predict the outcome of court cases. You give them a stack of old textbooks from 2008 to 2013 (a time of peace) and ask them to take a test today, in 2026, after a massive war and a complete overhaul of the legal system.

This paper is about what happens when you try to use that "old textbook" student to solve "new world" problems. The researchers found that the student fails spectacularly, not because they are stupid, but because the world they are studying has fundamentally changed.

Here is the breakdown of their findings using simple analogies:

1. The "Time Travel" Problem

The researchers tested AI models (computer programs that understand language) on Ukrainian court decisions from three distinct time periods:

Pre-war (2008–2013): A calm, stable time.
Hybrid War (2014–2021): A time of conflict and reform.
Full-Scale Invasion (2022–2026): A time of martial law and entirely new laws.

The Finding: If you train an AI on the "Pre-war" data and test it on "Full-scale invasion" data, its performance crashes. It's like teaching someone to drive on a quiet country road and then asking them to navigate a chaotic, war-torn city with new traffic laws they've never seen. The AI loses up to 27% of its accuracy.

2. The "One-Way Street" of Legal Knowledge

The most interesting discovery is that this failure isn't equal in both directions.

Forward (Old to New): If you train on old laws and test on new ones, the AI fails. It doesn't know about the new "martial law" rules or the new crimes created by the war.
Backward (New to Old): If you train the AI on the complex, chaotic "Full-scale invasion" data and test it on the simple "Pre-war" data, the AI actually gets better.

The Analogy: Think of legal language like a growing tree.

New laws are like new branches and leaves. They grow on top of the old ones.
If you study the whole tree (including the new branches), you can easily recognize the old trunk and lower branches.
But if you only study the old trunk, you have no idea what the new branches look like.
The researchers call this the "Additive Hypothesis": Law adds new things; it rarely deletes the old things entirely. Therefore, a model trained on the "newer, bigger" dataset understands the "older, smaller" dataset perfectly, but the reverse is impossible.

3. The "Specialist" vs. The "Generalist"

The researchers tried two types of AI:

Generalists: Standard AI models trained on all kinds of text.
Specialists: AI models pre-trained specifically on legal documents.

The Surprise: You might think the "Specialist" would be better. But the "Generalist" actually performed better overall.

The Specialist was more stable (it didn't crash as hard when moving from old to new data), but it was just generally worse at the task to begin with. It was like a specialist who knows the rules of the game but is terrible at playing it.
The Generalist was a better player but suffered more when the rules changed.

4. The "Foreign Teacher" Experiment

They tried to see if teaching the AI with data from a different country (Switzerland) would help it understand Ukrainian law better.

Result: It helped the AI get slightly smarter overall (like giving it a better dictionary), but it did not fix the problem of time. The AI still failed badly when moving from old Ukrainian data to new Ukrainian data.
Takeaway: The problem isn't that the AI lacks knowledge of other countries; the problem is that the language of law itself evolves over time in a way that foreign data can't predict.

5. The Solution: "Chronological Retraining"

How do you fix a student who keeps failing because the world changed? You don't throw away the old textbooks and start fresh. You update them.

The researchers tested Continual Learning:

The Right Way (Chronological): Train the AI on Pre-war, then add Hybrid war data, then add Full-scale data.
- Result: The AI remembers the old laws perfectly and learns the new ones. It becomes a super-expert that knows the whole history.
The Wrong Way (Reverse): Train on Full-scale, then try to go back to Hybrid, then Pre-war.
- Result: The AI "forgets" the complex new laws and crashes. It's like trying to un-learn advanced calculus to go back to basic math; the brain gets confused and loses the advanced knowledge.

Summary

This paper proves that in the legal world, time matters more than the model you choose.

If you build a legal AI today using old data, it will become unreliable very quickly.
The best strategy is to keep the AI updated chronologically, feeding it new data as it arrives, so it builds its knowledge like a growing tree rather than a static snapshot.
The "Specialist" legal models didn't save the day; in fact, the standard models, when updated correctly, worked best.

The researchers have released their data (428,000 court decisions) so others can test this "time travel" problem in their own systems.

Technical Summary: Temporal Concept Drift in Legal Judgment Prediction

Problem Statement

Current Legal NLP benchmarks (e.g., LexGLUE, LEXTREME, SCALE) predominantly rely on random train/test splits, implicitly assuming that legal language is stationary. This design obscures the reality of temporal concept drift, where model performance degrades as the temporal distribution of test data diverges from training data. In the legal domain, this drift is amplified by legislative changes, evolving judicial practices, and exogenous shocks (e.g., conflicts, reforms). While previous work established that classical TF-IDF baselines suffer significant degradation across Ukrainian court decision epochs, the extent of this phenomenon in neural transformer models and the efficacy of domain-specific pretraining or continual learning strategies remained unexplored.

Methodology

Dataset and Temporal Epochs

The study utilizes 428,000 Ukrainian court decisions (civil and commercial jurisdictions) extracted from the Unified State Register of Court Decisions (EDRSR). The data is partitioned into three distinct temporal epochs reflecting geopolitical disruptions:

Pre-war (2008–2013): Peacetime baseline (128k decisions).
Hybrid war (2014–2021): Post-Crimea annexation and judicial reform (150k decisions).
Full-scale invasion (2022–2026): Martial law and new procedural frameworks (150k decisions).

Within each epoch, splits are chronological (80% train, 10% val, 10% test) to prevent temporal leakage. Labels are classified into three categories: Approved, Dismissed, and Partial.

Models and Experimental Design

Four transformer variants based on XLM-RoBERTa were fine-tuned to isolate the effects of model scale and domain pretraining:

General-purpose: XLM-R Base (278M) and Large (560M).
Legal-domain: Legal-XLM-R Base and Large (pretrained on MultiLegalPile).

The evaluation protocol consists of four experiments:

In-epoch Baselines: Training and testing on the same epoch to establish diagonal performance.
Cross-epoch Generalization: A $3 \times 3$ matrix where models trained on one epoch are tested on all three. Metrics include Forward Degradation (Pre-war $\to$ Full-scale) and Backward Degradation (Full-scale $\to$ Pre-war).
Cross-Jurisdictional Transfer: Using Swiss Judgment Prediction (SJP) data to test if foreign-jurisdiction pretraining mitigates temporal drift.
Continual Learning: Sequential fine-tuning in two directions:
- Forward (Chronological): Pre-war $\to$ Hybrid $\to$ Full-scale.
- Backward (Reverse-chronological): Full-scale $\to$ Hybrid $\to$ Pre-war.

Key Results

1. Severe and Asymmetric Forward Degradation

Temporal drift is a dominant source of performance degradation, exceeding the impact of model selection or domain pretraining.

Forward Degradation: Models trained on pre-war data lose 11.2 to 27.2 percentage points (pp) in macro-F1 when applied to full-scale invasion decisions. The largest model (XLM-R Large) suffered a 27.2 pp drop.
Asymmetry: Backward transfer is significantly more robust. Models trained on the complex full-scale era often perform better on pre-war test data than on their own training epoch (e.g., XLM-R Base gains +9.0 pp when moving backward). This supports the "additive language" hypothesis: newer legal frameworks subsume older ones, but older models lack representations for new concepts (e.g., martial law provisions).

2. The Paradox of Legal-Domain Pretraining

Contrary to expectations, legal-domain pretraining (Legal-XLM-R) did not improve absolute performance compared to general XLM-R.

Absolute Performance: Legal-XLM-R models scored 7–9 pp lower than general models on in-epoch tasks.
Temporal Robustness: Legal models exhibited less forward degradation (11–12 pp vs. 20–27 pp) and a smaller asymmetry gap. However, this "robustness" stems from lower baseline discriminative power; their representations are less epoch-specific but also less effective for the task.
Directional Fragility: Unlike general models, legal models degraded in both forward and backward directions, suggesting their representations are brittle to any temporal shift.

3. Continual Learning Dynamics

Chronological (Forward) Training: For general XLM-R models, sequential fine-tuning (Pre-war $\to$ Full-scale) eliminated catastrophic forgetting. Pre-war knowledge was retained or improved (+1.8 to +6.2 pp), while full-scale performance gained +16.5 to +19.0 pp. The resulting model outperformed any single-epoch model.
Reverse-Chronological Training: This direction caused severe catastrophic forgetting (−12.2 to −14.3 pp on full-scale data).
Legal Models: Legal-XLM-R failed to retain knowledge even in the forward direction, losing 5.6 pp on pre-war data.

4. Cross-Jurisdictional Transfer Limits

Pretraining on Swiss Judgment Prediction data improved absolute performance across all epochs (+3 to +10 pp) but did not reduce the magnitude of temporal degradation. The forward degradation gap remained virtually identical (20.3 pp vs. 21.3 pp), confirming that temporal drift is an intrinsic property of legal language evolution, not a jurisdiction-specific artifact.

5. Neural vs. Classical Baselines

Neural models (mean F1 ~63–65%) substantially underperformed the TF-IDF baseline (mean F1 ~79.8%). The authors attribute this to tokenizer fertility penalties on Ukrainian Cyrillic text, where SentencePiece fragments legal terms into 3–5 subwords, reducing effective context within the 512-token window.

Significance and Claims

The paper establishes the first neural temporal robustness benchmark for legal NLP. Its primary contributions are:

Quantifying Drift: Demonstrating that temporal drift is a more critical factor for model reliability than model architecture or domain pretraining.
Mechanistic Insight: Validating the "additive language" hypothesis through asymmetric transfer results and continual learning trajectories.
Mitigation Strategy: Identifying chronological continual learning as an effective strategy for general-purpose models to maintain historical knowledge while adapting to new legal frameworks, whereas reverse-chronological training is detrimental.
Benchmarking Standards: Arguing that legal NLP benchmarks must report temporal robustness alongside aggregate performance to reflect real-world deployment risks.
Resource Release: Providing a publicly available dataset of 428k decisions with temporal annotations as a contribution to LEXTREME.

The authors conclude that for deployed legal NLP systems, periodic chronological retraining on recent data is essential, and reliance on legal-domain pretraining may be counterproductive for jurisdiction-specific tasks requiring high discriminative power.