Technical Summary: UA-Legal-Bench

Problem Statement

Current legal Natural Language Processing (NLP) benchmarks are overwhelmingly English-centric, creating a significant blind spot for morphologically rich, non-Latin script languages and civil-law jurisdictions. Existing benchmarks like LegalBench, LexGLUE, and CUAD evaluate reasoning exclusively in English, often within common-law frameworks. This gap is critical for Ukrainian legal NLP due to three compounding factors:

1. Tokenization Inefficiency: Subword tokenizers fragment Ukrainian legal text at rates 1.6× higher than English, inflating inference costs and degrading in-context learning.
2. Morphological Complexity: Ukrainian's agglutinative morphology and seven grammatical cases present a fundamentally different tokenization surface compared to English.
3. Jurisdictional Differences: The Ukrainian civil-law system relies on codified statutes rather than common-law precedents, requiring distinct reasoning patterns not captured by English benchmarks.

Prior to this work, no comprehensive benchmark existed for Ukrainian legal reasoning, and general Ukrainian NLP resources are scarce (constituting only 0.5% of the mC4 corpus).

Methodology

Data Source and Construction

The benchmark is constructed from the Unified State Register of Court Decisions (EDRSR), an open judicial corpus containing 99.5 million decisions published since 2006. The authors sampled 2,000 substantive decisions from the 2024 partition, stratified across four jurisdictions (civil, criminal, commercial, administrative) and judgment forms. The dataset includes full decision text (median length: 12,060 characters) and structured metadata.

Benchmark Tasks

UA-Legal-Bench comprises five tasks of increasing difficulty:

1. Case-Type Classification (CTC): Classifying decisions into four jurisdictional categories (n=2,000).
2. Judgment Form Classification (JFC): Identifying the document type (decision, resolution, sentence, ruling) based on procedural nature (n=2,000).
3. Case-Outcome Prediction (COP): Predicting the judicial ruling (6 classes) from the facts section alone, with the ruling masked (n=800).
4. Norm Extraction (NE): Extracting legal norm citations (e.g., specific article numbers) from the text (n=1,794).
5. Cause Category Prediction (CCP): Classifying the legal subject matter into 22 macro-categories (n=1,871).

Evaluation Protocol

The study evaluated 11 Large Language Models (LLMs) ranging from 3B to 675B parameters across five families (Mistral, Meta, NVIDIA, Amazon, Qwen) via AWS Bedrock.

• Conditions: Models were tested under zero-shot and 3-shot prompting.
• Metrics: Accuracy was used for balanced tasks; Macro-F1 was prioritized for imbalanced tasks (specifically COP, where the majority class "granted" comprises 61% of the data) to prevent majority-class bias.
• Volume: The evaluation involved 158,419 API calls.

Key Results

1. Task Difficulty Gradient

Tasks exhibit a clear difficulty gradient. CTC is nearly solved, with all frontier models exceeding 96% zero-shot accuracy. JFC is moderately difficult (74–84% zero-shot), while COP is the most challenging, with frontier models achieving only 23–41% Macro-F1 zero-shot. NE and CCP fall in between.

2. Task-Dependent Few-Shot Effects

The impact of few-shot prompting is not uniform; it varies sharply by task:

• JFC: Few-shot prompting consistently improves performance, with gains up to +38.6 percentage points (pp) for a 12B model (Nemotron Nano), nearly matching the performance of a 120B model. This suggests JFC relies heavily on learning document format signatures.
• COP: Effects are mixed. While some models improve, others (e.g., Llama 3.3) degrade. Crucially, accuracy is a misleading metric here; the model with the highest COP accuracy (62%) was a majority-class predictor with a Macro-F1 of only 23%. The genuinely best model (Nova Pro) achieved 44% Macro-F1.
• CTC: Few-shot effects are minimal due to a ceiling effect (zero-shot accuracy >96%).
• NE: Few-shot prompting has negligible effects, suggesting norm extraction relies on pattern recognition rather than deep reasoning.

3. Scaling and Model Families

Scaling laws for Ukrainian legal reasoning are highly family-dependent:

• Mistral Family: Shows "flat" scaling. An 8B model (Ministral 8B) matches the 675B frontier model on CTC and nearly matches it on COP, suggesting 8B parameters are sufficient for many legal tasks in this family.
• Meta Family: Shows steep scaling. The 8B model (Llama 3.1) performs poorly compared to its 70B and 400B counterparts, indicating a much higher parameter threshold is required for competence in this family.
• NVIDIA Family: Shows intermediate scaling, with significant gaps between 12B and 120B models on COP, though few-shot prompting can partially bridge this gap on format-based tasks.

4. Metric Sensitivity

The study highlights that Macro-F1 is essential for legal benchmarks with imbalanced classes. Relying on accuracy alone would incorrectly identify a majority-class predictor as the top-performing model.

Significance and Claims

The paper positions UA-Legal-Bench as the first comprehensive legal reasoning benchmark for a Cyrillic-script, civil-law jurisdiction. Its primary contributions and claims are:

1. Filling a Critical Gap: It addresses the exclusion of Cyrillic jurisdictions and civil-law systems from existing legal NLP evaluations.
2. Revealing Metric Pitfalls: It demonstrates that accuracy is dangerously misleading on imbalanced legal tasks, advocating for class-aware metrics like Macro-F1.
3. Task-Dependent Prompting: It establishes that few-shot prompting is not a universal solution; its efficacy depends on the ratio of "format signal" to "length cost" (tokenization inefficiency).
4. Scaling Nuance: It challenges the assumption that legal AI universally requires frontier-scale models, showing that 8B models can match frontier performance on surface-level tasks (CTC, NE) but that the parameter threshold for complex reasoning varies dramatically across model families.
5. Resource Availability: The authors release all data, prompts, model predictions, and evaluation code to facilitate further research in multilingual legal NLP.

The authors conclude that the observed patterns—specifically the misleading nature of accuracy in imbalanced tasks and the variable efficacy of few-shot prompting—are invisible in English-only, accuracy-only evaluations, underscoring the necessity of multilingual benchmarks with appropriate metrics.