Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions

Technical Summary: Multi-Legal-Bench

Problem Statement
Current legal Natural Language Processing (NLP) benchmarks suffer from a "siloed" evaluation design. Existing resources like LegalBench, LexGLUE, and LEXTREME either focus exclusively on English common-law systems or aggregate fundamentally different tasks across languages (e.g., comparing German topic classification to French court-type prediction). This design prevents true cross-lingual comparison because performance differences conflate language ability with task difficulty. Furthermore, no prior benchmark evaluates identical reasoning tasks across multiple national legal systems using frontier Large Language Models (LLMs), leaving a critical gap in understanding how models generalize across jurisdictions and language families.

Methodology
The authors introduce Multi-Legal-Bench, a cross-jurisdictional benchmark constructed from the SecondLayer Legal Corpus, which contains 134 million court decisions from 20 jurisdictions. The benchmark focuses on six civil-law jurisdictions: Ukraine, France, the Netherlands, Poland, the Czech Republic, and Lithuania. These were selected to span four language families (Slavic, Romance, Germanic, Baltic) and two scripts (Latin, Cyrillic).

The benchmark defines five identical legal reasoning tasks, mapped to structured metadata from national court registries:

1. Court-Type Classification (CTC): Classifying the judicial branch (e.g., civil, criminal).
2. Judgment Form Classification (JFC): Identifying the procedural document type.
3. Case-Outcome Prediction (COP): Predicting the judicial result given only the facts section.
4. Norm Extraction (NE): Extracting legal norm citations.
5. Cause Category Prediction (CCP): Classifying the legal subject matter.

Due to varying metadata richness across jurisdictions, the evaluation forms a deliberately sparse 5×6 task–jurisdiction matrix (20 of 30 cells filled). The authors explicitly avoid fabricating labels or using pseudo-labels; empty cells represent genuine metadata absence.

Experimental Setup
The study evaluates 11 LLMs via AWS Bedrock:

• 7 Frontier Models: Including Llama 4 Maverick, Llama 3.3 70B, Mistral Large 3, Nemotron Super 3, Amazon Nova Pro, and Qwen3 variants (235B and 32B).
• 4 Small/Medium Models: Ministral 3B/8B, Llama 3.1 8B, and Nemotron Nano 12B, used for scaling analysis.

Evaluation protocols include zero-shot and 3-shot prompting in the native language of each jurisdiction. Additionally, the authors conducted cross-lingual transfer experiments, testing whether Ukrainian few-shot examples could improve performance on non-Ukrainian test data.

Key Results

1. Task-Dependent Few-Shot Effects: The finding that few-shot learning effects are task-specific (discovered in prior Ukrainian-only work) replicates across all jurisdictions. JFC consistently benefits from few-shot prompting (gains up to +66.2 pp), while COP shows mixed, unpredictable effects, and CTC often reaches ceiling performance.
2. No Dominant Model: No single model dominates across all languages or tasks. Rankings shift significantly based on the specific task and jurisdiction. For instance, a model performing well on French legal text may perform poorly on Polish or Czech text.
3. Cross-Lingual Transfer and Label Alignment: Transfer quality does not correlate with language proximity. Ukrainian-to-French (Romance) transfer outperformed Ukrainian-to-Polish (Slavic) transfer on Court-Type Classification (−2.1 pp drop vs. −13.7 pp drop). The authors conclude that label-set alignment is a better predictor of transfer quality than linguistic similarity.
4. Tokenizer Fertility vs. Accuracy: While tokenizer fertility (tokens per word) varies significantly (2.3× spread) and correlates with inference cost, it does not significantly predict cross-lingual accuracy (r=−0.27, p=0.14). This contrasts with prior monolingual findings, suggesting that pretraining data composition and model architecture dominate over tokenizer efficiency in multilingual settings.
5. Scaling Gaps: Small models (3–12B) show significant performance degradation compared to frontier models, particularly on Court-Type Classification in under-represented languages (e.g., a −20 pp gap for Lithuanian). However, they occasionally outperform frontier models on specific tasks like Cause Category Prediction in the Netherlands.

Significance and Claims
The paper claims to provide the first legal benchmark that enables true cross-lingual comparison by evaluating identical tasks across multiple national legal systems. Its primary contributions are:

• Methodological Rigor: By using a sparse matrix based on real-world metadata availability, the benchmark honestly reflects the heterogeneity of open judicial data rather than forcing artificial uniformity.
• Empirical Insights: It challenges the assumption that language proximity drives cross-lingual transfer, highlighting the importance of label-set compatibility. It also decouples tokenizer efficiency from model performance in multilingual contexts, advising practitioners to prioritize jurisdiction-specific evaluation over tokenizer statistics.
• Resource Availability: The authors release all data, prompts, model predictions, and evaluation code to facilitate further research in multilingual legal NLP.

The authors maintain a modest stance, noting limitations such as the exclusion of common-law systems, imbalanced label distributions in specific cells, and the focus on generative LLMs rather than encoder models. They emphasize that their findings regarding task difficulty stability and the non-universality of model rankings are specific to the evaluated civil-law jurisdictions.