Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study

Technical Summary: Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text

Problem Statement
Foundation models exhibit significant disparities in efficiency and performance when processing Ukrainian legal text compared to English. Ukrainian, a morphologically rich language using the Cyrillic script, often suffers from suboptimal tokenization, leading to inflated token counts, higher inference costs, and reduced effective context windows. While general-domain evaluations exist, there is a lack of systematic comparison regarding how different foundation models perform on specific, high-stakes legal tasks within the Ukrainian domain. Practitioners building legal technology platforms face uncertainty regarding model selection, as model size (parameter count) may not correlate with performance on non-English, domain-specific tasks.

Methodology
The study, designated as "Experiment A" of the LEX AI Test Training program, evaluated seven foundation models from five providers (Meta, Mistral AI, NVIDIA, Amazon, and Qwen) accessed via AWS Bedrock. The evaluation utilized a corpus of 300 authentic court decisions from Ukraine's Unified State Register of Court Decisions (EDRSR), stratified across civil, criminal, commercial, and administrative jurisdictions. After rigorous label validation using a three-source majority vote (regex parser, Claude Sonnet 4.5, and NVIDIA Nemotron Super 3), a validated subset of 273 documents was used for final metrics.

The evaluation measured three primary dimensions:

1. Tokenizer Fertility: The ratio of subword tokens to whitespace-delimited words on 6,000-character document samples.
2. Task Performance: Zero-shot and few-shot performance on three tasks:
   • Case Type Classification: 4-class jurisdictional categorization.
   • Case Outcome Classification: 5-class prediction of case disposition (e.g., granted, denied).
   • Legal Norm Extraction: Structured JSON extraction of cited laws and articles.
3. Cost Analysis: Total API costs incurred during inference, including ablation studies on prompt sensitivity and example distribution.

Key Contributions

1. Tokenizer Fertility Benchmarking: The paper quantifies a 1.6× variance in tokenizer efficiency across models. It demonstrates that tokenizer design is a vocabulary-level choice independent of model scale, with Llama-family models being significantly more efficient than Qwen models on Ukrainian text.
2. Performance vs. Scale Decoupling: The study provides empirical evidence that total parameter count is a poor predictor of performance on Ukrainian legal tasks. A 120B-parameter model (Nemotron Super 3) outperformed a 675B-parameter model (Mistral Large 3) across all tasks at a fraction of the cost.
3. The Few-Shot Degradation Effect: The research documents a counterintuitive phenomenon where few-shot prompting degrades performance on Ukrainian legal classification tasks. For several models, providing examples reduced accuracy by up to 26 percentage points. Ablation studies confirm this is intrinsic to the morphological complexity of Ukrainian demonstrations rather than an artifact of example selection or prompt formulation.
4. Cost-Performance Optimization: The authors provide a composite ranking and a proposed multi-model routing architecture that optimizes for both quality and cost, suggesting that different models should be deployed for different sub-tasks within a legal pipeline.

Key Results

• Tokenizer Efficiency: Llama 4 Maverick achieved the highest efficiency (2.43 tokens/word), while Qwen3 models were the least efficient (3.90 tokens/word), consuming 60% more tokens for identical input.
• Model Performance:
  • NVIDIA Nemotron Super 3 achieved the highest composite score (83.1) and the best zero-shot accuracy on case outcome classification (96.0%), outperforming Mistral Large 3 (675B total parameters) despite having significantly fewer active parameters.
  • Llama 4 Maverick tied for the best case type classification (98.9%) at the lowest cost ($0.81 for the entire experiment).
  • Llama 3.3 70B performed best on legal norm extraction (F1 = 0.604), suggesting dense architectures may excel at structured extraction tasks.
• Few-Shot Impact: Few-shot prompting resulted in performance degradation for 12 out of 21 model-task combinations. The most severe drop was observed in Qwen3 235B on case outcome classification (−26.0 percentage points). Stratified ablation confirmed that even distribution-matched examples failed to improve performance, supporting the hypothesis of morphological interference.
• Cost Efficiency: The routed ensemble (using Llama 4 Maverick for classification, Nemotron for outcomes, and Llama 3.3 for extraction) achieved a higher composite score (85.1) at 37% lower cost than using Nemotron alone.

Significance and Claims
The paper claims that for practitioners deploying foundation models on morphologically rich, Cyrillic-script languages like Ukrainian, standard heuristics such as "larger models are better" or "few-shot is always superior" are invalid. The authors assert that:

1. Tokenizer analysis must precede model selection to avoid unnecessary cost inflation and context window reduction.
2. Zero-shot prompting should be the default baseline for Ukrainian legal tasks, as few-shot examples may actively interfere with model performance due to the language's rich morphology.
3. Model selection should be task-specific. No single model dominates all legal NLP tasks; a routing strategy that leverages the specific strengths of different models (e.g., using a cheaper, efficient model for simple classification and a more capable model for complex reasoning) offers the optimal balance of quality and cost.
4. Comprehensive, language-specific benchmarking is feasible and affordable, with the total cost of this extensive evaluation (including ablations) amounting to approximately $60.

The study concludes that English-centric benchmarks and parameter counts are insufficient proxies for performance in non-English legal domains, urging the community to prioritize domain-specific, language-aware evaluation before deployment.