HUKUKBERT: Domain-Specific Language Model for Turkish Law

1. Problem Statement

The application of general-purpose Natural Language Processing (NLP) models to the Turkish legal domain faces significant challenges due to the unique characteristics of legal texts. The authors identify three primary limitations in existing general-domain Turkish models (such as BERTurk and TabiBERT):

• Lexical and Semantic Divergence (Semantic Shift): Legal texts utilize archaic terminology and specific jargon where common words acquire precise technical meanings. General models often fail to distinguish between colloquial terms and their legal equivalents (e.g., predicting "miras" [inheritance] instead of the legally precise "tereke" [estate]).
• Tokenization Fragmentation: Turkish is an agglutinative language. General-purpose tokenizers frequently break down complex legal compound phrases into excessive, semantically uninformative subwords. For instance, the phrase "İçtihadı Birleştirme Kararı" (Decision on the Unification of Judgments) is fragmented into seven subwords by generic models, whereas a domain-optimized tokenizer preserves it as three meaningful units. This fragmentation disrupts semantic coherence and artificially inflates sequence lengths.
• Syntactic Complexity: Turkish legal documents feature long, convoluted sentences and dense boilerplate expressions that general models, trained on conversational or journalistic data, fail to parse effectively for deep reasoning.

Currently, the Turkish legal NLP ecosystem lacks a high-volume, domain-specific language model comparable to English models like LEGAL-BERT, creating a bottleneck for LegalTech applications.

2. Methodology

The authors developed HukukBERT, a domain-adaptive pre-trained (DAPT) language model, through a rigorous four-stage pipeline:

A. Corpus Collection and Curation

• Data Source: A raw corpus of approximately 9 million documents (27.02 GB) was aggregated from official Turkish government sources, including the Supreme Court of Appeals (Yargıtay), Council of State (Danıştay), Constitutional Court (AYM), and the Ministry of Justice.
• Cleaning & Deduplication: Using MinHash Locality-Sensitive Hashing (LSH) with a 0.90 similarity threshold, nearly 2 million duplicate documents were removed, reducing the corpus to 24.6 GB.
• Sub-domain Balancing: To prevent overfitting to repetitive appellate rulings, the authors downscaled the dominant Yargıtay data and up-scaled underrepresented but semantically dense categories (legislation, academic theses, scholarly articles). The final balanced training corpus consists of 2.3 million documents totaling ~18.93 GB.

B. Domain-Optimized Tokenization

• Custom Vocabulary: A new 48K WordPiece tokenizer was trained from scratch on the balanced legal corpus.
• Seeding: The initialization vocabulary was seeded with verified terms from the Ministry of Justice Legal Dictionary to ensure critical legal terms are mapped to single discrete tokens.
• Efficiency: This tokenizer achieved an average of 4.82 subwords per line, a significant improvement over generic models (e.g., TabiBERT at 7.38 subwords/line), preserving morphological integrity.

C. Hybrid Masking Strategy (DAPT)

Instead of standard random masking, the authors implemented a hybrid masking strategy with an overall probability of 0.25 to force the model to learn deep legal concepts:

1. Whole Word Masking (20%): Masks all sub-tokens of a word simultaneously.
2. Token Span Masking (20%): Masks consecutive blocks of tokens (based on SpanBERT).
3. Word Span Masking (30%): Masks consecutive word blocks to learn multi-word idioms.
4. Keyword Masking (30%): Selectively masks tokens from a curated list of 40,000+ legal terms.

D. Training Infrastructure

The model underwent Domain-Adaptive Pre-Training (DAPT) for 2 epochs on an NVIDIA H200 environment using AdamW optimization, a peak learning rate of 1e-5, and a batch size of 960.

3. Key Contributions

1. HukukBERT Model: The first comprehensive, large-scale (18 GB) legal language model specifically for Turkish law.
2. Custom Tokenizer: A 48K WordPiece tokenizer that significantly reduces token fragmentation for Turkish legal morphology.
3. Hybrid DAPT Methodology: An innovative masking strategy combining whole-word, span, and keyword masking tailored to legal reasoning.
4. Hukuki Cloze Testi (Legal Cloze Test): A novel intrinsic benchmark consisting of 750 high-difficulty questions designed to evaluate legal domain adaptation and semantic precision in Turkish NLP.
5. Open-Source Release: The model, tokenizer, and benchmark are released to the community to accelerate Turkish LegalTech research.

4. Results

Intrinsic Evaluation (Legal Cloze Test)

HukukBERT was evaluated on the Legal Cloze Test against general-domain models (TabiBERT, BERTurk) and existing domain-specific attempts (BERTurk-Legal).

• Top-1 Accuracy: HukukBERT achieved 84.40%, outperforming the next best domain-specific model (BERTurk-Legal at 75.47%) by 8.93 percentage points.
• Comparison: It significantly surpassed general models like TabiBERT (68.13%) and BERTurk-cased (63.73%), proving that scaling general data does not substitute for domain adaptation.
• Qualitative Analysis: The model successfully resolved semantic shifts, correctly predicting precise legal terms (e.g., "tereke") where general models defaulted to colloquialisms (e.g., "miras").

Extrinsic Evaluation (Structural Segmentation)

The model was fine-tuned for the structural segmentation of official court decisions (a BIO token classification task) to identify headers, claims, reasoning, rulings, etc.

• Document Pass Rate: HukukBERT achieved a 92.8% document pass rate (where a single boundary error fails the whole document), outperforming BERTurk-Legal (81.9%) and BERTurk-128k (84.3%).
• Boundary Accuracy: Achieved 99.0% boundary accuracy.
• Convergence: HukukBERT converged faster and reached peak performance at epoch 3.67, whereas other models plateaued earlier or exhibited fluctuating performance.

5. Significance and Future Directions

• New State-of-the-Art: HukukBERT establishes a new baseline for Turkish legal NLP, demonstrating that specialized tokenization and DAPT are essential for handling the morphological and semantic complexities of Turkish law.
• LegalTech Enablement: By providing a robust foundation, the authors enable downstream applications such as Named Entity Recognition (NER), judgment prediction, and document classification, which were previously hindered by the lack of suitable models.
• Future Work: The authors plan to extend HukukBERT to ModernBERT architectures to support larger context windows (up to 8,192 tokens) for full-document analysis and to explore ColBERT architectures for fine-grained retrieval. They also aim to expand into generative tasks and multi-label classification.

In conclusion, this paper argues that for low-resource or morphologically complex domains like Turkish law, simply scaling general-domain data is insufficient. A combination of curated data, domain-specific tokenization, and targeted pre-training strategies is required to achieve state-of-the-art performance.