Technical Summary: Mecellem Models

Problem Statement

Large Language Models (LLMs) trained predominantly on English-centric data exhibit performance degradation in non-English languages and highly formal domains. The Turkish legal domain presents specific challenges due to its morphological richness (agglutinative structure), complex syntactic variations, intensive terminology, and strict normative constraints. Existing approaches for Turkish legal NLP have largely relied on traditional machine learning, transformer-based classification, or parameter-efficient fine-tuning (PEFT) of small models. However, these methods often prove insufficient for deep domain adaptation, failing to capture the nuanced linguistic patterns and long-context reasoning required for effective legal information retrieval and generation. Furthermore, standard pre-training objectives (e.g., Masked Language Modeling loss) do not always correlate with downstream embedding utility, particularly for morphologically rich languages.

Methodology

The paper introduces Mecellem, a framework for developing specialized Turkish legal language models through two complementary approaches:

1. Encoder Models: Pre-trained from Scratch

The authors developed bidirectional encoder models based on the ModernBERT architecture, pre-trained entirely from scratch on a curated Turkish-dominant corpus.

• Data: The corpus consists of 112.7 billion tokens, combining Turkish legal texts (Court of Cassation, Council of State, academic theses) with general web data (FineWeb2, CulturaX). Data underwent rigorous preprocessing, including VLM-based OCR extraction (DotsOCR) for scanned documents, semantic deduplication (SemHash), and Turkish-specific quality filtering based on suffix entropy and lemma diversity.
• Architecture: Two model sizes were trained: Base (155M parameters) and Large (403M parameters). Both utilize alternating local and global attention, RoPE positional embeddings, and support sequence lengths up to 8,192 tokens.
• Training Strategy:
  • Pre-training: Used Masked Language Modeling (MLM) with a 30% masking rate (80-10-10 strategy).
  • Checkpoint Selection: Instead of relying solely on minimizing MLM loss, the authors implemented a strategy evaluating downstream retrieval performance throughout training. This revealed that optimal checkpoints for retrieval often occur before the pre-training loss reaches its minimum.
  • Post-training: The encoders were converted into embedding models using contrastive learning on the MS MARCO-TR dataset. Techniques included standard InfoNCE, Qwen3-style InfoNCE, and GISTEmbed with cached guide models (BGE-M3, EmbeddingGemma-300M) for false-negative filtering.

2. Decoder Models: Continual Pre-training (CPT)

The authors adapted existing Qwen3-1.7B and Qwen3-4B decoder models to the Turkish legal domain via Continual Pre-training.

• Strategy: A four-phase curriculum learning approach was employed for the 1.7B model, progressing from general Turkish texts to domain-specific legal content, long normative documents, and finally extended domain refinement. The 4B model underwent a single-phase large-scale CPT.
• Catastrophic Forgetting Mitigation: The curriculum structure and the inclusion of a replay buffer (re-introducing Phase 1 data in Phase 2) were used to preserve general language capabilities while acquiring legal expertise.
• Evaluation: Performance was measured via perplexity on the EuroHPC-Legal dataset and evaluated using Muhakim, a domain-specific reward model trained to assess statutory accuracy, legal reasoning, and citation quality.
• Decoder-to-Encoder Conversion: The CPT decoder models were converted to embedding models by removing the language modeling head, enabling bidirectional attention, and applying mean pooling.

Key Contributions

1. Scratch Pre-training for Turkish Legal Embeddings: The creation of ModernBERT-based encoders pre-trained from scratch on 112.7B tokens, demonstrating that domain-specific pre-training from scratch is effective for Turkish legal tasks.
2. Downstream-Aware Checkpoint Selection: A novel finding that for morphologically rich languages, the optimal checkpoint for downstream retrieval performance does not necessarily coincide with the minimum pre-training loss. Intermediate checkpoints often yield superior embedding quality.
3. Controlled Continual Pre-training: A successful application of CPT to Qwen3 models using a four-phase curriculum, achieving significant perplexity reductions (up to 43.1% for the 1.7B model) while mitigating catastrophic forgetting.
4. Turkish-Specific Data Engineering: Development of a robust pipeline for extracting and cleaning Turkish legal data, including VLM-based OCR for complex documents and morphology-driven quality filters (suffix entropy, lemma diversity).
5. Comprehensive Benchmarking: Introduction of MTEB-Turkish, an evaluation framework extending MTEB with Turkish-specific tasks and three domain-specific legal retrieval benchmarks (Contracts, Regulation, Caselaw).

Results

• Encoder Performance:
  • The Mursit-Base-TR-Retrieval (155M) and Mursit-Large-TR-Retrieval (403M) models achieved top-3 rankings on the Turkish retrieval leaderboard.
  • Despite having significantly fewer parameters than state-of-the-art models (e.g., EmbeddingGemma-300M, BGE-M3), the Mecellem models achieved comparable performance. The 155M model achieved a 55.86 MTEB Score and 47.52 Legal Score, outperforming larger converted decoder models.
  • The models achieved 92.36% production efficiency relative to SOTA models, ranking fourth overall while requiring fewer computational resources.
• Decoder Performance:
  • CPT resulted in a 36.2% perplexity reduction on Turkish legal text for the Qwen3-4B model and up to 43.1% for the Qwen3-1.7B model.
  • Evaluation with the Muhakim reward model showed that CPT models consistently outperformed base models in legal accuracy, statute reference, and depth of coverage, particularly under short-context constraints.
• Ablation Insights:
  • Sequence Length: Optimizing sequence length solely based on training data distribution (e.g., 256 tokens) caused significant performance degradation (up to 23.6% in regulation retrieval) compared to longer sequences (2048 tokens), highlighting the need for long-context understanding in legal domains.
  • False-Negative Filtering: Using GISTEmbed with guide models improved legal retrieval scores by 15.9% over standard InfoNCE.
  • Architecture: Purpose-built encoder models pre-trained from scratch outperformed decoder-to-encoder converted models, which struggled with legal retrieval tasks despite larger parameter counts.

Significance and Claims

The paper claims that effective Turkish legal language models can be developed through domain-specific pre-training from scratch and controlled continual pre-training, provided that training strategies are tailored to the linguistic complexities of Turkish.

Key claims include:

• MLM Loss is Not a Proxy for Embedding Quality: For morphologically rich languages like Turkish, minimizing pre-training loss does not guarantee optimal downstream embedding performance. Direct evaluation of downstream tasks is essential for checkpoint selection.
• Parameter Efficiency: Smaller, domain-specific models (155M parameters) can achieve performance comparable to much larger, multi-stage post-trained models, offering a cost-effective alternative for legal NLP applications.
• Curriculum Learning Necessity: For smaller models, a structured, multi-phase curriculum learning approach is critical for successful domain adaptation without catastrophic forgetting, whereas larger models can tolerate single-phase training.
• Limitations: The paper modestly notes that decoder-to-encoder conversion without extensive multi-stage training infrastructure (synthetic data generation, model merging) results in suboptimal performance for embedding tasks, particularly in specialized legal domains.

The authors emphasize that their work addresses the underserved nature of Turkish legal NLP, providing open-source models and benchmarks to facilitate the development of AI-powered legal tools tailored to Turkey's unique legal framework.