Technical Summary: Mi:dm 2.0 – Korea-Centric Bilingual Large Language Models

Problem Statement

Despite the proliferation of Large Language Models (LLMs) supporting the Korean language, a significant gap remains in models that are truly grounded in the realities of Korean society. Existing models often suffer from insufficient or low-quality Korean training data, leading to limited linguistic performance and a misalignment with Korean cultural sensibilities. This results in unnatural responses, emotional incongruence, or even language switching from the perspective of Korean users. Furthermore, existing Korean datasets are structurally imbalanced, heavily skewed toward humanities and social sciences while lacking representation in STEM, applied sciences, and diverse structural formats (e.g., long-form reasoning, code). The challenge lies in developing a model that not only processes Korean text but also internalizes the unique values, cognitive frameworks, and commonsense reasoning inherent to Korean culture, all while operating efficiently within resource constraints.

Methodology

1. Data Engineering and Curation

The development of Mi:dm 2.0 prioritizes data quality over quantity, employing a rigorous, multi-stage pipeline to address the scarcity of high-quality Korean corpora.

• Data Classification: A multidimensional framework categorizes data by language, domain (Humanity, STEM, Applied Science, etc.), source (Organic vs. Synthetic), and linguistic style (Formal/Informal, Written/Spoken).
• Filtering Pipeline: An 8-stage filtering process is applied to Korean web data, including document de-duplication, heuristic filtering, perplexity filtering, broken document correction, model-based quality filtering, toxic content removal, line-level de-duplication, and PII anonymization.
• Synthetic Data Generation: To compensate for domain imbalances (particularly in STEM and applied sciences) and structural diversity, the authors generate high-fidelity synthetic data. Strategies include:
  • Domain Balancing: Converting seed concepts into textbook-style explanations and scenario-driven narratives for underrepresented domains.
  • Reconstruction: Rewriting low-quality but topically relevant filtered-out Common Crawl documents to recover usable tokens.
  • Cross-lingual Augmentation: Rewriting English web documents into natural Korean formats (e.g., converting content into university entrance exam questions) to introduce structural diversity without relying on direct translation.
  • Reasoning Data: Constructing Long Chain-of-Thought (LongCoT) datasets for math and code reasoning.
• Tokenizer: A custom tokenizer is designed specifically for the morphological structure of Korean to improve token compression and computational efficiency.

2. Model Architecture and Training Strategies

Mi:dm 2.0 is released in two configurations: Base (11.5B parameters) and Mini (2.3B parameters). Both utilize a Transformer decoder-only architecture with Grouped Query Attention (GQA) and SiLU activation.

• Mi:dm 2.0 Base (Depth-up Scaling):

  • Stage 1: An 8B parameter model is pre-trained from scratch on a curated corpus.
  • Stage 2: The model is expanded to 11.5B parameters using a Depth-up Scaling (DuS) strategy. Layers with high cosine similarity in embedding changes (indicating stable information preservation) are identified and duplicated to increase depth from 32 to 48 layers without complex architectural changes.
  • Long-Context Extension: Rotary Position Embedding (RoPE) base frequency is adjusted to extend context length from 8K to 32K tokens.

• Mi:dm 2.0 Mini (Pruning and Distillation):

  • Derived from the Base model through a multi-stage process to optimize for resource-constrained environments.
  • Stage 1: Width pruning reduces the model to an intermediate 5B "Base-half" model.
  • Stage 2: Knowledge distillation is performed using both the intermediate model and the full Base model as teachers. The student model is further pruned to 2.3B parameters, utilizing weight-sharing for embeddings.

3. Post-Training (Alignment)

The post-training phase focuses on six key capabilities: Instruction Following (IF), Reasoning, Retrieval-Augmented Generation (RAG), Agent Ability, Safety, and Long Context Handling.

• Supervised Fine-Tuning (SFT): Multiple SFT models are trained on diverse datasets (General Chat, IF, Reasoning, RAG, Tool Use, Safety) and their weights are merged to unify capabilities.
• Preference Optimization (RL): Reinforcement Learning is applied using human and AI preferences to align the model with safety, reliability, and user intent.
• Safety Training: Extensive datasets are constructed to handle harmlessness, honesty, and role consistency, including specific handling for borderline queries.

Key Contributions

• Korea-Centric AI Definition: The paper establishes a framework for "Korea-centric AI," emphasizing the internalization of Korean socio-cultural values, norms, and reasoning patterns rather than mere linguistic proficiency.
• Data-Centric Innovation: The introduction of a proprietary, multi-stage data cleansing pipeline and strategic synthetic data generation specifically designed to correct domain imbalances (e.g., STEM underrepresentation) in Korean corpora.
• Efficient Scaling Techniques: The successful application of Depth-up Scaling (DuS) to expand model capacity and multi-stage pruning/distillation to create a high-performance 2.3B model, demonstrating competitive results with significantly lower computational overhead.
• Comprehensive Evaluation Suite: The development of proprietary Korean-specific benchmarks (K-Pragmatics, K-Referential, Ko-Sovereign) and Responsible AI (RAI) evaluations tailored to the Korean context, moving beyond English-centric metrics.

Results

• Korean Benchmarks: Mi:dm 2.0 Base achieves state-of-the-art performance on Korean-specific benchmarks. It outperforms the domestic baseline (Exaone-3.5-7.8B) by 9.6% in reasoning and 8.7% in general knowledge. Notably, it shows a 17.1% advantage over Exaone-3.5-7.8B on the K-Referential-hard benchmark and leads in the HAERAE benchmark for cultural understanding.
• General Benchmarks: On English benchmarks (MMLU, GSM8K, etc.), Mi:dm 2.0 Base demonstrates competitive performance, particularly in instruction following and mathematics, often surpassing similarly sized global models.
• Efficiency: Mi:dm 2.0 Mini (2.3B) achieves performance comparable to or better than the 2.4B Exaone model in instruction following and Korean society/culture tasks, despite having fewer parameters.
• Safety and Robustness: In RAI evaluations, Mi:dm 2.0 Base achieves a 97.75% "Not Unsafe Rate" in content safety, outperforming Llama-3.1-8B. In red teaming scenarios, it demonstrates the lowest Attack Success Rate (36.72%) among comparable models.

Significance and Claims

The paper claims that Mi:dm 2.0 sets a new standard for genuine Korea-centric AI by moving beyond linguistic processing to embody cultural nuance and societal values. The authors argue that their approach—prioritizing rigorous data curation, strategic synthetic augmentation, and efficient scaling techniques—allows for the development of high-performance models even with constrained computational resources and smaller training corpora compared to global counterparts.

By releasing both the Base and Mini models under the MIT license, KT aims to accelerate AI adoption across Korean industries, public services, and education. The models are positioned as foundational elements of "K-intelligence," intended to strengthen the Korean AI developer community and provide reliable, culturally appropriate tools for real-world applications. The paper concludes that the success of Mi:dm 2.0 validates the critical role of precise data composition and efficient training strategies in enhancing model performance, offering a blueprint for developing specialized, high-quality LLMs in resource-constrained or linguistically specific environments.