LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model

1. Problem Statement

The rapid proliferation of open-source Large Language Models (LLMs) has led to efforts in creating domain-specific specialists. However, existing domain-specialized LLMs (particularly in the legal field) often suffer from a critical disconnect:

• Misalignment with Real-World Needs: Many models are trained on datasets and protocols based on naive assumptions of utility rather than the nuanced, high-stakes requirements of actual legal workflows.
• Lack of Rigor: Current approaches often fail to address the ambiguity inherent in legal tasks or justify their data compositions and input/output formats.
• Catastrophic Forgetting: Adapting general models to specific domains often risks degrading performance on general tasks or failing to maintain the model's identity (e.g., answering "Who are you?").
• Context: While there is high demand for AI in the Korean legal sector, there is a lack of systematic frameworks for adapting open LLMs to Korean law specifically.

2. Methodology

The authors propose LEGALMIDM, a vertical LLM for the Korean legal domain, built upon the Mi:dm-2.0-Base (an 11.5B parameter Korean-English bilingual model). The methodology is a use-case-driven, end-to-end framework consisting of four key pillars:

A. Requirement Investigation & Task Selection

Instead of generic training, the authors analyzed 20 operational legal tech services (US and South Korea) and consulted practicing attorneys. They identified six high-demand tasks:

1. Document Summarization (DS)
2. Document-based Question Answering (DQA)
3. Open Question Answering (Open QA)
4. Complaint Drafting (DD)
5. Petition Drafting (DD)
6. Multiple Choice QA (MC)

B. Data Curation & Construction

• Human-Curated Data: A team of law majors, industry professionals, and attorneys constructed high-quality datasets. This includes 100 held-out test samples per task.
• Automatic Synthetic Data Generation: To scale data volume, they leveraged the Korean Legislative Information Center statutes. Using GPT-4o, they generated QA pairs where the model was prompted to cite specific legal provisions.
  • Verification: A string-matching verification step ensured that cited references were actual substrings of the source statutes, ensuring factual grounding.
• General Domain Preservation: To prevent catastrophic forgetting, the authors analyzed real user queries (from ChatKoAlpaca) to identify common identity questions (e.g., "Who are you?") and included general domain data (KoAlpaca, KOpen-HQ) in the training mix.

C. Training Pipeline

The training process involved two stages with specific ablation studies to determine optimal strategies:

1. Continual Pre-Training (CPT): Adapting the base model to the legal domain.
2. Instruction-Tuning (IT): Refining the model for specific tasks.

Key Design Decisions (Validated via Ablation):

• Data Composition: Integrating general domain data (at a 7:3 ratio of General:Legal) during both CPT and IT stages yielded superior performance compared to training solely on legal data. This preserved general capabilities while enhancing legal specialization.
• Synthetic Data Formatting: The authors tested three formats for legal references ($R$):
  1. $M(Q) \to A$ (No reference)
  2. $M(Q) \to A + R$ (Reference in output)
  3. $M(Q + R) \to A$ (Reference in input)
  • Finding: While generating references in the output worked well for document tasks, it caused instability in multiple-choice questions. The optimal strategy was placing legal references in the input ($Q + R$) for all tasks.
• System Prompt Optimization: The authors tested training with vs. without system prompts (defining the model as "Mi:dm").
  • Finding: Training without system prompts yielded the best legal performance. The system prompt should be applied only at inference time to ensure the model correctly identifies itself without compromising task-specific reasoning.

3. Key Contributions

1. Use-Case-Driven Framework: A systematic recipe for transforming open LLMs into legal experts, grounded in actual industry requirements rather than theoretical assumptions.
2. LEGALMIDM Model: The release of a specialized 11.5B Korean legal LLM that outperforms significantly larger models (e.g., Llama-3.3-70B, Qwen2.5-32B) on Korean legal benchmarks.
3. Empirical Validation of Training Strategies:
   • Demonstrated that mixing general domain data is crucial for maintaining robustness.
   • Identified that inputting legal references is superior to generating them for synthetic data.
   • Proved that system prompts should be decoupled from training data and applied only during inference.
4. High-Quality Dataset Construction: A rigorous pipeline involving legal experts and automated verification to create factually grounded legal datasets.

4. Results

The model was evaluated on six legal tasks and two general benchmarks (KMMLU, HAERAE).

• Legal Performance: LEGALMIDM (11.5B) achieved the highest scores across all legal tasks (Complaint, Summary, Petition, QA, MRC, MC).
  • Example: On the Complaint task, LEGALMIDM achieved a Rouge-L score of 67.67, significantly outperforming Llama-3.3-70B (53.40) and Qwen2.5-32B (58.81).
  • It also achieved the highest LLM-judge scores (evaluated by GPT-4o) across the board.
• General Performance: Despite being smaller, LEGALMIDM maintained competitive performance on general benchmarks (KMMLU, HAERAE), comparable to or exceeding much larger models, confirming that the inclusion of general data prevented catastrophic forgetting.
• Ablation Insights:
  • Removing general data from CPT/IT caused a significant drop in legal task performance.
  • Including references in the input ($Q+R$) consistently outperformed other formats in stability and accuracy.

5. Significance

This paper addresses a critical gap in the deployment of AI in the legal sector by shifting the focus from "more data" to "better, use-case-aligned data and training strategies."

• Practical Utility: The framework provides a reproducible path for developing vertical LLMs that are not just theoretically capable but practically useful for high-stakes tasks like drafting complaints and petitions.
• Efficiency: It demonstrates that a smaller, well-tuned model (11.5B) can outperform massive foundation models (70B+) in specialized domains when the training pipeline is optimized for the specific domain's constraints.
• Korean Legal AI: It fills a specific void in the Korean legal tech landscape, providing a model trained on Korean statutes and cultural contexts, which is essential for accurate legal reasoning in that jurisdiction.

In conclusion, LEGALMIDM establishes a new standard for domain specialization, proving that rigorous data curation, strategic data mixing, and specific architectural choices (like input-based referencing) are more impactful than simply scaling model size.