Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging

Technical Summary: Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging

Problem Statement
The standard approach for developing task-specific multilingual Large Language Models (LLMs) involves fine-tuning a single base model on a combined dataset containing examples from all target languages. While effective for initial development, this "retrain-all" strategy creates significant computational inefficiencies and maintenance bottlenecks in enterprise settings. Specifically, adding support for a new language or updating an existing one with new data necessitates retraining the entire model on the full combined dataset. This process is computationally expensive, unscalable, and costly. While recent research has explored model merging to improve quality and generalization, its potential as a strategy for computational efficiency and maintenance cost reduction remains largely unstudied.

Methodology
The authors propose and evaluate a "train-once, merge-as-needed" paradigm as an alternative to the traditional combined training approach. In this strategy, individual models are fine-tuned independently for each language and task, and then merged into a single set of weights.

• Merging Techniques: The study utilizes three established merging techniques:

  • TIES (Trim, Elect Sign, and Merge): Retains top-k weights, selects signs, and merges by calculating the mean.
  • DARE (Drop And REscale): Randomly drops weights and rescales the remaining ones before merging.
  • KnOTS (Knowledge Orientation Through SVD): Concatenates fine-tuned weights layer-by-layer and applies Singular Value Decomposition (SVD) to create task-specific matrices before merging.
  • The experiments combine these techniques (e.g., TIES-KnOTS, DARE-TIES-KnOTS) with varying hyperparameters for weight density and vector weighting.

• Experimental Setup:

  • Base Model: Llama-3.1-8b-Instruct (and Llama-3.2-3b-Instruct for size ablation).
  • Tasks: Text Summarization (WikiLingua), Commonsense Reasoning (mCSQA), and Sentiment Analysis (MultilingualSentiment).
  • Languages: English (EN), German (DE), French (FR), Japanese (JA), and Chinese (ZH).
  • Baselines: The proposed merging approach is compared against two baselines: a model fine-tuned on the combined dataset (COMB) and models trained/evaluated independently per language (INDV).
  • Maintenance Simulation: The study simulates maintenance by adding new data to a single language (English) and retraining only that specific adapter, followed by re-merging, rather than retraining the full model.

Key Contributions

1. First Focused Efficiency Analysis: This work provides the first dedicated analysis of language-specific model merging from the perspective of computational efficiency and maintenance costs, moving beyond purely quality-based evaluations.
2. Quantified Savings: The authors quantify the economic and temporal benefits of the merging strategy, demonstrating significant reductions in training time and cost compared to the "retrain-all" approach.
3. Industrial Validation: The approach is validated not only on public academic datasets but also on a proprietary industry dataset, confirming its applicability to real-world enterprise use cases.
4. Maintenance Workflow Demonstration: The paper demonstrates a practical workflow where updating a single language adapter and re-merging is sufficient to maintain or improve overall model performance, avoiding full retraining.

Results

• Performance Parity: Across the three public tasks, the merged models achieved performance parity with the combined baseline (COMB).
  • Summarization & Reasoning: Merged models (specifically those using TIES-KnOTS and DARE-TIES-KnOTS) performed comparably to the combined model, with slight improvements observed in English, Japanese, and Chinese for Summarization (BertScore) and Reasoning.
  • Sentiment Analysis: The combined model generally outperformed merged models in this classification task. The authors hypothesize that the limited label space in classification tasks may not interact as well with merging techniques as generative tasks do.
  • Model Size: Experiments with the smaller Llama-3b model showed similar trends to the 8b model, indicating the approach is size-agnostic, though absolute performance scales with model capacity.
• Efficiency Gains:
  • Initial Training: The merging approach reduced initial training time by approximately 35% (from 3.4h to 2.2h) due to the ability to train language models in parallel.
  • Maintenance/Updates: When updating a single language (e.g., adding English data), the merging approach reduced training time by 73.7% (from 3.8h to 1.0h) and training costs by over 60% compared to retraining the full multilingual model.
• Case Study (Proprietary Data): In a proprietary summarization task, the merging approach reduced initial setup time by 50% (45h to 22.5h) and update costs by 62.4%. The merged model achieved comparable or better performance than baselines for most languages, with the exception of Japanese, which required a specific adapter update to improve hallucination rates.

Significance and Claims
The paper claims that language-specific model merging is a practical and efficient solution for the lifecycle management of multilingual models in industry. It asserts that the "train-once, merge-as-needed" strategy offers a viable alternative to the computationally expensive "retrain-all" method. The authors emphasize that this approach allows enterprises to:

• Achieve performance on par with fully retrained models.
• Significantly reduce computational costs and training time.
• Scale maintenance operations by updating individual language adapters rather than the entire model.
• Gain granular control over hyperparameters for specific languages based on business needs.

The work concludes that while performance parity is not guaranteed for every task type (notably sentiment analysis), the substantial gains in efficiency and cost reduction make model merging a compelling strategy for industrial multilingual model deployment. Future work is planned to explore different model families, optimize adapter weight selection, and investigate low-resource language scenarios.