Technical Summary: Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

Problem Statement

The reliable deployment of Large Language Models (LLMs) requires balanced optimization across three critical dimensions: Helpfulness (accurate, task-aligned responses), Honesty (avoiding hallucinations), and Harmlessness (preventing toxic outputs), collectively termed 3H optimization. While existing alignment techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have improved individual aspects, achieving a simultaneous balance remains a significant challenge due to inherent trade-offs; for instance, optimizing for helpfulness can inadvertently generate harmful content, while harmlessness alignment may lead to dishonest responses.

Traditional approaches to resolving these conflicts rely on data mixture strategies, which combine diverse datasets to fine-tune a single model. However, these methods face limitations, including heavy reliance on expert knowledge for data curation, high computational costs, and the difficulty of managing conflicting optimization signals during fine-tuning. While model merging (integrating parameters from specialized models) offers a cost-effective, parameter-level alternative to mitigate catastrophic forgetting and balance objectives, its specific effectiveness and limitations for 3H optimization have remained underexplored. Key challenges in this domain include preference noise accumulation (where merging multiple aligned models introduces outlier weight updates that obscure true task directions) and layer sparsity adaptation (the failure of existing methods to account for varying sparsity patterns across different layers of an LLM).

Methodology

Benchmarking and Analysis

The authors first established a comprehensive benchmark to systematically compare data mixture and model merging techniques for 3H optimization. This involved:

• Datasets: 10 preference datasets covering 5 annotation dimensions (Helpfulness, Honesty, Harmlessness, and their combinations).
• Baselines: 15 representative methods, including 12 training-free model merging strategies (e.g., Weight Average, Task Arithmetic, TIES, DARE, TSVM) and 3 data mixture strategies (Heuristic, ArmoRM, Hummer).
• Models: Two instruction-tuned LLM families (Llama-3-8B and Mistral-7B) under both static (single-stage) and continual (multi-stage) optimization settings.

Through this benchmark, the authors revealed previously overlooked collaborative and conflicting relationships among the 3H dimensions. Notably, they found that while "Helpful" training benefits all 3H dimensions simultaneously, "Honest" and "Harmless" training often weaken each other.

Proposed Method: RESM

To address the specific challenges of preference noise and layer sparsity in multi-objective merging, the paper proposes Reweighting Enhanced task Singular Merging (RESM). This method builds upon Task Singular Vector (TSV) merging but introduces two novel mechanisms:

1. Outlier-Aware Weighting (Addressing Preference Noise):

   • Recognizing that merging multiple aligned models accumulates preference noise, RESM employs a statistical significance filtering mechanism based on the 3σ principle.
   • It calculates row-wise mean ($\mu$) and standard deviation ($\sigma$) of weight deviations ($\Delta$) for each layer.
   • A thresholding function suppresses low-magnitude fluctuations (noise) and retains only statistically significant deviations.
   • An aggregation weight ($\alpha$) is computed via L1-normalization of these sparse outlier magnitudes, ensuring balanced contributions from diverse tasks and preventing high-magnitude updates from dominating the merge.

2. Sparsity-Adaptive Rank Selection (Addressing Layer Sparsity):

   • Conventional methods use a uniform rank selection threshold ($k$) across all layers, which fails to account for the heterogeneity of parameter importance.
   • RESM dynamically determines the rank ($k_l$) for each layer based on its sparsity consensus ($\Omega_l$).
   • The sparsity is calculated by counting the proportion of parameter deviations below a small threshold $\epsilon$.
   • The dynamic rank is defined as $k_l = \lfloor d_l(\gamma_0 + \gamma\Omega_l) \rfloor$, where $d_l$ is the layer dimension. This allows for higher rank retention in sparse layers (preserving critical directional features) and stronger dimensionality reduction in dense layers (suppressing noise).

The final merged layer is computed by integrating these reweighted singular vectors:
$$\theta^{RESM}_l = \theta^0_l + \sum_{i=1}^n U^{(i)}_{l\perp}[:, : k_l] \cdot (\alpha^{(i)}_l S^{(i)}_l) \cdot V^{(i)\top}_{l\perp}[: k_l, :]$$

Key Results

Performance Comparison

Extensive evaluations on Llama-3 and Mistral models demonstrate that model merging generally outperforms data mixture methods in achieving balanced 3H optimization:

• Static Optimization: On Llama-3, the best data mixture method (Hummer) achieved a normalized gain of 10.16%. In contrast, model merging methods consistently surpassed this, with the proposed RESM achieving a 14.97% gain. TSVM (the baseline for RESM) achieved 13.00%.
• Mistral Results: Similar trends were observed, with RESM achieving a 3.36% gain over the baseline, outperforming the best data mixture method (1.53%).
• Stability: Unlike random sparsification methods (e.g., DARE, DELLA) which showed unstable performance across different random seeds, RESM demonstrated robust and consistent results due to its SVD-based approach and adaptive strategies.

Ablation Studies

Ablation studies confirmed the necessity of both components in RESM:

• Removing outlier weighting reduced the normalized gain on Llama-3 from 14.97% to 13.60%.
• Removing adaptive rank selection reduced the gain to 13.71%.
• The combination of both strategies yielded the highest performance, validating that addressing both noise accumulation and layer sparsity is critical.

Continual Optimization

In continual DPO training settings (simulating real-world scenarios with sequential data), RESM maintained superior performance over final-stage models. The authors observed that catastrophic forgetting effects paradoxically enhanced 3H alignment through interactive optimization, and RESM's strategy of merging from a foundational checkpoint (rather than intermediate ones) effectively mitigated overfitting to prior objectives.

Significance and Claims

The paper claims to provide the first systematic benchmark comparing data mixture and model merging for 3H optimization. Its primary contributions are:

1. Empirical Insights: It reveals that model merging offers a phased optimization paradigm that effectively negotiates competing alignment objectives, often outperforming data mixture strategies which struggle with conflicting signals.
2. Methodological Advancement: The proposed RESM algorithm addresses specific technical hurdles in LLM merging—preference noise and layer sparsity—through outlier weighting and adaptive rank selection, achieving state-of-the-art balanced alignment without requiring additional training data.
3. Theoretical Foundation: The work provides theoretical bounds on conflict probability, demonstrating that adaptive rank selection reduces the likelihood of directional conflicts between task vectors compared to fixed-rank approaches.

The authors conclude that model merging, particularly when enhanced with reweighting and sparsity-aware strategies, represents a viable and efficient pathway for constructing ethically constrained, multi-objective aligned LLMs. They emphasize that while RESM significantly improves Honesty and Harmlessness, the "Helpfulness" dimension still presents challenges due to intra-dimension conflicts, suggesting a need for future hierarchical conflict resolution frameworks.