From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

Technical Summary: From "Weak" Signals to Strong Models via Preference Delta Aggregation

Problem Statement
Training strong Large Language Models (LLMs) typically requires high-quality supervision data, which is often scarce or prohibitively expensive to annotate. While recent work (e.g., Geng et al., 2025) demonstrated that "weak" supervision signals derived from relative quality differences between weak–weaker model pairs (e.g., Qwen3 4B vs. 1.7B) can improve stronger models, a critical question remains: can multiple such weak signals be constructively aggregated to yield further gains?

Naive aggregation strategies face significant hurdles. Sequential training on multiple preference datasets leads to catastrophic forgetting, while joint training on mixed datasets often fails to outperform the best single dataset due to gradient conflicts. Furthermore, existing methods for combining model updates (such as simple weight averaging) often ignore the geometric structure of low-rank updates, leading to destructive interference when merging diverse preference directions.

Methodology
The authors propose Preference Delta Aggregation (PDA), a two-stage framework designed to aggregate multiple "weak" supervision signals without relying on high-quality data.

1. Preference Delta Derivation:

   • The framework constructs preference datasets from $n$ distinct weak–weaker model pairs. For each pair, the response from the stronger weak model is treated as the "chosen" sample and the weaker model's response as "rejected."
   • A strong student model ($\theta_s$) is independently fine-tuned on each preference dataset using parameter-efficient methods (specifically LoRA).
   • Each resulting LoRA adapter ($\Delta W_i$) represents a preference delta, capturing the specific improvement direction induced by the relative quality gap of that specific model pair.

2. Geometric Alignment Merging (GAM):

   • To aggregate these independent adapters, the authors introduce Geometric Alignment Merging (GAM), a novel merging method that addresses directional interference.
   • Decomposition: Each adapter's low-rank update matrix is decomposed via Singular Value Decomposition (SVD) into orthogonal bases ($U, V$) and singular values ($S$).
   • Subspace Alignment: Recognizing that independently trained adapters may learn similar subspaces with rotationally misaligned coordinate systems, GAM aligns all adapters to a reference adapter using Orthogonal Procrustes alignment on the Grassmannian manifold. This synchronizes the basis orientations while preserving the underlying subspaces.
   • Reconstruction: The aligned directional bases ($\tilde{U}, \tilde{V}$) and magnitudes ($\tilde{S}$) are averaged independently and then reconstructed to form the final merged weight update ($\Delta W_{GAM}$), which is added to the pre-trained student model.

Key Contributions

1. Preference Delta Aggregation (PDA): The first framework to improve strong models by aggregating paired preference data from multiple weak–weaker model pairs via LoRA merging, bypassing the need for high-quality supervision.
2. Geometric Alignment Merging (GAM): A geometry-aware merging technique that mitigates directional interference by decomposing adapters, aligning their low-rank subspaces via SVD and Procrustes alignment, and independently averaging directional and magnitude components.
3. Empirical Validation: Extensive experiments demonstrating that PDA-GAM effectively composes complementary capabilities encoded in distinct preference deltas, outperforming both single-delta and multi-delta baselines.

Experimental Results
The framework was evaluated on knowledge reasoning (MATH, GPQA, MMLU, GSM8K) and agentic search (single-hop and multi-hop QA benchmarks like MuSiQue, HotpotQA).

• Performance Gains: Using only low-quality preference data from weaker models, PDA-GAM improved the Qwen3-8B model by an average of 6.8 points on knowledge reasoning and 7.3 points on agentic search.
• Comparison to Baselines:
  • PDA-GAM outperformed all single-delta baselines and training-based aggregation methods (sequential and joint training), which suffered from catastrophic forgetting or gradient conflicts.
  • It exceeded the best single-delta baseline by 2.1 points (reasoning) and 4.3 points (agentic search).
  • It consistently outperformed existing LoRA merging methods (Naive Averaging, TIES-Merging, KnOTS). Notably, while TIES-Merging plateaued on agentic search tasks, PDA-GAM continued to improve as more signals were added.
• Generalization: Similar improvements were observed on the Tülu3-8B student model, indicating robust generalization across different architectures.
• Compositional Analysis: Qualitative analysis revealed that single-delta models often exhibit strong specialization (e.g., high adaptive search but poor verification), whereas PDA-GAM successfully composed these complementary behaviors, achieving high performance in both domains simultaneously.

Significance and Claims
The paper claims that aggregating multiple "weak" signals constructively allows strong models to surpass the performance ceiling of any single signal source. The authors attribute these gains to the effective composition of complementary capabilities encoded across distinct preference deltas.

The work validates the Delta Learning Hypothesis in more complex, multi-turn agentic search environments, extending its applicability beyond simple reasoning tasks. By introducing GAM, the paper addresses the geometric incompatibility of diverse LoRA adapters, suggesting that aligning subspaces is crucial for robust model composition. The authors note that these gains rely on directionally diverse signals; aggregating behaviorally homogeneous pairs yields only marginal improvements, highlighting the importance of signal diversity.

Limitations
The authors acknowledge three primary limitations:

1. The method relies on directionally diverse signals; aggregating homogeneous pairs yields limited gains.
2. The study is limited to a specific set of weak–weaker model pairs, and scaling behavior with larger collections remains unexplored.
3. While training-based and LoRA-based methods were compared, other potential aggregation strategies were not investigated.