Signature-Guided Capacity Occupancy for Dense Expert Merging

Technical Summary: SigMerge (Signature-Guided Capacity Occupancy)

Problem Statement
Dense expert merging aims to combine multiple domain-specialized language models into a single checkpoint without further training, typically by operating on task vectors in weight space. However, existing methods face a fundamental limitation: a single set of shared parameters must encode skills acquired in isolation, leading to "expert interference" where updates for one domain erase capabilities learned by another.

While existing strategies address this partially, they fail to answer three critical questions regarding the admission of task-vector support:

1. Where: In which layers should capacity be opened, and how much?
2. Who: Which domains should occupy that capacity based on demand?
3. How: How should support be admitted without relying on costly search over merge recipes?

Empirical analysis reveals that dense merges leave a significant "residual expert gap" (an average of 12.13 percentage points across various settings), indicating that specialist capabilities are not fully retained. Current methods often apply uniform rules or rely on search-based proxies that may not capture true specialist performance, failing to route capacity to the domains that need it most.

Methodology: SigMerge
The authors propose SigMerge, a training-free operator that recovers specialist ability by allocating a bounded number of task-vector coordinates across layers and domains. The method operates on top of a dense base merge (the "anchor") and consists of three core components:

1. Conflict-to-Capacity (Where):
   SigMerge determines the capacity of each layer ($C_i$) based on cross-expert disagreement. It aggregates three complementary views of conflict at each layer:

   • Hidden-state geometry: Measured via Centered Kernel Alignment (CKA) of pooled hidden states.
   • Task vector direction: Measured via cosine similarity between expert updates.
   • Signed coordinate demand: Measured by the frequency of opposing signs on individual weights.
     These signals are normalized and averaged to form a conflict score ($\bar{S}_i$). The layer capacity is then derived linearly: layers with high conflict receive less capacity (to avoid interference), while layers with low conflict receive more.

2. Deficit-to-Allocation (Who):
   The method allocates the opened capacity among domains based on "residual expert deficits." For each domain $d$, the deficit $g_d$ is defined as the performance gap between the base merged model and the best expert in the pool for that specific domain. The share of capacity ($\rho_d$) assigned to each domain is proportional to its deficit. Domains that the base merge already matches perfectly receive zero capacity, ensuring the repair targets only missing capabilities.

3. Sequential Occupancy (How):
   To fill the layer-domain budget without search, SigMerge employs a sequential claiming rule. Domains are ordered by their deficit magnitude (largest deficit first). Each domain claims its allocated quota of coordinates from the set of still-unclaimed coordinates within a layer, selecting those with the largest magnitude in its task vector ($\Delta_{d,i}$). This prevents multiple domains from competing for the same high-magnitude coordinates and ensures the most under-served domains get priority access to the most impactful updates.

The final merged model is constructed as $\hat{\theta} = \theta_{base} + \sum_d V_d$, where $V_d$ represents the bounded increment admitted by the sequential occupancy rule.

Key Contributions

1. Empirical Characterization: The authors demonstrate that specialist deficits in dense merging are large and heterogeneous across models, domains, and merge families, arguing that repair requires coordinate allocation rather than just better averaging rules.
2. Training-Free Framework: SigMerge resolves the three challenges of capacity assignment (where, who, how) without training or search. It uses measurable signals (conflict signatures and performance deficits) to construct a rank-one budget matrix ($B = C\rho^\top$).
3. Theoretical and Algorithmic Guarantees: The paper provides a proof (Theorem 1) that the sequential occupancy rule is stagewise optimal for maximizing admitted task-vector mass given fixed quotas and a precedence order.
4. Comprehensive Evaluation: The method is evaluated across 21 paired settings (7 anchors $\times$ 3 model pools) and compared against six baselines, including conflict-aware, layer-wise, and search-based methods.

Results

• Performance Improvement: SigMerge improves every one of the 21 paired settings, achieving an average relative improvement of 15.0% in Macro-5 scores.
• Gap Reduction: The average residual expert gap is reduced from 12.13 to 5.77 percentage points.
• Ranking: SigMerge achieves the best average rank (1.67) among six merging methods, outperforming all baselines across three categories (conflict-aware, layer-wise, and search-based).
• Ablation Studies: Removing either the conflict-derived capacity or the deficit-driven allocation causes significant performance drops (comparable to the total gain), confirming that both signals are essential. The specific choice of coordinates (sequential occupancy) accounts for roughly 80% of the gain at a fixed budget.
• Efficiency: The method adds approximately 0.6 GPU hours to the anchor's cost, placing it among search-free baselines and significantly faster than search-based methods like MERGE3 (which takes ~26.8 hours).

Significance and Claims
The paper claims that SigMerge offers a principled, training-free approach to dense expert merging that explicitly addresses the heterogeneity of domain deficits and layer-wise conflicts. By grounding capacity decisions in measurable signals rather than search or uniform heuristics, SigMerge effectively recovers specialist capabilities lost in standard dense merges.

The authors modestly note that the method does not determine the anchor or the expert pool; it operates within the interval defined by them. Furthermore, the current evaluation is limited to five domains, and the scalability to larger pools or the necessity of higher-rank budget matrices remains an open question. However, the results suggest that measuring and routing based on actual performance deficits is a more effective strategy than relying on task-vector geometry alone or expensive search procedures.