Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging

Technical Summary: Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging

1. Problem Statement

As Large Language Model (LLM) development increasingly relies on families of checkpoints—including base models, instruction-tuned variants, domain experts, and delta updates—the task of consolidating these into a single deployable model via weight-space merging has become critical. While existing research focuses on how to combine weights (e.g., via averaging, TIES, or DARE), a significant execution bottleneck has emerged at scale: the cost of reading expert parameters.

In naive merging scripts, the system treats checkpoints as opaque files, scanning all expert parameters regardless of their actual contribution to the final merge. As the number of experts ($K$) grows, the expert-read I/O scales nearly linearly, making the merging process I/O-bound rather than compute-bound. This limits the scalability of iterative merging workflows, particularly when dealing with large pools of experts on disk-resident storage.

2. Methodology: MergePipe

The authors introduce MergePipe, a budget-aware execution layer that reframes model merging as an expert access-set problem. Instead of treating the merge rule as the primary constraint, MergePipe decouples the logical combination of weights from the physical access of those weights.

Core Abstraction:
MergePipe operates under the assumption that experts are aligned in a shared weight coordinate system (e.g., fine-tuned from a shared base). It introduces an access mask ($A$) that determines which expert delta blocks are physically read.

• Full-Read Merge: $A = 1$ (all experts read).
• Budgeted Merge: Select a mask $A$ such that the expert-read cost $C_{expert}(A) \leq B$, where $B$ is a user-defined I/O budget.
• Execution: The system executes a mask-aware operator $\Psi_{op}$. Omitted entries are represented by the mask itself and do not trigger storage reads.

System Components:

1. Catalog: Exposes checkpoints as block-structured data rather than opaque files. It stores metadata (size, layout, hashes, sketches, coverage hints) to enable cost estimation without scanning full checkpoints.
2. Planner: Constructs a deterministic access mask $A$ satisfying the budget constraint. It ranks candidate expert deltas using catalog statistics (e.g., norms, sketches) and employs greedy or score-per-byte heuristics to select blocks. It does not alter the merge operator logic but decides which deltas are materialized.
3. Executor: Streams base blocks and uses a DeltaIterator to materialize only the selected expert deltas. It applies the mask-aware operator and records a replayable manifest containing the access mask, plan hash, and lineage.

Theoretical Guarantees:

• Budget Soundness: The execution trace is guaranteed to stay within the specified budget $B$ by construction.
• Full-Budget Consistency: If the budget is set to cover all experts ($A=1$), the system recovers the standard full-read merge exactly.
• Error Bounds: For fixed-coefficient additive operators, the omitted-update error is bounded by the norm of the omitted deltas. For non-additive operators (like TIES or DARE), the system provides the same abstraction, with fidelity evaluated empirically.

3. Key Contributions

1. Expert Access Sets as a Budgeted Object: The paper proposes treating expert access as a first-class, budgeted resource, separating the "which" (access) from the "how" (combination).
2. Theoretical Analysis: The authors prove budget soundness, full-budget consistency, and derive an omitted-update bound for additive merges, justifying the use of norm-based access scores.
3. MergePipe Implementation: A concrete system that implements this abstraction, achieving order-of-magnitude reductions in expert-read I/O and significant speedups without requiring changes to the underlying merge operators.

4. Experimental Results

The authors evaluated MergePipe on Qwen and Llama checkpoint families (ranging from 0.6B to 8B parameters) with up to 25 experts.

• I/O and Speed: MergePipe reduces expert-read I/O by up to an order of magnitude. In specific scenarios (e.g., TIES merging with 20 experts), it achieves up to 11× speedups compared to naive full-read baselines.
• Scalability: While naive merging shows near-linear growth in I/O and wall time as the number of experts increases, MergePipe keeps expert reads nearly flat by enforcing the budget cap.
• Operator Generality: The approach is compatible with various operators (AVG, TIES, DARE). Sparse methods like TIES benefit most, with I/O dropping from ~174GB to ~3.5GB in high-expert scenarios.
• Fidelity:
  • Parameter Deviation: At 50% of the full-read budget, the relative $\ell_2$ deviation from the full-read merge remains on the order of $O(10^{-3})$.
  • Downstream Performance: Benchmarks (HumanEval, IFEval, DROP) show no monotonic degradation. In some cases, performance even fluctuates slightly above the full-read baseline, indicating a favorable speed-fidelity frontier.
• Overhead: System overhead is minimal; planning time accounts for ~1% of execution time, and metadata storage is a small fraction of total I/O.

5. Significance and Claims

The paper claims that for LLM-scale checkpoint families, the limiting resource in merging is often the set of expert weights that must be read, not the computation required to combine them.

• Paradigm Shift: The authors argue that LLM-scale merging requires a weight-access abstraction, not just better merge rules. By treating weights as structured, budgeted data rather than opaque files, MergePipe exposes a practical speed–fidelity frontier.
• Complementarity: MergePipe is presented as complementary to existing symmetry- or permutation-alignment methods. Once weights are aligned, MergePipe optimizes the physical execution.
• Future Implication: As model families continue to grow, the authors suggest that weight-space methods must be paired with execution layers that explicitly manage parameter I/O patterns to remain scalable.

The work does not claim to invent new merging algorithms but rather provides the necessary execution infrastructure to make existing merging algorithms scalable in the era of massive expert pools.