Technical Summary: SkillWeaving: Efficient LLM Improvement via Modular Skillpacks

Problem Statement

Large Language Models (LLMs) face a critical bottleneck in balancing multi-domain specialization with strict memory and inference constraints. While monolithic models struggle to maintain high performance across heterogeneous domains without catastrophic forgetting or task interference, existing solutions present trade-offs:

• Full-parameter fine-tuning offers high capacity but incurs prohibitive storage and inference overhead.
• Parameter-Efficient Fine-Tuning (PEFT) methods (e.g., LoRA) are lightweight but often underperform in retaining specific domain capabilities.
• Delta-compression techniques reduce storage size but frequently require runtime dequantization, failing to deliver significant inference latency improvements.
• Model merging approaches often suffer from task interference when combining capabilities into a single parameter space.

Consequently, practitioners are forced to maintain multiple specialized models or deploy massive monolithic models, both of which are impractical for real-world, resource-constrained deployments.

Methodology: SkillWeave Framework

The authors propose SkillWeave, a modular framework designed to enable LLMs to specialize under fixed memory and inference budgets. The framework decomposes a general-purpose model into skillpacks—lightweight, domain-specific delta modules—and integrates SkillZip, a compression strategy for efficient deployment. The pipeline consists of three stages:

1. Skillpack Building (Self-Specialization)

Instead of relying on external human annotations, SkillWeave utilizes self-generated data for domain-specific improvement:

• Task Decomposition & Self-Generation: A base model is prompted with domain-specific instructions to generate candidate responses.
• Rule-Based Classification: A lightweight filter separates generated data into "helpful" ($D^+$) and "harmful" ($D^-$) subsets based on task-specific rules (e.g., regex for math answers, sandbox execution for code, heuristic filters for reasoning).
• Preference Optimization: The base model is fine-tuned using Direct Preference Optimization (DPO) on the preference-labeled dataset. This produces a full-parameter delta ($\Delta_k$) for each domain, capturing refined capabilities while isolating domain-specific knowledge.

2. Skillpack Compression (SkillZip)

To address the storage and inference costs of retaining full-parameter deltas, the authors introduce SkillZip, a fully quantized compression strategy:

• Model Merging for Shared Knowledge: Before compression, shared cross-task knowledge is extracted via model merging ($\Delta_{shared}$) and reintegrated into the backbone. The remaining deltas are subtracted from this shared component, resulting in sparser, more task-specific skillpacks.
• Full Quantization: Unlike prior methods that quantize only weights, SkillZip quantizes both delta weights and activation inputs. This allows direct computation in low-bit integer formats (INT8/INT4) without runtime dequantization.
• Double Smoothing: To mitigate quantization errors caused by outliers (activation channels with magnitudes $100\times$ larger than others), the method employs:
  1. Channel-wise Smoothing: Rebalances activation and weight magnitudes to shift quantization burden.
  2. Rank-wise Smoothing: Applies orthogonal rotation to low-rank weight matrices ($A, B$) to disperse energy evenly, preventing singular vectors from dominating the quantization range.

3. Skillpack Integration (Modular Deployment)

At inference time, the system maintains a shared backbone that remains active for all tasks. Based on the inferred task type (via a lightweight router or explicit tool invocation), a single specialized skillpack is dynamically selected and applied. The output is computed as:
$$Output_i = X_i W + \hat{X}_i \hat{A}_i \hat{B}_i$$
where $W$ represents the shared backbone weights, and $\hat{A}_i, \hat{B}_i$ are the quantized low-rank matrices for the active skillpack.

Key Contributions

1. SkillWeave Framework: A modular improvement framework that decomposes model capabilities into compact, self-specialized skillpacks. It reorganizes internal knowledge to achieve scalable, interpretable self-improvement under fixed memory budgets.
2. SkillZip Compression: A fully quantized delta compression strategy that jointly quantizes weights and activations. Unlike prior approaches focused on storage reduction, SkillZip eliminates runtime dequantization, delivering low-latency inference with hardware-aware optimization.
3. Empirical Validation: Comprehensive evaluation on multi-task and agentic benchmarks demonstrating that a 9B SkillWeave model (8B backbone + 0.5B skillpacks) outperforms task-specific models and a 32B monolithic LLM, while achieving up to 4× inference speedup.

Experimental Results

The authors evaluated SkillWeave on general capabilities (Math, Coding, Dialogue, Reasoning) and LLM-as-Agent scenarios (AgentBench).

• Performance: The 9B SkillWeave model achieved state-of-the-art results among baselines of comparable size, surpassing larger open-source models (e.g., Gemma2-27B) in specific domains (e.g., +8.1 on MATH). It outperformed multi-teacher distillation, model merging, and routing-based MoE baselines.
• Interference Mitigation: By isolating domain-specific training, SkillWeave avoided the catastrophic forgetting and task interference observed in mixed-domain and sequential fine-tuning baselines.
• Efficiency:
  • Memory: In agent scenarios, SkillWeave required only a 7B backbone plus five 0.5B skillpacks (9.5B total resident parameters), compared to 35B for five separate 7B models or 32B for a monolithic model.
  • Latency: SkillWeave achieved a 4.2× speedup over a 32B monolithic model and a 5.5× speedup over a system deploying five separate 7B models, due to efficient batch processing and kernel-level optimizations.
  • Compression Fidelity: SkillZip maintained performance comparable to full-precision deltas, outperforming delta-compression baselines like BitDelta, ASVD, and DeltaCome in both accuracy and inference speed.

Significance and Claims

The paper claims that SkillWeave offers a promising direction for the scalable and efficient enhancement of LLMs. By decomposing monolithic capabilities into modular skillpacks and utilizing full quantization for deployment, the framework successfully balances performance gains with parameter efficiency.

The authors emphasize that their approach enables self-improvement without reliance on external teachers or human annotations, while the SkillZip component bridges the gap between high-fidelity specialization and low-latency inference. The results suggest that modular skill composition can outperform sheer model scaling, making specialized LLM capabilities accessible within strict resource constraints.

The authors acknowledge limitations regarding the reliance on rule-based verification for structured tasks (e.g., code, math) and the challenges of applying this verification paradigm to open-ended tasks (e.g., creative writing) where clear correctness criteria are absent. They posit that future work should focus on automatic skill discovery and verification in these open-ended settings.