💬 NLP

SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

The paper introduces SPARKLING, a novel framework for mid-stage width expansion in progressive learning that balances signal preservation and symmetry breaking to overcome training instabilities, thereby reducing computational costs by up to 35% while outperforming training from scratch.

Original authors: Qifan Yu, Xinyu Ma, Zhijian Zhuo, Minrui Wang, Deyi Liu, Shiyi Zhan, Yiyuan Ma, Liang Xiang, Xingyan Bin, Di He

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qifan Yu, Xinyu Ma, Zhijian Zhuo, Minrui Wang, Deyi Liu, Shiyi Zhan, Yiyuan Ma, Liang Xiang, Xingyan Bin, Di He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, incredibly complex library (a Large Language Model) to store and understand human knowledge. Usually, to build the biggest, smartest library, you have to construct the entire building from the ground up, brick by brick, all at once. This is incredibly expensive and takes a huge amount of time and energy.

Progressive Learning is a smarter way to do this: you start with a small, cozy reading room and gradually add more rooms as you go. This saves money because you don't need to build the whole palace before you start reading.

However, there's a catch. While researchers have figured out how to easily add more floors (depth), adding more rooms on the same floor (width) in the middle of the construction process is a nightmare. If you just slap new rooms onto the existing building without care, the whole thing starts to wobble and collapse.

The paper introduces a new method called SPARKLING (which stands for balancing Signal Preservation And symmetRy breaKing for width-progressive LearnING) to fix this. Think of SPARKLING as a master architect's toolkit for safely expanding the library mid-construction.

Here is how it works, using simple analogies:

The Two Big Problems

When you try to double the size of the library in the middle of construction, two things go wrong:

The "Volume Shock" (Signal Preservation):
Imagine the library has a strict rule: the noise level in every hallway must be exactly the same. If you suddenly add a bunch of new, empty rooms, the acoustics change. The sound (the data) gets too quiet or too loud in the new areas, confusing the librarians (the model's neurons).
- The SPARKLING Fix: They use a technique called RMS-scale consistency. It's like installing a "volume knob" on every new room. Before the new rooms are used, they are tuned so the noise level matches the old rooms perfectly. This ensures the data flows smoothly without shocking the system.
The "Clone Trap" (Symmetry Breaking):
The easiest way to add a new room is to copy the blueprint of an existing one exactly. But here's the problem: if you have two identical rooms with identical furniture and identical librarians, they will do the exact same thing. They will get the exact same instructions and move in the exact same way. They become "clones" that don't learn anything new; they just repeat what the original room does. The library gets bigger, but it doesn't get smarter.
- The SPARKLING Fix: They break the "clone trap" with two tricks:
  - The "Fresh Start" (Optimizer State Reset): Even though the new room looks like the old one, they wipe the slate clean for the new librarians. They don't give them the old librarians' memories or habits. This forces the new librarians to figure things out for themselves.
  - The "Special Boost" (Asymmetric Learning Rate): They give the new librarians a temporary, slightly louder megaphone (a higher learning rate) for a short while. This encourages them to try new things and explore, while the old librarians keep working at their normal pace. This ensures the new rooms develop their own unique personality instead of just copying the old ones.

The Results

The researchers tested this on two types of libraries:

Dense Models: A standard library where every book is on every shelf.
MoE (Mixture-of-Experts): A high-tech library with specialized "expert" rooms that only open when needed.

They found that SPARKLING works like magic:

It saves money: By expanding the library in the middle of the process, they saved up to 35% of the total construction cost (computing power) compared to building the big library from scratch.
It works better: Despite using less energy, the final library was actually smarter and performed better on tests than libraries built the traditional, expensive way.

The Bottom Line

SPARKLING is a new way to grow AI models that says: "Don't just copy-paste new parts; tune them to fit the existing sound, and then give them a little push to be unique." This allows us to build massive, powerful AI systems much faster and cheaper than before, without the system falling apart during the expansion.

Technical Summary: SPARKLING

Problem Statement

Progressive Learning (PL) aims to reduce the computational overhead of training Large Language Models (LLMs) by gradually expanding model scale during training rather than training the full scale from scratch. While existing PL research has extensively explored depth expansion (e.g., layer stacking), width expansion remains significantly understudied. Prior width-expansion methods are largely limited to the early stages of training (e.g., <10–30% of tokens), offering negligible computational advantages over training the target-width model from scratch.

Expanding width during the mid-stage of pre-training is essential for maximizing computational savings but presents formidable challenges. The paper identifies two core issues that cause training instability and performance degradation in this regime:

Signal Preservation Failure: Naive initialization at the expansion point disrupts the statistical distribution of intermediate activations (specifically the root-mean-square or RMS scale). This mismatch alters layerwise signal magnitudes, destabilizing optimization even if the immediate loss does not spike.
Symmetry Locking: Copy-based initialization, while preserving the forward function, induces backward symmetry. Duplicated channels receive identical gradients and, if optimizer states are also symmetric, evolve identically. This prevents the new capacity from diversifying into distinct features, rendering the expanded parameters functionally redundant. This issue persists across different optimizer families, including element-wise optimizers (AdamW) and spectral-style optimizers (Muon).

Methodology: SPARKLING

The authors propose SPARKLING (balancing Signal Preservation And symmetRy Keeping for width-progressive LearnING), a framework designed to balance signal preservation and symmetry breaking during mid-stage width expansion. The framework consists of four concrete mechanisms:

1. Signal Preservation via RMS-Scale Consistency

The authors argue that true signal preservation requires maintaining the statistical distribution of hidden representations, specifically the RMS scale, rather than just function preservation (loss continuity).

Mechanism: They derive rescaling factors for weights during expansion to ensure the output RMS scale remains invariant.
- Fan-out expansion: New output channels are initialized by copying or random sampling from the same distribution as existing weights.
- Fan-in expansion: New input channels require specific weight rescaling. If new coordinates are random or one-sided copied, weights are scaled by $\sqrt{d_{in}/d'_{in}}$ . If both sides are copied (duplicating existing dimensions), weights are scaled by $1/\sqrt{1+3c}$ (where $c$ is the copy ratio) to account for the quadratic contribution of covariance.
- RMSNorm Expansion: New RMSNorm parameters are initialized by copying or random sampling without additional rescaling, as the parameter count is negligible relative to linear layers.

2. Symmetry Breaking via Asymmetric Interventions

To resolve the "symmetry lock" where copied components fail to diversify, SPARKLING introduces targeted interventions that act only on backward dynamics while preserving the forward function.

Asymmetric Optimizer State Reset: Instead of copying or globally resetting optimizer states (momentum), the framework retains the optimizer states for the original parameters ( $W$ ) but resets the states for the newly introduced parameters ( $\tilde{W}$ ) to zero. This breaks the gradient coupling immediately, allowing new parameters to diverge.
Asymmetric Learning Rate Re-warmup: At the expansion point, the original parameters continue on the baseline learning rate schedule. The new parameters undergo a re-warmup starting from the current learning rate ( $\eta_e$ ) to a higher peak ( $\hat{\eta}_{max} = \rho \cdot \eta_e$ ) over a short window ( $\tau_w$ ), before decaying back to the shared minimum. This injects controlled optimization asymmetry to stimulate feature diversification without globally perturbing the well-adapted pre-trained model.

Key Contributions

Identification of Mid-Stage Challenges: The paper establishes that successful mid-stage width expansion hinges on two complementary principles: Signal Preservation (stabilizing activation statistics via RMS consistency) and Symmetry Breaking (resolving gradient coupling in copy-based initialization).
The SPARKLING Framework: A practical implementation combining RMS-preserving scaling, copy-based initialization, asymmetric optimizer state reset, and asymmetric learning rate re-warmup.
Empirical Validation: Extensive experiments across dense and Mixture-of-Experts (MoE) architectures, multiple width axes (hidden dimension and MoE expert intermediate dimension), and optimizer families (AdamW and Muon).

Results

Performance: Under a fixed token budget, SPARKLING consistently outperforms training the full-scale model from scratch on downstream evaluations (e.g., ARC, MMLU, HellaSwag) and matches or exceeds the performance of naive function-preserving baselines.
Efficiency: SPARKLING reduces training costs significantly. Under a 2× width expansion, it achieves up to 35% reduction in training FLOPs and up to a 1.49× wall-clock speed-up compared to training the expanded model from scratch.
Convergence: RMS-preserving scaling improves late-stage convergence compared to naive unscaled initialization. Asymmetric interventions (state reset and re-warmup) are critical for breaking the symmetry lock, with copy-copy initialization combined with these interventions achieving the best final loss among all variants.
Generality: The method generalizes to dense models, different optimizers (including Muon), and iterative multi-stage expansions (e.g., 256→512→1024).

Significance and Claims

The paper claims that SPARKLING makes mid-stage width expansion practically viable, addressing the primary motivation of Progressive Learning: reducing training costs while retaining or surpassing the performance of target-width models.

The authors emphasize that their approach resolves the tension between forward continuity (maintaining the pre-trained function) and backward diversity (allowing new parameters to learn distinct features). By decoupling these two aspects—preserving signal statistics while breaking symmetry in optimization dynamics—SPARKLING enables efficient scaling of LLMs. The work suggests that the framework offers a more favorable compute-loss scaling behavior, achieving lower loss at matched compute budgets compared to from-scratch training.

The paper concludes by noting that while results are promising, future work is needed to establish a unified principle for simultaneous width and depth expansion and to investigate whether the RMS preservation strategy satisfies the $\mu$ P (maximal update parameterization) condition for hyperparameter transferability.