Technical Summary: Nora (Normalized Orthogonal Row Alignment)

Problem Statement

The training of Large Language Models (LLMs) relies heavily on adaptive optimizers like Adam, which treat parameters as flat vectors, thereby ignoring the rich structural interactions within the loss landscape. While matrix-based optimizers like Muon have emerged to address this by applying orthogonalized updates and Newton-Schulz iterations, they incur prohibitive computational costs of $O(m^2n)$, limiting scalability.

Conversely, simplified approaches like RMNP (Row-wise Momentum Normalized Preconditioning) reduce complexity to $O(mn)$ by leveraging the row-block diagonal dominance of the Transformer Hessian. However, RMNP fails to account for the scale-invariance inherent in modern neural networks (due to BatchNorm, RMSNorm, LayerNorm). It allows "radial updates" (updates aligned with the weight vector) which do not change the functional output but introduce noise, destabilize the effective learning rate, and cause chaotic oscillations in weight norms.

Existing methods generally fail to unify three core desiderata: efficiency (Muon-like preconditioning), stability (adherence to scale-invariance), and speed (minimal computational overhead).

Methodology: The Nora Optimizer

The authors propose Nora (Normalized Orthogonal Row Alignment), an optimizer designed to satisfy all three desiderata simultaneously. The core design philosophy rests on the geometric observation that for scale-invariant parameters, effective learning occurs along angular (tangential) directions rather than radial ones.

Algorithmic Steps

For a weight matrix $w \in \mathbb{R}^{m \times n}$ and momentum $v_t$:

1. Row-wise Orthogonal Projection: Nora explicitly projects the momentum $v_t$ onto the row-wise orthogonal complement of the weights $w$. For each row $i$, the component parallel to the weight row is removed:
   $$v_{t,i:}^{\perp} = v_{t,i:} - \frac{\langle v_{t,i:}, w_{t,i:} \rangle}{\|w_{t,i:}\|^2} w_{t,i:}$$
   This ensures $\langle v_{t,i:}^{\perp}, w_{t,i:} \rangle = 0$, strictly filtering out radial noise and stabilizing weight norms.

2. Row-wise Normalization (Preconditioning): Leveraging the row-block diagonal dominance of the Transformer Hessian, Nora approximates the structured preconditioning of Muon. Instead of expensive Newton-Schulz iterations, it applies a diagonal preconditioner to the projected momentum. Algebraically, this simplifies to a simple row-wise $\ell_2$ normalization:
   $$d_t = \text{RN}(v_t^{\perp}) \implies d_{t,i:} = \frac{v_{t,i:}^{\perp}}{\|v_{t,i:}^{\perp}\|_2}$$
   This step achieves Muon-like preconditioning efficiency while maintaining the orthogonality established in the projection step.

3. Update: The weights are updated using the normalized direction $d_t$ with a learning rate $\eta_t$.

Theoretical Foundations

• Scalability: Using the Maximal Update Parametrization ($\mu$P) framework, the authors prove that for Nora to achieve stable feature learning as network width $n \to \infty$, the learning rate must scale as $\eta_t \propto 1/\sqrt{n}$.
• Convergence: The paper provides rigorous convergence guarantees for non-convex optimization landscapes. It establishes that Nora reaches an $\epsilon$-stationary point with a complexity of $O(m L \sigma^2 \Delta \epsilon^{-4})$ under matched $(\infty, 2)$-smoothness assumptions.
• Implementation: The core logic requires only two lines of matrix-based code, making it a plug-and-play module compatible with existing frameworks.

Key Contributions

1. Novel Optimizer Design: Introduction of Nora, which unifies efficient preconditioning estimation, stability via scale-invariance respect, and high computational speed.
2. Theoretical Rigor: Proof that Nora is a scalable optimizer with derived scaling laws ($\eta \propto 1/\sqrt{n}$) and non-convex convergence guarantees.
3. Empirical Validation: Extensive experiments training LLaMA-style models (60M and 135M parameters) demonstrate that Nora outperforms Muon, Mano, and RMNP in both convergence speed and wall-clock execution time.

Experimental Results

The authors evaluated Nora on autoregressive language modeling tasks across various learning rates and model scales:

• Performance: On the 135M model, Nora achieved the lowest validation loss (3.079) and perplexity (21.74) among all compared optimizers (Muon: 3.142/23.17; Mano: 3.097/22.13; RMNP: 3.112/22.46).
• Training Dynamics: While RMNP showed strong early-stage performance, it plateaued. Nora started with higher initial loss but continued to improve late in training, overtaking other methods to reach the best final metrics.
• Efficiency: Runtime benchmarks on representative matrix shapes (e.g., 1B-scale MLP matrices) showed that Nora's row normalization is approximately 70 times faster than the 5-step Newton-Schulz orthogonalization used by Muon.
• Ablation: Comparisons with Mano (a similar row/column projection method) confirmed that Nora's superiority stems from its specific reliance on row-block diagonal dominance and its streamlined architecture, rather than hyperparameter tuning or weight decay settings.

Significance and Claims

The paper claims that Nora demonstrates that efficiency, stability, and speed can be unified through a simple row-wise geometric principle. By rigorously stabilizing weight norms and angular velocities while approximating structured preconditioning, Nora offers a "practical and principled direction for scalable optimizer design." The authors emphasize that Nora achieves state-of-the-art data efficiency without the prohibitive computational overhead of full matrix orthogonalization, making it a highly promising candidate for large-scale LLM training.