🤖 machine learning

Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

This paper introduces Mean Root Square Normalization (MRSNorm), a novel normalization technique that pairs channels into 2D phasors to invert the traditional scaling paradigm, thereby reducing parameters by half while enforcing geometric constraints that ensure gradient homogeneity and prevent numerical instability.

Original authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Published 2026-07-21

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, hyper-intelligent robot how to recognize patterns, like spotting a cat in a photo or finishing a sentence. To do this, the robot uses a massive digital brain made of layers of math. But here's the problem: as the robot gets deeper and smarter, its internal calculations can get a little crazy. Sometimes, one tiny number gets too big, blowing up the whole calculation like a sugar rush in a nervous system. Other times, the robot gets so focused on that one big number that it forgets to listen to all the other, quieter numbers, leaving them "starved" for attention. This is a bit like a classroom where one loud student screams so hard that the teacher can't hear anyone else, or the chalkboard explodes because someone wrote a number too big for the space.

To fix this, scientists use special "rules" called normalization layers. Think of these as the teacher's gentle hand, constantly smoothing out the numbers so they stay in a healthy, manageable range. For a while, the most popular rule was called RMSNorm. It was fast and efficient, but it had a secret weakness: it relied on squaring numbers (multiplying them by themselves) to find an average. In the digital world, squaring a number that is already a little too big makes it massive instantly. This can cause the robot's brain to crash or stop learning properly, especially when the robot is trying to learn very quickly or with very small groups of data.

This paper introduces a new, clever rule called MRSNorm (Mean Root Square Normalization). Instead of squaring numbers and hoping for the best, MRSNorm changes the order of operations. It pairs up numbers like dance partners, treats them as a single unit, and averages them in a way that keeps the whole system balanced. The researchers found that by doing this, they could not only stop the robot's brain from exploding but also cut the number of "knobs" the robot needs to tune in half, making it faster and more stable. They tested this on a standard image-recognition model and found that while other methods crashed when the learning speed was turned up high, MRSNorm kept dancing smoothly, proving that sometimes, changing the rhythm is better than just turning up the volume.

The Dance of the Phasors

To understand how MRSNorm works, imagine a group of dancers. In the old way (RMSNorm), every dancer was treated as a separate person. If one dancer started spinning wildly fast (a "magnitude outlier"), the teacher would calculate the average speed by squaring everyone's speed. That one fast dancer would make the average speed skyrocket, causing the teacher to panic and stop the music for everyone. The other dancers, who were spinning normally, would get ignored because the teacher was too busy dealing with the chaos.

MRSNorm changes the choreography completely. Instead of treating dancers as individuals, it pairs them up into 2D phasors. Think of this as tying two dancers together with a rope, so they move as a single unit. Now, instead of looking at how fast each person is spinning, the teacher looks at the size of the pair's movement.

Here is the magic trick: MRSNorm calculates the "size" of each pair first (the Root Square), and then takes the average of all those sizes (the Mean). This is the opposite of the old way, which averaged the squares first. By doing this, MRSNorm creates a "phasor manifold." Imagine this as a special dance floor where the dancers are strictly forbidden from running off the edge. No matter how hard they try to spin, the rules of the dance floor (the math) ensure they stay within a safe, circular boundary. This prevents any single dancer from becoming so loud that they drown out the rest of the orchestra.

The Secret Superpower: Gradient Homogeneity

The paper suggests that this new dance floor has a hidden superpower called Gradient Homogeneity. In the world of robot learning, "gradients" are the signals that tell the robot how to improve. If a signal is too weak, the robot doesn't learn (gradient starvation). If it's too strong, the robot goes crazy (gradient explosion).

The authors explain that because MRSNorm pairs dancers and treats them as a unit, it naturally creates a "trigonometric gradient clipper." This is a fancy way of saying the math itself acts like a safety valve. Because of a famous math rule called the Pythagorean identity (the one about triangles where $a^2 + b^2 = c^2$ ), the "strength" of the learning signal for every pair is automatically equalized. It doesn't matter if one dancer is huge and the other is tiny; together, they always send a perfectly balanced signal. This happens automatically, without the robot needing to be told to "be careful." It's like having a self-correcting compass that always points north, no matter how stormy the weather gets.

Cutting the Clutter

Another surprising finding is that MRSNorm is incredibly efficient. In the old methods, every single dancer needed their own special "weight" or "knob" to adjust how they moved. MRSNorm, however, shares a single knob between the paired dancers. This means the robot needs half as many learnable parameters.

The paper argues that having a separate knob for every single channel was actually a mistake—a "harmful redundancy." It was like giving every dancer in a pair a different volume control, which just made the music sound messy and distorted. By forcing the pair to share one control, MRSNorm removes the noise and lets the robot focus on the actual shape of the data, rather than getting distracted by unnecessary adjustments.

The Stress Test: When Things Get Extreme

To prove their idea works, the researchers put MRSNorm through a series of "stress tests." They took a standard image-recognition model (ResNet) and tried to teach it to recognize 100 different types of images (CIFAR-100). They didn't just test it under normal conditions; they turned the learning speed (learning rate) way up and made the training groups (batch sizes) very small. This is like asking a student to learn a whole textbook in one hour while being distracted by loud noises.

Under these extreme conditions, the old methods (LayerNorm and RMSNorm) completely failed. The researchers observed that when the learning rate was set to 0.3 or 0.4 (which is 3 to 4 times the standard speed), the old models suffered a "catastrophic optimization collapse." Their accuracy dropped to zero almost immediately, and the numbers in their brains exploded into errors (NaNs).

In contrast, MRSNorm held its ground. Even at the extreme learning rate of 0.4, MRSNorm was the only method that didn't crash instantly. While it did struggle a bit with very small groups (batch size of 32), it successfully trained with a batch size of 128 and maintained stable learning trajectories where others failed. The paper notes that in these simulations, MRSNorm prevented the "numerical explosion" that usually happens when outliers dominate the calculation.

Why 2D is the Magic Number

The authors also explain why this pairing is necessary. They argue that trying to do this with just one number (1D) is mathematically impossible if you want to keep the learning signals stable and periodic. It's like trying to draw a perfect circle with a single straight line; you just can't do it. You need at least two dimensions (a 2D plane) to create a stable, repeating pattern that doesn't break.

Furthermore, they warn against simply forcing every pair to be the exact same size (unit-norm projection). They explain that doing so creates a "Radial Gradient Blockade," which is like cutting the rope between the dancers and the teacher. If you force the size to be exactly 1, the robot loses the ability to learn about the size of the information, only its direction. MRSNorm avoids this by using a global average (the Mean) rather than forcing every single pair to be identical, allowing the robot to learn both the direction and the importance of the data.

A New Way to Listen

Finally, the paper suggests that this method could change how robots pay attention to information. In standard attention mechanisms, if one piece of information gets too big, it drowns out everything else. MRSNorm acts like a "soft geometric constraint" that turns the robot's attention into an "Energy-Weighted Cosine Similarity."

Imagine a voting system. In the old way, the loudest voice wins, regardless of whether the voice is saying something important. With MRSNorm, the vote is based on both the direction of the idea (does it make sense?) and the energy of the idea (is it important?). The math shows that this new method naturally balances these two factors, allowing the robot to focus on the most informative parts of the data without getting overwhelmed by the loudest, most chaotic numbers.

In short, the paper proposes a fundamental shift: instead of trying to manage chaos by squaring numbers and hoping for the best, we should pair them up, respect their geometry, and let the math naturally keep the peace. The results suggest that this approach offers a more stable, efficient, and robust way to build the next generation of deep learning models.

Technical Summary: Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

Problem Statement
While Root Mean Square Normalization (RMSNorm) has become the standard for accelerating modern sequence models due to its computational efficiency over Layer Normalization (LN), it possesses a critical structural vulnerability. RMSNorm relies on the quadratic accumulation of independent scalar variances ( $\sum x^2$ ). In mixed-precision environments or under extreme optimization conditions, this mechanism allows a single magnitude outlier to disproportionately dominate the denominator. This leads to catastrophic numerical explosions (NaNs), gradient starvation for low-magnitude channels, and anisotropic phase distortion. Furthermore, traditional normalization methods often introduce redundant degrees of freedom through unconstrained spatial scaling, which can distort the geometric phase information essential for stable deep representation learning.

Methodology: Mean Root Square Normalization (MRSNorm)
The authors propose Mean Root Square Normalization (MRSNorm), a method that mathematically inverts the traditional scaling paradigm to preserve the phase manifold. The core methodology involves three structural shifts:

2D Phasor Pairing: Instead of treating channels as independent 1D scalars, MRSNorm restructures the feature space by pairing adjacent channels into 2D phasors ( $z_i = [x_i, y_i]^T$ ).
Operational Inversion (Mean of Roots): Rather than computing a global quadratic mean of scalars, MRSNorm first computes the localized $L_2$ magnitude (Root Square) for each 2D phasor, and subsequently aggregates these magnitudes via a global $L_1$ average (Mean). The scaling factor $S_{MRS}$ is defined as:
$S_{MRS} = \frac{1}{P} \sum_{i=1}^{P} \sqrt{x_i^2 + y_i^2 + \epsilon}$
where $P$ is the number of phasor pairs.
Conformal Invariance and Parameter Sharing: The normalized output is obtained via isotropic vector scaling where a single affine weight ( $\gamma_i$ ) is shared across the components of each 2D phasor. This ensures that the phase angle ( $\theta_i$ ) of the original phasor is strictly preserved ( $\theta(\hat{z}_i) = \theta_i$ ). By sharing weights across the 2D plane, MRSNorm halves the total number of learnable affine parameters compared to standard norms, eliminating the redundancy that distorts phase.

Key Contributions and Theoretical Insights
The paper provides analytical proofs demonstrating that MRSNorm offers structural advantages over existing methods:

Built-in Gradient Clipping via Pythagorean Identity: The authors demonstrate that the geometric constraint of the 2D phasor manifold yields a built-in, trigonometric gradient clipper. Because the unit direction vector $u_i$ satisfies $\|u_i\|_2 = \sqrt{\cos^2 \theta_i + \sin^2 \theta_i} \equiv 1$ , the local gradient update magnitude for each phasor is unconditionally equalized. This enforces Gradient Homogeneity, preventing gradient starvation without distorting the directional optimization trajectory.
Resolution of the 1D Periodicity Paradox: The paper argues that stable periodic gradient propagation is structurally impossible in 1D real analysis for continuously differentiable functions with constant gradient norms. MRSNorm overcomes this by ascending to 2D phasor space, where the Pythagorean identity allows components to behave as smooth, periodic functions while maintaining a constant update norm.
Avoidance of Radial Gradient Blockade: Unlike instance-wise amplitude normalization (e.g., Squashing functions) which creates a "Radial Gradient Blockade" by making the Jacobian singular in the direction of the vector's magnitude, MRSNorm uses ensemble-based scaling (global $L_1$ average). This keeps the radial gradient flow unblocked and stable.
Geometric Resolution of Logit Imbalance: In the context of attention mechanisms, the paper suggests that integrating MRSNorm transforms the standard dot product into an Energy-Weighted Cosine Similarity Operator. This allows the network to dynamically allocate magnitude to informative phasor blocks while suppressing irrelevant features, potentially mitigating "softmax collapse" caused by magnitude outliers.

Experimental Results
Empirical evaluations were conducted using a ResNet architecture trained on the CIFAR-100 dataset under rigorous stress tests involving extreme hyperparameter settings (high learning rates $lr \in \{0.1, 0.3, 0.4\}$ and small batch sizes $bs \in \{32, 64, 128\}$ ).

Standard Regimes: Under standard learning rates ($lr=0.1$), MRSNorm performed comparably or superiorly to LayerNorm and RMSNorm, particularly maintaining stability when batch sizes were reduced to 32, where RMSNorm exhibited high variance and instability.
Extreme Regimes: Under extreme conditions ( $lr \ge 0.3$ ), standard LayerNorm and RMSNorm suffered from catastrophic optimization collapse, with training accuracy flattening near zero. In contrast, MRSNorm maintained stable gradient flow. Notably, at $lr=0.4$, MRSNorm was the only method that prevented immediate numerical divergence, successfully converging even with a batch size of 128.
Parameter Efficiency: Despite having half the number of learnable affine parameters, MRSNorm did not compromise representational capacity, validating the claim that unconstrained spatial scaling in standard norms is a harmful redundancy.

Significance and Claims
The authors position MRSNorm as a fundamental paradigm shift toward phasor-based deep representation learning. The paper claims that by strictly constraining activations to a phasor manifold and enforcing conformal invariance, MRSNorm resolves the inherent numerical instability and gradient starvation issues of quadratic accumulation norms. The method proves that geometric constraints can serve as a more robust alternative to heuristic gradient clipping, ensuring stable optimization trajectories even under conditions where standard normalizations fail. The implementation is provided in the paper's appendix.