🤖 machine learning

LionVote: Per-Layer Learning Rate Adaptation for Lion

This paper introduces LionVote, a per-layer learning rate adaptation mechanism that utilizes gradient stability and momentum diagnostics to dynamically adjust learning rates, achieving statistically significant accuracy improvements over standard Lion and AdamW on ViT-Tiny/CIFAR-100 by addressing the inherent cross-layer learning rate disparities in Transformer architectures.

Original authors: Kris Atallah (New York University, New York, USA)

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Kris Atallah (New York University, New York, USA)

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a giant digital brain to recognize cats, dogs, and cars. This brain is made of many different layers, like a multi-story building. Usually, when we teach this brain, we give every single floor the exact same amount of "study time" (a learning rate) at the same speed. It's like a teacher telling the math class, the art class, and the gym class to all run at the exact same pace, regardless of what they are doing.

But what if the math students need to sprint, while the art students need to stroll?

That's the problem LionVote solves. It's a new way to train a specific type of digital brain called "Lion." The researchers found that if you treat every part of the brain the same, some parts get overwhelmed and others get bored.

The Big Discovery: The Brain's "Effective Scale" is Off

The authors measured how the Lion brain learns and found something surprising. On a specific test (using a model called ViT-Tiny on the CIFAR-100 dataset), the brain's "effective scale" was 2.6 to 2.8 times too high for the parts that handle attention and complex thinking (MLP parameters). For the parts that keep everything organized (normalization layers), it was about 2 times too high.

Think of it like this: The attention parts of the brain are trying to drink from a firehose, while the organization parts are drinking from a garden hose. The attention and MLP parts are actually receiving a 32% higher effective scale than the normalization parts, even though the teacher (the global learning rate) is turning the faucet to the same setting for both. This mismatch makes it hard for the brain to learn efficiently.

The Solution: A Voting System for Every Layer

To fix this, the researchers invented LionVote. Instead of one teacher shouting instructions to the whole building, they gave every single layer of the brain its own tiny, persistent "level counter."

Here is how the voting works:

The Compound Level: Every layer has a score (an integer) that starts at zero. This score acts like a volume knob, turning the learning rate up or down.
The Two Votes: Every few epochs (training cycles), the layer checks its own health using two specific tests:
- Vote 1 (Direction Check): Is the layer moving in a steady direction, or is it wobbling back and forth? If the gradient (the direction of learning) is stable, it gets a "thumbs up." If it's flipping wildly, it gets a "thumbs down."
- Vote 2 (Momentum Health): Is the layer's momentum (its speed and inertia) too strong compared to its current progress? If it's spinning out of control, it gets a "thumbs down."
The Tiebreaker: Sometimes the two votes disagree (one says "go faster," the other says "slow down"). When this happens, the system looks at the validation loss (a score of how well the brain is doing on a practice test). If the practice test score improved, it votes to keep going; if it got worse, it votes to slow down.

Based on these votes, the layer's "volume knob" (the compound level) turns up or down. If a layer is stable, it might get a boost. If it's chaotic, it gets a timeout.

The Results: A Small but Real Win

When they tested this on the ViT-Tiny model:

LionVote achieved 69.71% accuracy.
The standard Lion optimizer got 68.95%.
The popular AdamW optimizer got 68.75%.

The difference between LionVote and standard Lion is statistically significant (meaning it's likely a real improvement and not just luck), with a probability of less than 2% that this happened by chance. On the ViT-Tiny/CIFAR-100 test, LionVote also beat AdamW, though on the ViT-Tiny/CIFAR-10 test, LionVote tied with AdamW rather than beating it.

What This Does NOT Mean

It's important to know what this method doesn't do, because the paper is very clear about that:

It's not a magic bullet for everything. On simpler, uniform networks (like WideResNet), the old-school method of tuning the learning rate manually (using SGD with cosine annealing) still wins. LionVote didn't help there; in fact, on some tests, it made things slightly worse.
It's not a permanent fix for all tasks. The benefit depends heavily on how different the layers are. The more "heterogeneous" (different) the architecture is, the more LionVote helps. On the uniform WideResNet, the layers were too similar for the voting system to find much to fix.
It's not a "set it and forget it" for all optimizers. The specific rules for the "Momentum Health" vote were derived specifically for the Lion optimizer. If you tried to use these exact rules on a different optimizer like Adam, they might not work because the math behind the momentum is different.

The Takeaway

The paper suggests that per-layer adaptation is a powerful tool, but only when the brain's architecture is complex enough to have different needs. LionVote proves that by letting each layer vote on its own learning speed, we can squeeze out a little extra performance (about 0.7 percentage points on this specific test) without needing to guess the perfect settings by hand.

It's like realizing that in a busy school, the math teacher, the art teacher, and the gym teacher shouldn't all be shouting the same instructions at the same volume. Sometimes, the math class needs a whisper, and the gym class needs a megaphone. LionVote is the system that lets each class decide for itself.

Technical Summary: LionVote: Per-Layer Learning Rate Adaptation for Lion

Problem Statement

Deep neural networks are not monolithic; different layer types (e.g., attention, MLP, normalization) exhibit distinct optimization characteristics. Standard optimizers like Lion apply a single global learning rate (LR) to all parameters, potentially leading to miscalibration. Specifically, the Lion optimizer's sign-based update rule discards gradient magnitude, preventing directionally unstable layers from compensating via larger updates. This makes Lion particularly sensitive to per-layer LR miscalibration compared to second-moment methods like Adam.

Prior work has shown that per-layer adaptive preconditioning can recover advantages of adaptive methods, but existing approaches either operate statelessly (e.g., LARS, LAMB), apply per-coordinate adaptation without layer-level history, or rely on hardcoded layer assignments (e.g., Muon). There is a gap in methods that combine stateful layer-level adaptation with data-driven rate adjustment that does not require manual tuning of divergence magnitudes.

Methodology: LionVote

LionVote is a per-layer learning rate mechanism designed for the Lion optimizer. It introduces a persistent integer "compound level" ( $s_i$ ) for each parameter tensor, which modulates the base learning rate exponentially. The system operates without modifying Lion's core update rule but adds a voting mechanism that runs every $c$ epochs.

Core Components

Compound Level ( $s_i$ ):
Each parameter $i$ maintains a level $s_i \in \{-L, \dots, +L\}$ (default $L=4$ ). The effective learning rate is:
$\alpha_i = \text{lr} \cdot \exp(s_i \cdot \beta_1/2)$
where $\text{lr}$ is the global base rate and $\beta_1$ is Lion's momentum interpolation coefficient. At $\beta_1=0.9$ , a unit step in $s_i$ corresponds to a $\times 1.57$ multiplier.
Voting Diagnostics:
Every $c$ epochs, two diagnostics vote on whether to increase, decrease, or maintain the rate for each layer:
- Vote 1 (Gradient Direction Stability): Computes the cosine similarity between the current and previous epoch-mean gradients.
  - Thresholds: $+1$ if alignment $> 0.5$ (stable), $-1$ if $< 0$ (reversed), $0$ otherwise.
  - Derivation: The $0.5$ threshold corresponds to a $2/3$ supermajority of coordinate sign pairs agreeing under coordinate isotropy (Sheppard's formula).
- Vote 2 (Momentum Health): Computes the ratio of the momentum EMA norm to the gradient norm ( $r = \|m\| / \|\bar{g}\|$ $r = ∥ m ∥/∥ \overset{g}{ˉ} ∥$ ).
  - Thresholds: $-1$ if $r > e$ , $+1$ if $r < 1/e$ , $0$ otherwise.
  - Derivation: Derived from the intrinsic time constant of the $\beta_2$ -EMA ( $\beta_2^\tau = e^{-1}$ ).
Tiebreaker (Validation Loss):
If the two votes conflict or both abstain, a global tiebreaker resolves the decision based on the relative change in validation loss ( $\delta$ ).
- Threshold: $\pm 1\%$ , chosen to match the noise floor of cross-entropy loss estimation for typical validation set sizes.
- Note: On ViT-Tiny/CIFAR-100, this tiebreaker is the dominant signal path, firing for $\sim 59\%$ of decisions.
Update Rule:
The compound level $s_i$ is updated via an asymmetric rule: a $+1$ vote resets negative levels to 0 and increments positive ones; a $-1$ vote resets positive levels to 0 and decrements negative ones. A vote of $0$ decays the level toward zero.

Key Contributions

A Stateful Per-Layer Mechanism for Lion: The paper proposes a voting system where thresholds are derived from geometric identities and EMA time constants rather than tuned. The methodology for deriving these thresholds (e.g., the $2/3$ supermajority convention and the $e$ -factor for momentum) is presented as reusable for other optimizers.
Quantified Miscalibration in Lion: Through compound level trajectories, the authors demonstrate that on ViT-Tiny/CIFAR-100, Lion's effective scale is systematically too high:
- $2.6\times$ – $2.8\times$ too high for Attention and MLP parameters.
- $\sim 2\times$ too high for Normalization layers.
- Normalization layers receive a $32\%$ higher effective scale than Attention layers. This disparity cannot be reproduced by a single global rate reduction.
Architecture and Task Dependence: The value of per-layer adaptation is shown to depend on architectural heterogeneity and the specific task.
- ViT-Tiny: Shows significant gains on CIFAR-100 ( $69.7\%$ vs. Lion's $69.0\%$ ) due to the large spread in layer types.
- WideResNet: On uniform CNN architectures, tuned SGD with cosine annealing remains dominant, and LionVote provides no consistent advantage.
- Task Sensitivity: On ViT-Tiny, LionVote improves performance on CIFAR-100 but significantly degrades performance on CIFAR-10 ($-0.88$ pp), indicating that architectural heterogeneity alone is insufficient; task-level factors also dictate the utility of adaptation.

Experimental Results

Experiments were conducted on WideResNet and ViT-Tiny across CIFAR-10 and CIFAR-100 with 8 random seeds.

ViT-Tiny / CIFAR-100: LionVote (cadence 8) achieves 69.71% top-1 accuracy, outperforming Lion (68.95%, $p < 0.02$ ) and AdamW (68.75%). It reaches the 68% accuracy threshold $\sim 80$ epochs faster than Lion.
ViT-Tiny / CIFAR-10: LionVote (cadence 8) achieves 91.52%, which is statistically significantly lower than standard Lion (92.40%, $p < 0.001$ ).
WideResNet: Tuned SGD with cosine annealing dominates all configurations (e.g., 96.08% on WRN-28-10/CIFAR-10). LionVote does not consistently outperform Lion or SGD on these uniform CNNs.
Ablation:
- The tiebreaker is critical for gains on CIFAR-100; removing it reduces performance below Lion.
- Vote 1 (gradient stability) is the primary driver of per-layer differentiation.
- Vote 2 (momentum health) rarely fires on this workload, acting as a dormant diagnostic.
- A symmetric update rule variant achieved the study's highest single accuracy (70.54% on ViT/CIFAR-100) but with higher variance.

Significance and Claims

The paper claims that LionVote provides a principled, non-tuned mechanism for per-layer adaptation in sign-based optimizers. Its significance lies in three areas:

Methodological Reusability: The derivation of voting thresholds from mathematical structures (geometric identities, EMA time constants) offers a blueprint for designing per-layer mechanisms in other optimizers without relying on meta-learning or extensive hyperparameter tuning.
Diagnosis of Lion: It quantifies a specific miscalibration in Lion's effective scale across layer types, suggesting that future work on Lion should consider layer-type-aware scaling even without the full voting mechanism.
Contextual Value: It establishes that per-layer adaptation is not a universal panacea. Its value is contingent on architectural heterogeneity (e.g., ViTs vs. uniform CNNs) and task difficulty. The paper modestly concludes that while the mechanism works well for heterogeneous architectures like ViTs on complex tasks (CIFAR-100), it offers no consistent advantage on uniform CNNs or simpler tasks where Lion is already near its performance ceiling.

The authors note limitations, including the reliance on validation loss for the tiebreaker (which should ideally be on a held-out split), the higher cross-seed variance compared to standard Lion, and the scope being limited to CIFAR datasets and specific model sizes. They leave open questions regarding generalization to other optimizers and the potential for adaptive cadence.