📊 statistics

Vision Transformer Finetuning Benefits from Non-Smooth Components

This paper demonstrates that prioritizing the adaptation of high-plasticity (less smooth) components, specifically attention and feedforward layers, significantly improves Vision Transformer finetuning performance, challenging the prevailing assumption that smoothness is always desirable for transfer learning.

Original authors: Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a highly trained expert, like a master chef who has spent years cooking thousands of different dishes. This chef is your Vision Transformer (ViT), a powerful AI model that understands images. Now, you want this chef to specialize in a new, specific task, like making perfect sushi. This process of teaching the expert a new skill is called finetuning.

For a long time, researchers believed that to teach this chef well, you needed to keep their movements smooth, steady, and predictable. You wanted them to react gently to new ingredients. But this new paper argues that this "smoothness" might actually be holding the chef back. Instead, the paper suggests that the most successful chefs are the ones who are willing to be a little rough, reactive, and "plastic" (flexible).

Here is the breakdown of their findings using simple analogies:

1. The Concept: Smoothness vs. Plasticity

Think of the AI model as a machine with different parts:

LayerNorm (The Shock Absorbers): These parts smooth out the data, making sure nothing changes too wildly. They are like the suspension on a car, keeping the ride steady.
Attention Modules & Feedforward Layers (The Engine & Steering): These are the parts that actually look at the data, decide what is important, and change the output.

The paper introduces a concept called Plasticity.

Low Plasticity (Smooth): If you push the car, it barely moves. It's very stable, but it's hard to steer it quickly to a new destination.
High Plasticity (Non-Smooth): If you push the car, it reacts immediately and dramatically. It's jumpy and sensitive, but it can change direction very fast.

2. The Big Discovery

The researchers ran over 1,000 experiments to see which parts of the AI model should be updated when teaching it a new task. They found a surprising pattern:

The "Smooth" parts (LayerNorms) are actually the worst to update. Because they are designed to be stable and unchanging, they resist learning new things. Updating them is like trying to steer a car by only adjusting the shock absorbers; it's slow and ineffective.
The "Rough" parts (Attention and Feedforward layers) are the best to update. These parts have high plasticity. They are naturally sensitive and reactive. When you tweak them, they change the model's behavior quickly and effectively, allowing it to learn the new task (like sushi making) much faster and better.

3. The Analogy of the Loss Landscape

Imagine the learning process is like a hiker trying to find the bottom of a valley (the best solution).

Smooth components are like walking on a flat, muddy plain. You take small, careful steps, but you move very slowly. You might get stuck in a small dip and never find the true bottom.
High-plasticity components are like walking on a steep, rocky slope. You take big, bold steps. You might stumble a bit, but you can cover ground quickly and find the bottom of the valley much faster.

4. What This Means for Practitioners

If you are an engineer trying to adapt a large AI model to a new job, this paper gives you a clear rule of thumb:

Don't waste time trying to update the "smooth" parts (the normalization layers). They are too rigid.
Focus your energy on the "non-smooth" parts (the attention mechanisms and the feedforward layers). These are the flexible, reactive parts that drive the learning.

Summary

The paper flips the old idea on its head. We used to think that making AI models "smooth" and stable was always good. This research shows that for learning new tasks, you actually want some roughness and flexibility. The parts of the AI that are most sensitive to change (high plasticity) are the ones that learn the best.

In short: To teach an AI a new trick, don't try to smooth it out. Let the parts that are already jumpy and reactive do the heavy lifting.

Technical Summary: Vision Transformer Finetuning Benefits from Non-Smooth Components

Problem Statement

While the smoothness of transformer architectures has been extensively studied regarding generalization, training stability, and adversarial robustness, its specific role in transfer learning (finetuning) remains poorly understood. Prevailing assumptions suggest that promoting smoothness (e.g., via Lipschitz regularization) is beneficial for learning. However, in the context of adapting foundation models to downstream tasks, excessive smoothness may constrain a component's ability to adapt its outputs to changes in inputs, thereby hindering performance. The paper addresses the question: Which transformer components should be prioritized during finetuning, and why?

Methodology

The authors propose a systematic, component-wise analysis of Vision Transformers (ViTs) based on a new metric called plasticity.

1. Definition of Plasticity

The paper defines the plasticity of a transformer component $f$ as its average rate of change with respect to input perturbations. Formally, for a component $f$ and a uniform distribution $\nu$ over distinct pairs of token sequences $(x, y)$ , plasticity $P(f)$ is:
$P(f) = \mathbb{E}_{(x,y)\sim\nu} \left[ \frac{\|f(x) - f(y)\|_F}{\|x - y\|_F} \right]$

High Plasticity ( $P(f) > 1$ ): Indicates low smoothness (non-smooth), where the component amplifies input changes on average.
Low Plasticity ( $P(f) < 1$ ): Indicates high smoothness, where the component contracts input changes.

2. Theoretical Analysis

The authors derive theoretical upper bounds on the plasticity of standard ViT components:

LayerNorm: Proven to have a tight upper bound dependent on the inverse of the minimal standard deviation of tokens and the weight norm.
Feedforward Layers (FC): Bounded by the spectral norm of the weight matrices.
Multi-Head Self-Attention (MHA): Bounded by a term involving the spectral norms of weights and the sequence length $n$ . The analysis shows that MHA has a looser bound (higher potential plasticity) compared to LayerNorms and linear layers, particularly as sequence length increases.

This leads to a theoretical ranking of plasticity: MHA $\to$ FC1 $\approx$ FC2 $\to$ LN2 $\to$ LN1.

3. Empirical Validation

Plasticity Measurement: The authors computed the plasticity metric on pretrained ViTs (Base, Large, Huge) across 11 diverse classification benchmarks (including CIFAR, DomainNet, Flowers, Pets) and extended the analysis to self-supervised models (DINOv3) and LLMs (GPT-2).
Finetuning Experiments: To isolate the effect of specific components, the authors conducted over 1,000 finetuning runs on an 86M-parameter ViT. In each run, only one specific type of component (e.g., only MHA, only FC1, only LN1) was updated while others were frozen.
Optimization: Experiments were performed using both SGD and Adam optimizers across a sweep of learning rates and random seeds to ensure robustness.

Key Results

1. Empirical Plasticity Ranking

Experimental data confirmed the theoretical ranking. Across all benchmarks and model scales (ViT-Base to ViT-Huge, and extending to DINOv3 and GPT-2):

MHA consistently exhibited the highest plasticity ( $P(f) > 1$ ).
Feedforward layers (FC1, FC2) followed with high plasticity.
LayerNorms (LN2, LN1) exhibited the lowest plasticity ( $P(f) < 1$ ), acting as "rigid" or smooth components.

2. Finetuning Performance

The component-wise finetuning experiments revealed a direct correlation between plasticity and downstream performance:

Superior Accuracy: Adapting high-plasticity components (MHA and FC layers) consistently yielded higher top-1 accuracy than adapting low-plasticity components (LayerNorms).
Statistical Significance: On challenging datasets (e.g., CIFAR-100, Sketch, Clipart), MHA and FC layers significantly outperformed LayerNorms. For instance, tuning MHA alone often surpassed full finetuning baselines on small datasets.
Robustness: High-plasticity components demonstrated greater stability across different learning rates and initialization seeds, showing smaller variance in performance.
Optimization Dynamics: The paper observed that high-plasticity components generate larger gradient norms during training, leading to steeper loss descent and faster convergence toward local minima. This contrasts with smooth components, which constrain gradient norms and result in slower optimization.

3. Comparison with Parameter-Efficient Methods (PEFT)

The study found that selectively finetuning high-plasticity components (e.g., MHA) can be competitive with or superior to popular PEFT methods like LoRA, even with significantly fewer trainable parameters. For example, finetuning only LayerNorms with Adam outperformed LoRA on CIFAR-100 with 15 $\times$ fewer parameters, though MHA tuning generally yielded the best results.

Significance and Claims

The paper claims to offer a novel perspective on the functional properties of transformers, challenging the conventional wisdom that smoothness is universally desirable for learning.

Paradigm Shift: The authors argue that while smoothness aids generalization and stability in pretraining, non-smoothness (high plasticity) is the critical factor enabling effective adaptation during finetuning.
Practical Guidance: The work provides principled guidance for practitioners: when designing parameter-efficient finetuning strategies, priority should be given to attention modules and feedforward layers rather than normalization layers.
Theoretical-Practical Bridge: The paper successfully bridges theoretical bounds on Lipschitz constants with empirical optimization dynamics, demonstrating that the "plasticity" of a module dictates its capacity to carry information about downstream data and facilitate gradient-based optimization.

The authors conclude that understanding and leveraging the plasticity of non-smooth components is essential for designing more efficient adaptation methods for foundation models.