Feature Evolution and Migration during Vision Transformer Training
This paper introduces a novel framework using Sparse Autoencoders to visualize feature evolution across network depth and training time in Vision Transformers, revealing that feature migration is concentrated early in training, favors movement toward shallower layers, and that deeper layers stabilize earlier than shallow ones.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of artificial intelligence, computers learn to see by processing images through vast, layered networks of digital neurons. These systems, known as Vision Transformers, have become the standard for tasks ranging from identifying animals in photos to guiding autonomous vehicles. For years, scientists have understood that these networks organize information in a hierarchy: the earliest layers detect simple shapes like edges and textures, while the deeper layers combine those shapes into complex concepts like faces or entire objects. However, a crucial mystery remained about how this internal organization actually forms. We knew the final result looked like a well-ordered library, but we did not know how the books were sorted, moved, or shelved while the computer was still learning. It was unclear whether the features the network discovered stayed in the same place from the first moment of training to the last, or if they migrated to different layers as the system matured.
To solve this puzzle, researchers at the University of Tartu developed a new way to watch the learning process unfold in real time. Instead of looking at the network as a single, static block, they mapped its internal activity across two dimensions: the depth of the network, from the first layer to the last, and the passage of time, measured in the hundreds of training cycles the model underwent. They treated the network like a living ecosystem, tracking how specific patterns of information appeared, disappeared, or shifted locations. By using a mathematical tool that breaks down complex data into simple, distinct components, the team could isolate individual "features"—specific things the computer learned to recognize, such as a particular texture or a part of an object—and follow their journey through the layers. This approach allowed them to see the hidden dynamics of learning that standard methods, which only measure overall similarity, would miss.
The researchers trained a small but representative version of a Vision Transformer on a massive dataset of one million images, teaching it to recognize thousands of different categories. As the model trained over 300 cycles, the team recorded its internal state at every step. They discovered that the organization of features is not fixed; it is surprisingly fluid, especially at the beginning of the learning process. In the early stages, features often appear in the deeper layers of the network, where the system is initially trying to make sense of the whole image. As training progresses, many of these features migrate backward, moving toward the earlier, shallower layers. It is as if the system first discovers a concept in a high-level summary and then, over time, figures out exactly where in the processing pipeline that concept belongs, settling it into a more permanent home.
This migration is not random; it follows a clear pattern. The study found that features move much more frequently toward the earlier layers than toward the deeper ones. Once a feature finds its place, it tends to stay there, becoming increasingly stable as the training continues. The deeper layers of the network stabilize first, locking in their organization early on, while the shallower layers continue to adjust and refine their contents for a longer period. By the end of the training, the system has settled into a stable configuration where features are no longer wandering between layers. The researchers also observed that this behavior is consistent regardless of the size of the dataset used, suggesting that this migration is a fundamental part of how these networks learn, rather than a quirk of a specific training set.
Perhaps most surprisingly, the team found that the features which appear early in the training process are not necessarily the ones that last the longest. Some features that emerge quickly have very short lifespans, appearing and vanishing within a few training cycles. Others, particularly those that settle into the deeper layers, persist for the entire duration of the training. The study also examined whether these moving features correspond to specific human-understandable categories, such as "dog" or "car." They found that while some features are clearly tied to specific classes, others are more general or shared across groups of related items. However, the way a feature moves or stabilizes does not seem to depend on whether it represents a specific object or a broader concept; the migration pattern is a structural property of the learning process itself.
These findings offer a new perspective on how artificial intelligence learns. Rather than building a rigid structure from the bottom up, the network appears to explore different organizational strategies, moving information around until it finds the most efficient arrangement. The early phase of training is a time of high activity and reorganization, where the system tests different locations for its learned features. As the training concludes, this activity calms down, and the network settles into a stable state. This research provides a detailed map of that journey, showing that the path to intelligence in these machines involves a dynamic process of discovery and relocation, not just a static accumulation of knowledge. By understanding how features migrate and stabilize, scientists can gain deeper insights into the inner workings of these powerful tools, potentially leading to more efficient and interpretable systems in the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.