🤖 AI

Block-Recurrent Dynamics in Vision Transformers

This paper introduces the Block-Recurrent Hypothesis and the Raptor framework to demonstrate that trained Vision Transformers possess an underlying block-recurrent structure, enabling them to be accurately approximated by significantly fewer distinct blocks while revealing low-complexity dynamical attractors and token-specific behaviors amenable to principled systems analysis.

Original authors: Mozes Jacobs, Thomas Fel, Richard Hakim, Alessandra Brondetta, Demba Ba, T. Andy Keller

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mozes Jacobs, Thomas Fel, Richard Hakim, Alessandra Brondetta, Demba Ba, T. Andy Keller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Repeating Chapter" Theory

Imagine you are reading a very thick, 100-page novel (a Vision Transformer, or ViT). Usually, we assume that every single page is written by a different author, with a unique style and a completely new plot point. It's a massive book, and we think it needs 100 unique pages to tell the story.

The authors of this paper discovered something surprising: The book isn't actually 100 unique pages. Instead, it's written using only 3 or 4 distinct chapters that are just repeated over and over again.

They call this the Block-Recurrent Hypothesis. They found that deep inside these complex AI models, the "layers" (the pages of the book) aren't all different. They fall into groups. Once the AI gets past the first few pages, it just starts reusing the same few "blocks" of code, applying them again and again to finish the job.

The Detective Work: Finding the Pattern

How did they figure this out?

The "Similarity Map": Imagine taking a photo of every page in the book and comparing it to every other page. If you look at a standard book, Page 1 looks nothing like Page 99. But when the researchers did this with AI models, they saw a strange pattern: Page 10 looked a lot like Page 11, which looked a lot like Page 12. But Page 13 looked totally different.
- The Analogy: It's like a song where the first verse is unique, then the chorus repeats three times, then a bridge, then the chorus repeats again. The "similarity map" showed these repeating blocks clearly.
The "Copy-Paste" Test (Raptor): Just because pages look similar doesn't mean they do the same thing. Maybe Page 10 and Page 11 look alike but tell different parts of the story.
- To prove they were functionally the same, the researchers built a new, tiny AI model called Raptor.
- They took the original giant AI and said, "We are going to replace your 12 layers with just 2 or 3 special layers that we will repeat."
- The Result: The tiny Raptor model, using only 2 repeating blocks, could do 96% of the work of the giant 12-layer model. It didn't just guess the final answer; it recreated the entire thought process of the giant model, step-by-step.

Why Does This Happen? (The "Stochastic Depth" Secret)

The researchers asked: Why does the AI learn to repeat itself?

They found that a training technique called Stochastic Depth (which is like randomly skipping pages while the AI is learning) forces the AI to become more efficient.

The Analogy: Imagine a student studying for a test. If you tell them, "I might randomly delete a page from your textbook tomorrow," they stop trying to memorize every single unique fact. Instead, they learn the core principles that apply to many pages. They learn a "reusable strategy."
The more the AI is trained with this "random skipping," the more it naturally organizes itself into these repeating blocks.

What Does This Tell Us About How AI Thinks?

Once they proved the AI uses repeating blocks, they used this to understand how the AI thinks, like a physicist watching water flow. They found three cool things:

The "Magnet" Effect: As the image moves through the layers, the AI's internal representation (its "thought") gets pulled toward a specific direction, like a compass needle pointing North. Once it gets close to the right "direction" (e.g., "this is a cat"), it stops wandering and locks in.
The "Team Captain" vs. The "Crowd":
- The CLS token (the AI's "summary token" or team captain) stays quiet for a while, then suddenly spins around at the very end to make a sharp decision.
- The Patch tokens (the parts of the image) move together in a synchronized wave, like a school of fish, eventually all agreeing on the same direction.
The "Low-Rank" Collapse: In the final layers, the AI stops doing complex, high-dimensional math. It collapses its thinking into a very simple, low-dimensional space. It's like a complex orchestra suddenly playing just one simple note to finish the song.

Why Should We Care?

Simplicity: These massive, scary AI models aren't as complicated as they look. They are actually running on a very simple, repetitive loop. This makes them easier to understand and study.
Efficiency: If we know the AI only needs 3 repeating blocks, maybe we can build smaller, faster, and cheaper AI models that do the exact same job without needing thousands of unique layers.
Safety: If we can see the "flow" of the AI's thoughts as a simple, predictable pattern, it's easier to check if the AI is behaving safely or if it's going off the rails.

The Bottom Line

The paper argues that Vision Transformers are not a chaotic mess of 100 different layers. They are elegant, efficient machines that discover a simple, repeating rhythm to solve problems. By finding this rhythm, we can build better AI and finally understand how it "thinks."

In one sentence: The authors found that giant AI image models are actually just small, repeating loops in disguise, and by understanding this loop, we can make them simpler, faster, and easier to trust.

1. Problem Statement

Vision Transformers (ViTs) have become the standard backbone for visual tasks, yet their internal computational mechanisms remain largely opaque. While residual connections suggest a link to dynamical systems, there is no established framework to interpret ViT depth as a structured flow.

The Gap: Previous observations show that layer-to-layer representational similarity matrices exhibit "block-diagonal" structures (contiguous layers have high similarity). However, representational similarity does not guarantee functional equivalence. Layers might look similar but compute differently, or compute similarly but look different due to linear transformations.
The Core Question: Does the observed block structure in trained ViTs reflect a genuine functional recurrence, where a small set of distinct computational blocks are reused across many layers to perform the same operations?

2. Methodology

The authors propose and validate the Block-Recurrent Hypothesis (BRH) through a constructive approach involving three main stages:

A. The Block-Recurrent Hypothesis (BRH)

The hypothesis posits that a trained ViT with $L$ layers can be accurately rewritten using only $k \ll L$ distinct, parameter-tied blocks ( $B_1, \dots, B_k$ ) applied recurrently.

Formal Definition: The original function $f_\ell$ (up to layer $\ell$ ) can be approximated by a composition of $k$ blocks, where each block $B_j$ is applied $n_j$ times ( $\sum n_j = \ell$ ), such that the error $\epsilon$ between the original trajectory and the recurrent approximation is minimal.
Constraint: The approximation must match the entire internal trajectory (intermediate activations), not just the final output, to rule out trivial solutions.

B. Operationalization: The Raptor Framework

To test BRH, the authors introduce Raptor (Recurrent Approximations to Phase-structured TransfORmers):

Phase Discovery: They use a Max-Cut algorithm on layer-layer similarity matrices to automatically identify contiguous "phases" (boundaries where block boundaries should be).
Architecture: Raptor models consist of $k$ parameter-tied blocks. Each block is responsible for a specific segment of the original ViT's depth.
Training Strategy (Hybrid): Training recurrent models is notoriously unstable due to error accumulation. The authors employ a two-stage training process:
- Stage 1 (Teacher Forcing): Blocks are trained independently to predict the next layer's activation given the ground truth activation from the teacher ViT. This ensures stability.
- Stage 2 (Autoregressive): The blocks are connected, and the model is trained end-to-end using its own previous predictions as input. This forces the model to learn self-consistent, closed-loop dynamics.
Loss Function: The model minimizes the Frobenius norm between the student's intermediate activations and the teacher's activations across all layers.

C. Dynamical Interpretability

Once the recurrent structure is validated, the authors treat the ViT depth as a discrete-time dynamical system to analyze:

Directional Convergence: Analyzing token trajectories on the unit sphere.
Token-Specific Dynamics: Comparing the behavior of cls, register, and patch tokens.
Low-Rank Collapse: Examining the rank of layer-to-layer update matrices.

3. Key Contributions

Empirical Evidence for BRH: Demonstrated across diverse ViTs (including foundation models like DINOv2) that layer-layer similarity matrices exhibit distinct contiguous phases.
Constructive Verification (Raptor): Successfully trained Raptor models that reconstruct the entire internal trajectory of large ViTs.
- Key Result: A Raptor model with only 2 recurrent blocks recovers 96% of DINOv2's ImageNet-1k linear probe accuracy. With 3 blocks, it reaches 98%, matching the performance of the much larger untied ViT-B.
Mechanistic Insights:
- Stochastic Depth: Showed that increasing stochastic depth during training promotes block-recurrent structure (higher layer similarity and better Raptor reconstruction).
- Functional Uniqueness: Swapping layers within a block preserves accuracy, but swapping layers between blocks causes model collapse, proving functional distinctness of phases.
Dynamical Interpretability Framework:
- Angular Attractors: Token directions converge toward class-dependent angular basins with self-correcting trajectories.
- Token Specialization: cls tokens execute sharp reorientations late in the network, while patch tokens exhibit strong coherence (mean-field effect) and converge rapidly.
- Low-Rank Dynamics: Late-layer updates collapse to low-rank subspaces, indicating convergence to low-dimensional attractors.

4. Results

Performance on Foundation Models:
- ImageNet-1k: Raptor ( $k=3$ ) achieves 83.0% Top-1 accuracy (98% of DINOv2 ViT-B).
- Dense Prediction: Raptor performs competitively on semantic segmentation (ADE20k) and depth estimation (NYUv2), confirming the hypothesis holds for tasks beyond classification.
Ablation Studies:
- Training with only Teacher Forcing (Stage 1) leads to collapse (~3% accuracy), proving that autoregressive training (Stage 2) is essential for functional recurrence.
- Removing skip connections in untrained networks slightly reduces reconstruction, suggesting the residual architecture aids but does not solely create the recurrence.
Complexity Analysis:
- The authors argue that BRH implies low Levin Complexity. While the nominal architecture has $L$ distinct parameters, the effective algorithmic description length is much shorter ( $k$ blocks) without increasing computational cost (runtime), as the same block is iterated.

5. Significance

Principled Understanding: This work moves beyond empirical optimization to provide a mechanistic account of ViT depth, suggesting that successful vision models discover a "low-complexity normative solution" based on recurrence.
Interpretability: By reducing ViTs to a small set of recurrent blocks, the paper opens a new avenue for Dynamical Interpretability. Researchers can now study ViTs using tools from dynamical systems theory (attractors, stability, rank collapse) rather than treating them as opaque black boxes.
Efficiency & Compression: The findings suggest that current foundation models are "over-parameterized" in terms of distinct weights but "under-parameterized" in terms of unique computational primitives. This implies potential for more efficient model designs that explicitly leverage recurrence without sacrificing performance.
Safety: Understanding the internal flow and stability (self-correction) of these models is crucial for safety-critical deployments, offering a path to verify and diagnose model behavior.

In summary, the paper establishes that Vision Transformers are effectively recurrent systems in disguise, where depth is a flow of a few reusable computational phases. This discovery bridges the gap between static architecture and dynamic systems theory, offering a powerful new lens for analyzing and improving deep learning models.