Technical Summary: Vertical Fusion (VFusion) for Robust ViT Classification

1. Problem Statement

Vision Transformers (ViTs) have achieved state-of-the-art performance in computer vision, yet they are predominantly deployed as black-box feature extractors where only the final layer's representation is utilized for downstream tasks. This convention relies on the implicit assumption that the last layer optimally condenses all task-relevant information. However, recent observations suggest that intermediate layers often encode complementary information, particularly regarding robustness to distribution shifts and sample difficulty.

Current ensemble strategies (horizontal ensembling) improve performance by combining predictions from multiple independent models but incur high computational costs and ignore the "vertical" signal available within a single model's internal hierarchy. Furthermore, while intermediate layers in Convolutional Neural Networks (CNNs) have shown utility in correcting errors, their application in ViTs is less understood due to the more uniform representational profile and high redundancy inherent in ViT architectures. The central question addressed is whether intermediate ViT layers contain recoverable signals capable of correcting last-layer failures and, if so, how to effectively aggregate these signals without relying on predictive diversity (disagreement) between layers.

2. Methodology

2.1. Recoverability Analysis

The authors first quantify the recoverability of intermediate layers, defined as the capacity of intermediate probes to correctly classify samples misclassified by the last-layer probe. Across 16 diverse datasets, they found that intermediate probes recover 18% to 76% of last-layer errors.

Crucially, the analysis reveals that this gain is not driven by predictive diversity (i.e., layers making different types of errors). Instead, it stems from a redundancy-correctness correspondence: internal layers act as stable, slightly perturbed probes of a shared discriminative signal. High recoverability correlates with high "correction entropy" (distributed correction capacity across depth) but negatively with geometric disagreement (cosine distance between logits of misclassified samples). This implies that layers are correlated but non-identical, offering a redundant yet informative trace of the correct decision boundary.

2.2. VFusion Architecture

To exploit this redundancy, the authors propose VFusion, a vertical aggregation strategy that synthesizes features across the internal ViT hierarchy into a robust global token. The framework consists of three components:

1. Layer Selection: A subset of layers $S$ is selected from the frozen backbone. Strategies include holistic (all layers), last-half, and strided. The authors recommend the holistic strategy to preserve the complete information trace.
2. Layer Fusion: Selected layer representations (specifically the [CLS] tokens) are concatenated into a vector $v$. A learnable fusion encoder $E$ (a multi-layer perceptron) maps this high-dimensional vector into a low-dimensional latent space $z$. This encoder acts as a noise-suppression filter, discarding task-irrelevant residuals ($r^{(\ell)}$) to isolate the shared task-relevant signal ($s^{(\ell)}$).
3. Supervised Training: The fusion encoder and a lightweight classifier head are trained jointly on the frozen backbone. The training objective combines standard cross-entropy loss ($L_{CE}$) with an orthogonalization regularizer ($L_{orth}$). This regularizer penalizes linear dependencies in the latent dimensions, encouraging the model to learn complementary, decorrelated components of the fused representation.

3. Key Contributions

• Quantification of Recoverability: The paper introduces and measures the concept of recoverability in ViTs, demonstrating that intermediate layers can correct between 18% and 76% of last-layer misclassifications across 16 datasets.
• Mechanism Identification: It empirically establishes that corrective signals arise from a redundancy-correctness correspondence (stable, shared signals) rather than predictive diversity, challenging the assumption that ensemble gains require independent model disagreement.
• VFusion Proposal: A lightweight, principled vertical aggregation method that synthesizes internal hierarchy features into a single global token, outperforming established aggregation baselines (e.g., Super Learner, Neural Logit Controller, Mixture of Experts) in both in-distribution (ID) and out-of-distribution (OOD) settings.
• Generalization and Efficiency: The method generalizes across model sizes (ViT-S, B, L), backbone families (Supervised IN21k, Self-supervised DINOv2, Multimodal CLIP), and pre-training regimes. It achieves these gains with minimal computational overhead compared to horizontal ensembling.
• Empirical Guidelines: The study provides guidelines for layer selection, showing that holistic (contiguous) layer selection yields the best performance, while strided subsampling is less effective.

4. Experimental Results

• In-Distribution Performance: On 16 classification benchmarks, VFusion achieved an average accuracy of 91.4%, outperforming the best individual layer probe by 1.9% and the strongest baseline (Super Learner) by 1.7%. Notably, VFusion closed 45% of the accuracy gap between the best individual layer and the theoretical oracle (where a sample is correct if any layer predicts correctly).
• Out-of-Distribution Robustness: Under covariate and subpopulation shifts (5 datasets), VFusion achieved the highest average accuracy (84.6%), surpassing the best single layer by up to 1.3% on fine-grained tasks like C-100C and IN-200C.
• Ablation Studies:
  • Layer Selection: Holistic selection performed best; restricting to the last half caused a minor drop (0.3%), while strided selection caused a larger drop (0.9%).
  • Backbone Generalization: Gains were consistent across ViT-S, B, and L scales and different pre-training paradigms. The largest improvements were observed with self-supervised (DINOv2) and multimodal (CLIP) backbones.
  • Horizontal vs. Vertical: While horizontal fusion across different backbones (HFusion) yielded the absolute best results, VFusion (vertical fusion within a single backbone) remained competitive, outperforming logit-level aggregation methods like Super Learner by 0.5% on average.

5. Significance and Claims

The paper claims that latent redundancy in ViTs is a beneficial asset rather than a limitation. By treating the internal hierarchy as a series of stable, redundant probes, VFusion effectively condenses this information to form a global token that often exceeds the predictive capacity of any single layer.

The authors emphasize that VFusion offers a robust and efficient alternative to horizontal ensemble methods. Unlike horizontal ensembling, which requires access to multiple diverse pre-trained models (often unavailable in specialized domains like medical imaging) and incurs high computational costs, VFusion operates within a single frozen backbone. It provides a principled way to maximize the utility of off-the-shelf pre-trained models without requiring architectural changes or extensive fine-tuning of the backbone.

The work concludes that the "vertical signal" within a single ViT is substantial and that aggregating these representations via feature-level compression is a superior strategy to simple layer selection or logit averaging, particularly for fine-grained and degraded image tasks.