Technical Summary: SpiralFovea: Input-Adaptive Foveated Tokenization

1. Problem Statement

Current adaptive-inference techniques for foundation models primarily operate on two established "levers":

1. Architecture/Depth: Early exit, Mixture-of-Experts (MoE) routing, and dynamic depth.
2. Attention/KV-Cache: Sparse attention, deformable attention, and KV-cache compression.

However, these methods act upon a fixed-grid input tokenization. Regardless of image content, a standard Vision Transformer (ViT) divides a $224 \times 224$ image into $N=196$ uniform patches in raster order. This approach violates the rate-distortion principle, which posits that optimal capacity allocation should be proportional to local information density. Uniform tokenization allocates equal computational resources to low-entropy background regions and high-entropy discriminative regions, incurring unnecessary costs at every transformer layer before any adaptation occurs.

While token pruning (a Lever-1 technique) reduces token counts, it is structurally distinct because it first forms the full uniform grid, projects it, and processes it through at least one transformer block before dropping tokens. The cost of processing uninformative tokens is already paid.

The Gap: There is a missing "third lever": Input-Adaptive Tokenization. This lever asks which tokens should exist for a specific image before any backbone parameters are queried, rather than pruning them after formation.

2. Methodology: SpiralFovea

The authors propose SpiralFovea, a parameter-free, input-adaptive tokenizer that generates a variable-length set of tokens based on local visual entropy. The process occurs entirely before the backbone forward pass.

Core Pipeline

1. Entropy Map Generation:

   • The input image $I$ is converted to luminance (BT.601), downsampled to $D \times D$, and quantized into $B$ bins.
   • A sliding window computes empirical bin probabilities to derive a local Shannon entropy map $E(x, y)$.
   • This computation is $O(HW)$, parameter-free, and costs $<2\%$ of a single transformer block.

2. Hotspot Anchor Selection:

   • The image height is divided into $S$ strips.
   • Within each strip, the location of maximum entropy is selected as an anchor ($c_s$), ensuring horizontal diversity.
   • Anchors are normalized and deduplicated based on a radius threshold ($\tau_{dedup}$).

3. Multi-Scale Spiral Ring Extraction:

   • Around each retained anchor, patches are placed in $K=4$ concentric rings.
   • Ring 0: A single foveal patch.
   • Rings $k > 0$: Contain $n_k$ patches arranged in a spiral pattern with increasing radius and patch size.
   • The schedule is designed to tile space without gaps. For a $224 \times 224$ image with 4 active strips, the theoretical maximum is 116 patches, but an out-of-bounds filter typically reduces the final count to $\le 78$ patches (a $\sim 60\%$ reduction from the standard 196).

4. Token Acquisition and Embedding:

   • Patches are sampled via bilinear grid sampling.
   • Embedding: Patches are projected using the backbone's frozen patch_embed.proj.
   • Positional Encoding (PE): Standard raster sinusoidal PE is replaced with a Polar PE. A learned 2-layer MLP maps polar coordinates $(\hat{c}_s, k, \theta)$ to embeddings. This respects the geometric proximity of patches in the foveated layout (spatially adjacent regardless of ring index) and the rotational symmetry of the spiral.
   • Backbone Processing: The variable-length sequence (plus a CLS token) is passed through the frozen transformer stack.
   • CNN Adaptation: For CNN backbones, patches are encoded by a shared ResNet-18 trunk and fused via Mamba blocks (state-space models) to handle variable sequence lengths efficiently.

3. Key Contributions

1. Conceptual Framework: Positions input-adaptive tokenization as a distinct, complementary "third lever" of resource-adaptive inference, separate from architecture and attention modifications.
2. SpiralFovea Implementation: A parameter-free, $O(HW)$ entropy-guided tokenizer with provable anchor coverage and multi-scale spiral extraction. Selection completes before any backbone parameter is evaluated.
3. Theoretical Motivation: Justifies the design via rate-distortion theory (allocating capacity to information density) and biological vision (foveal centralis and peripheral saliency).
4. Empirical Performance: Demonstrates strict Pareto improvements across four fine-grained benchmarks and four backbone families.
5. Deployment Diagnostic: Identifies a boundary condition where the method is most effective: it yields the highest gains on backbones with weak or task-agnostic whole-image positional priors (e.g., self-supervised models like DINO), while gains diminish or reverse on models with strong, fixed raster priors (e.g., supervised ViT-S/16).

4. Experimental Results

The method was evaluated on four benchmarks: WikiArt (GAN-Genre, Style), Oxford Flowers-102, PatchCamelyon, and CUB-200-2011 (Genus grouping).

• Accuracy vs. Efficiency: SpiralFovea achieves a strict-dominance Pareto improvement.

  • Token Reduction: Reduces input tokens from 196 to $\le 78$ ($\sim 60\%$ reduction).
  • Accuracy Gain: Yields +1.7 to +2.1 percentage points in top-1 accuracy over matched static baselines.
  • Compute Reduction: Reduces self-attention FLOPs by 84% at every transformer layer (not just a single stage), as the attention matrix size scales quadratically with token count ($78^2$ vs $196^2$).
  • Throughput: Increases throughput by 18–29% on Tesla T4 GPUs.

• Ablation Studies:

  • Anchor Placement: Entropy-guided anchors are critical; random hotspots result in a $-1.5$ pp drop.
  • Token Selection: Selecting which patches matters more than the count; a random 78-patch subset performs worse than the full 196-patch baseline.
  • Design Choices: Multi-scale rings (+1.1 pp), spiral layout (+0.9 pp), and Polar PE (+0.5 pp) all contribute significantly.

• Boundary Analysis (CUB-Genus):

  • The gain is inversely proportional to the strength of the backbone's pre-training positional prior.
  • Highest Gain: Self-supervised DINO-ViT-S (+0.93 pp) and ResNet-18 (+0.9 pp).
  • Lowest/Negative Gain: Supervised ViT-S/16 (-0.28 pp). The authors attribute this to the model's pre-training having "committed" to a uniform raster grid, making the substitution of sparse tokens detrimental, combined with performance saturation.

5. Significance and Claims

The paper claims that SpiralFovea represents a fundamental shift in how foundation models process visual input. By treating the input token set as a primary efficiency lever, it avoids the "waste" of processing uninformative regions entirely.

• Composability: The method is orthogonal to Lever-1 (architecture) and Lever-2 (attention) techniques. The authors argue that combining these levers creates a multiplicative effect on savings (e.g., a shorter sequence for early exit or MoE routing).
• Target Regime: The work isolates self-supervised foundation models as the regime where input-adaptive tokenization is most valuable, as these models lack the rigid positional priors of supervised ImageNet-trained models.
• Practical Impact: Reducing tokens fed to a frozen backbone yields proportional savings in GPU memory and energy, lowering the carbon cost of inference. The method is particularly suited for fine-grained recognition and medical imaging (PatchCamelyon) where discriminative signals are spatially concentrated.

The authors conclude that the "third lever" is largely unexplored and invite the community to investigate the compositional Pareto frontier of combining input adaptation with other adaptive inference techniques, as well as its applicability to non-image modalities.