Technical Summary: Linear-Time Global Visual Modeling without Explicit Attention

Problem Statement

The prevailing paradigm in computer vision attributes the global sequence modeling capabilities of Transformers to the explicit computation of attention weights. This process involves calculating pairwise token similarities to form an $N \times N$ attention matrix, which is then used to recombine value representations. While effective, this mechanism inherently incurs quadratic computational complexity ($O(N^2)$) with respect to sequence length, creating a bottleneck for high-resolution inputs and long sequences. Existing efficiency improvements largely focus on approximating or sparsifying this explicit attention matrix, remaining bound to the paradigm of explicit feature re-weighting.

Methodology

Theoretical Reframing: Attention as a Dynamic MLP

The authors propose a novel perspective: attention can be mathematically reframed not as an explicit weighted aggregation, but as a Multi-Layer Perceptron (MLP) with dynamically predicted parameters.

• Mechanism: In standard attention, the output for a query $q_i$ is $o_i = \text{Softmax}(\frac{q_i K^\top}{\sqrt{d}})V$. The authors interpret $K^\top$ and $V$ not as static weights, but as dynamic parameters generated from the input $X$ (i.e., $K=XW_K, V=XW_V$).
• Implication: Global modeling emerges implicitly. The input $X$ is compressed into a set of dynamic parameters $W = G(X)$, which act as a compact representation of the global context. Forwarding the input through this dynamically parameterized network integrates long-range dependencies without ever computing or applying an explicit $N \times N$ attention matrix.
• Complexity Insight: The quadratic complexity of standard attention arises because the effective width of this "dynamic MLP" scales with the number of tokens $N$ (since $K^\top \in \mathbb{R}^{d \times N}$). By decoupling parameter generation from sequence length, one can achieve global modeling with linear complexity.

WeightFormer Architecture

To validate this hypothesis, the authors introduce WeightFormer, a vision architecture that replaces explicit attention with dynamic parameterization while maintaining linear complexity ($O(N)$).

• Sparse Distribution: Instead of applying dynamic layers to every block (which incurs high overhead), WeightFormer inserts dynamic blocks sparsely (specifically, every third block), keeping the rest static to balance performance and efficiency.
• Dynamic Linear Layers: The first linear layer in the MLP is replaced with a dynamic layer using Bilateral Activation. This strategy factorizes weight prediction into two non-linear branches acting on the input $X$ and its transpose $X^\top$:
  $$\Delta W(X) = W_1 \sigma(W_2 X^\top) \sigma(X W_3) W_4$$
  This allows the model to capture higher-order feature interactions and global context without explicit token-to-token routing.
• Dynamic Depthwise Convolutions (DWC): To replace the spatial mixing of attention, the authors employ dynamic depthwise convolutions. They utilize Spatially Adaptive Prediction, where the input is downsampled via Adaptive Average Pooling (AAP) to a fixed $K \times K$ resolution (matching the kernel size) before predicting the kernel weights. This preserves spatial structure while ensuring the parameter generation process is independent of the input resolution.

Key Contributions

1. Mathematical Reframing: The paper provides a new theoretical explanation for attention's global modeling power, positing it as a dynamic MLP with input-conditioned parameters rather than explicit token-wise aggregation.
2. Feasibility Demonstration: It demonstrates that explicit attention weights are not essential for global sequence modeling. The authors prove that dynamic parameterization can completely replace attention to achieve Transformer-level performance with linear complexity.
3. WeightFormer Architecture: The introduction of a prototype architecture that systematically explores dynamic parameter strategies (Bilateral Activation for linear layers and Spatially Adaptive Prediction for depthwise convolutions) within a standard CNN-like framework.

Experimental Results

The authors conducted extensive experiments on ImageNet-1K, COCO, ADE20K, and ImageNet class-conditional generation.

• Image Classification (ImageNet-1K): WeightFormer achieves competitive or superior accuracy compared to strong baselines (DeiT, PVT, ConvNeXt) with similar or lower computational costs.
  • WeightFormer-S achieves 81.3% top-1 accuracy, outperforming DeiT-S (79.8%) and ConvNeXt-S (79.7%) while using comparable resources.
  • WeightFormer-T achieves 76.3% accuracy with significantly higher throughput (3515 img/s) compared to DeiT-T (3661 img/s) but with much lower FLOPs at high resolutions.
• Efficiency and Scalability: Due to its linear complexity, WeightFormer scales efficiently to high resolutions. At 1248×1248 resolution, it achieves 7.7× higher throughput and 91% memory reduction compared to DeiT.
• Downstream Tasks:
  • Object Detection (COCO): WeightFormer-T improves box/mask AP over DeiT-T (45.0/38.3 vs. 44.4/38.1) while reducing backbone FLOPs from 106G to 77G.
  • Semantic Segmentation (ADE20K): WeightFormer-S achieves 45.6 mIoU, surpassing DeiT-S (44.0 mIoU) with reduced backbone FLOPs.
  • Image Generation: In class-conditional generation, WeightFormer variants consistently achieve lower FID scores than comparable DiT and DiG models, indicating improved sample quality.
• Ablation Studies: Experiments on the frequency of dynamic blocks reveal that a sparse distribution (one dynamic block every three) offers the best trade-off, as replacing too many static layers leads to underfitting and optimization challenges.

Significance and Claims

The paper claims that dynamic parameterization is a fundamentally more efficient alternative to explicit attention for global sequence modeling. By shifting the focus from explicit weight aggregation to implicit parameter generation, the authors open a new design space for efficient architectures that do not rely on the quadratic bottleneck of Transformers.

The authors maintain a modest tone regarding the scope of their work:

• The evaluation is currently restricted to vision tasks, and generalization to other domains (e.g., NLP) remains an open question.
• The theoretical understanding of the expressivity and inductive biases of dynamic parameterization is not yet complete.
• Optimizing these input-conditioned parameters presents challenges regarding gradient flow and training stability.

Ultimately, the work serves as a proof-of-concept that global modeling can be decoupled from quadratic complexity, motivating further exploration of dynamic parameterization as a viable replacement for attention-based architectures.