⚡ electrical engineering

Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

This paper proposes two Discrete Cosine Transform (DCT)-based methods for Vision Transformers: a structure-preserving initialization strategy for self-attention projections that improves classification accuracy, and a frequency-domain compression technique that reduces computational overhead by truncating high-frequency noise without sacrificing performance.

Original authors: Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Ahmet Enis Cetin, Ulas Bagci

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Ahmet Enis Cetin, Ulas Bagci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Vision Transformer (ViT) as a super-smart student trying to learn how to recognize objects in a picture. To do this, the student breaks the image into tiny puzzle pieces (patches) and looks at how they relate to one another. The part of the brain that does this "looking and connecting" is called Self-Attention.

The paper by Hongyi Pan and colleagues suggests that this student is currently starting their studies with a few bad habits: they are guessing randomly at the beginning, and they are trying to remember every single tiny detail, even the static and noise. The authors propose two simple fixes using a mathematical tool called the Discrete Cosine Transform (DCT)—the same math used to compress JPEG images and MP3s.

Here is how their two new methods work, explained with everyday analogies:

1. The "Smart Start" (DCT-Based Initialization)

The Problem:
Usually, when a Vision Transformer starts training, it assigns random numbers to its "brain cells" (weights) to figure out what to look for. It's like giving a new student a stack of blank flashcards and telling them to guess what's on the other side. They have to start from zero, which takes a long time and can be unstable.

The Solution:
The authors say, "Let's give the student a head start." Instead of random numbers, they initialize the brain cells using a DCT matrix.

The Analogy: Imagine the student's brain is a radio tuner. Random initialization is like turning the dial to static noise. The DCT initialization is like tuning the radio to a specific, clear station right away.
How it works: The DCT matrix is made of specific patterns (like different musical notes or colors) that cover the entire spectrum of possibilities. By starting with these patterns, the model doesn't have to "guess" what features exist; it starts with a structured, organized view of the world.
The Result: The model learns faster and gets better at recognizing things (like cats or cars) because it started with a "clean" and organized foundation rather than random noise.

2. The "Noise Filter" (DCT-Based Compression)

The Problem:
When the model looks at an image, it tries to process every single detail. However, in the world of signals (like images), the most important information is usually in the "low frequencies" (the big shapes and main colors), while the "high frequencies" are often just tiny, jagged details or noise (like grain in a photo).

The Analogy: Imagine you are trying to describe a landscape to a friend. You tell them about the mountains, the river, and the trees (the important stuff). But then you also spend 10 minutes describing every single blade of grass and a speck of dust on a rock (the noise). It's a waste of time and energy.

The Solution:
The authors propose a way to "trim the fat" before the model does its heavy thinking.

The Analogy: They use the DCT to translate the image into a "frequency report." Then, they simply cut off the top 25% to 75% of the report, throwing away the high-frequency noise.
How it works: The model only keeps the "low-frequency" coefficients (the important shapes) and ignores the rest. This makes the data much smaller.
The Result: The model becomes much lighter and faster (requiring less computing power and memory) because it isn't wasting energy on the noise. Surprisingly, the paper found that even with less data, the model's accuracy stayed the same or even got slightly better because it was focusing on the right things.

The Bottom Line

The paper claims that by using these two DCT tricks:

Initialization: The model starts with a better "map" of the world, leading to higher accuracy on tests like identifying objects in CIFAR-10 and ImageNet.
Compression: The model can ignore the visual "static," making it run faster and use less memory without losing its ability to see clearly.

The authors tested this on standard image recognition tasks and found that their "DCT-Transformer" models were more efficient and often more accurate than the standard versions, proving that sometimes, a little bit of mathematical structure goes a long way.

1. Problem Statement

Vision Transformers (ViTs) have revolutionized computer vision by applying self-attention mechanisms to image patches. However, two critical challenges remain:

Initialization Sensitivity: The standard approach initializes query ( $W_Q$ ), key ( $W_K$ ), and value ( $W_V$ ) projection matrices with random numbers (e.g., Xavier or He initialization). This treats weights as "white noise," covering the spectrum uniformly but lacking structural prior knowledge. This can lead to unstable training dynamics, especially on small datasets, and requires significant computational resources to converge.
Computational Efficiency: Self-attention mechanisms are computationally expensive ( $O(N^2)$ complexity regarding sequence length and high parameter counts). Existing compression methods often sacrifice accuracy or fail to exploit the specific correlation structures inherent in visual data and attention matrices.

2. Methodology

The authors propose two complementary methods leveraging the Discrete Cosine Transform (DCT) to address these issues:

A. DCT-Based Initialization

Instead of random initialization, the authors propose initializing one of the projection weight matrices ( $W_Q$ , $W_K$ , or $W_V$ ) using the orthogonal Type-II DCT matrix.

Mechanism: The DCT matrix serves as a structured feature extractor. Unlike random weights which are uncorrelated noise, DCT basis vectors cover the entire frequency spectrum systematically.
Rationale:
- Spectrum Coverage: Random initialization starts all weight vectors with the same spectral distribution. DCT initialization ensures each weight vector covers a different bandwidth (from low to high frequency), providing a more robust starting point for learning diverse features.
- Decorrelation: The DCT basis vectors approximate the Karhunen-Loève Transform (KLT) for highly correlated signals (like natural images), effectively decorrelating the input features at the start of training.
Implementation: Only one matrix is initialized with DCT to avoid weight symmetry issues (where identical initialization of multiple matrices leads to poor learning). The others remain randomly initialized.

B. DCT-Based Compressed Attention

This method exploits the frequency domain to reduce dimensionality and computational cost.

Mechanism:
1. Transformation: Input patches are transformed into the frequency domain using DCT along the channel axis.
2. Truncation: High-frequency components (which typically correspond to noise and less critical details) are truncated. Only the top $\tau$ fraction of coefficients (e.g., 25%, 50%, 75%) are retained.
3. Projection: The Query, Key, and Value projections are computed on this reduced-dimension tensor ( $\tau C$ instead of $C$ ).
4. Reconstruction: The attention output is padded back to the original dimension and transformed back via Inverse DCT (IDCT).
Optimization: The IDCT and the final linear output layer ( $L_O$ ) are fused into a single linear layer to eliminate redundant operations, as there is no non-linearity between them.
Efficiency: This reduces the computational complexity of the linear layers from $O(C^3)$ to $O(\tau^2 C^3)$ and reduces the Softmax matrix size significantly.

3. Key Contributions

Novel Initialization Strategy: Introduced the first DCT-based initialization for ViT attention weights, demonstrating that structured, frequency-domain priors outperform random initialization.
Efficient Compression: Proposed a DCT-based compression technique that reduces parameters and FLOPs by discarding high-frequency noise while preserving essential low-frequency semantic information.
Theoretical Justification: Provided a mathematical link between DCT and the decorrelation of highly correlated image patches (approximating KLT), explaining why DCT is superior to random noise or simple pooling for attention compression.
Comprehensive Evaluation: Validated the methods across multiple architectures (Swin-T, Swin-S, ViT-B) and tasks (Classification, Detection, Segmentation).

4. Experimental Results

The methods were evaluated on CIFAR-10, ImageNet-1K, and COCO 2017 benchmarks using Swin Transformers as the backbone.

Image Classification (CIFAR-10 & ImageNet-1K):
- Initialization: Initializing $W_K$ (Keys) with DCT consistently improved Top-1 accuracy. For Swin-T on ImageNet-1K, accuracy increased from 81.47% to 81.73%.
- Compression: Retaining 75% of DCT coefficients ( $\tau=0.75$ ) reduced parameters by ~13% (Swin-T) and ~10% (Swin-S) while maintaining or slightly improving accuracy compared to the baseline.
- Comparison: The proposed method outperformed DCFormer (a previous DCT-based method) by retaining low-frequency components accurately rather than averaging them, thus preserving discriminative features.
Object Detection & Segmentation (COCO 2017):
- Using DCT initialization ( $W_K$ ) and 75% compression, the models achieved higher Average Precision (AP) scores than vanilla Swin Transformers while reducing parameter counts by 3M (Swin-T) and 5M (Swin-S).
Ablation Studies:
- Single vs. Multiple Initialization: Initializing multiple matrices (e.g., both $W_Q$ and $W_K$ ) with DCT degraded performance due to symmetry breaking issues; initializing a single matrix yielded the best results.
- Necessity of DCT: Removing the DCT transform from the compression pipeline (leaving only spatial pruning) caused significant accuracy drops, confirming that frequency-domain decorrelation is essential for the method's success.

5. Significance

This paper demonstrates that frequency-domain analysis is a powerful tool for optimizing Vision Transformers.

Training Stability: It offers a principled, non-random way to initialize deep attention networks, potentially reducing training time and improving convergence on data-scarce tasks.
Efficiency without Sacrifice: It proves that high-frequency noise can be safely discarded from attention mechanisms, allowing for lighter models with fewer parameters and lower FLOPs without compromising (and sometimes enhancing) performance.
Generalizability: The approach is architecture-agnostic (tested on Swin and ViT) and task-agnostic (works for classification, detection, and segmentation), making it a versatile addition to the ViT toolkit.

The code for the implementation is publicly available, facilitating further research into frequency-based attention mechanisms.