🤖 AI

Text-Guided Multi-Scale Frequency Representation Adaptation

The paper proposes FreqAdapter, a parameter-efficient fine-tuning method that integrates textual guidance to perform multi-scale signal adaptation in the frequency domain, thereby overcoming the redundancy and fixed-receptive-field limitations of existing approaches to significantly improve performance and efficiency in multimodal models.

Original authors: Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Tuning a Giant Radio

Imagine you have a massive, incredibly powerful radio (a pre-trained AI model like CLIP or LLaVA) that has listened to the entire world's music and news. It knows almost everything. But now, you want to teach it a specific new dialect or a niche genre of music without buying a whole new radio.

Current methods for teaching this radio (called "fine-tuning") are like trying to adjust the sound by twisting every single knob on the front panel (the image pixels) at once.

The Issue: There are too many knobs! Most of them are just making the same noise (redundancy). Also, current methods usually try to adjust the whole song at once, ignoring that a song has different layers: the deep bass (global structure) and the high-pitched cymbals (fine details). They treat the whole song as one flat block.

The Solution: FreqAdapter (The "Sound Engineer" Approach)

The authors propose a new tool called FreqAdapter. Instead of fiddling with the raw knobs (pixels), they turn the radio signal into a sheet music score (the frequency domain).

Here is how it works, step-by-step:

1. Changing the View: From Photo to Sheet Music

Imagine you have a photo of a cat.

Old Way (Spatial Domain): You look at the photo and try to change the color of the fur pixel by pixel. It's messy and repetitive.
FreqAdapter Way (Frequency Domain): You translate that photo into a musical score.
- The low notes on the score represent the big shapes (the outline of the cat, the background).
- The high notes represent the tiny details (the whiskers, the texture of the fur).
- Why do this? The paper found that the "important meaning" of the image is packed tightly into the low notes. It's like realizing you only need to adjust the bass and drums to change the vibe of a song, rather than touching every single instrument.

2. The Text Guide: The Conductor

The AI isn't just looking at the image; it's also reading a text description (like "a cat sitting on a mat").

FreqAdapter acts like a conductor holding a baton.
The conductor reads the text ("Cat on a mat") and then points to specific parts of the sheet music (the frequency score) to tell the AI: "Hey, boost the low notes that match 'cat' and dampen the high notes that don't match 'mat'."
This ensures the AI focuses its attention exactly where the text says it should.

3. Multi-Scale Strategy: The Zoom Lens

The paper introduces a "Multi-Scale" strategy. Imagine looking at a map.

Zoomed out: You see the whole country (global structure).
Zoomed in: You see the streets and houses (local details).
FreqAdapter looks at the sheet music at three different zoom levels simultaneously. It adjusts the "bass" (global shape) and the "cymbals" (tiny details) separately, then blends them back together. This prevents the AI from getting confused about whether it's looking at a whole building or just a brick.

4. Putting it Back Together

Once the "sheet music" has been tweaked by the text-conductor, FreqAdapter translates it back into a photo (the spatial domain). The result is a photo that the AI understands much better, specifically tailored to the text it was given.

Why is this better? (The Results)

The paper tested this on two famous AI models: CLIP (which matches images to text) and LLaVA (which answers questions about images).

Speed: It's incredibly fast. The model learned the new task in just one round of training (one epoch). It's like learning a new song in a single practice session.
Efficiency: It adds very few new "knobs" (parameters) to the system. It's a lightweight tool, not a heavy upgrade.
Performance:
- Better Memory: When asked to find an image based on text, it was more accurate than previous methods.
- Better Understanding: In a test where the AI had to read a gas price sign and do math, FreqAdapter got the right answer, while other methods either failed to read the numbers or couldn't do the math.
- No Overfitting: Old methods often "memorized" the training data and forgot how to handle new data (overfitting). FreqAdapter stayed stable and didn't get confused, likely because it was working with the "clean" musical notes rather than the "noisy" raw pixels.

The Bottom Line

FreqAdapter is a smart, lightweight tool that teaches big AI models to understand images better by:

Turning images into "sound" (frequencies) where the important info is easier to find.
Using text as a conductor to tweak specific parts of that sound.
Looking at the image from different "zoom levels" to catch both big shapes and small details.

It allows these giant models to learn new tasks quickly and accurately without needing to be rebuilt from scratch.

Technical Summary: Text-Guided Multi-Scale Frequency Representation Adaptation (FreqAdapter)

1. Problem Statement

Existing parameter-efficient fine-tuning (PEFT) methods for multimodal foundation models (e.g., CLIP, LLaVA) face two primary limitations:

Spatial Domain Redundancy: Most methods operate directly in the spatial domain, where visual features contain significant information redundancy, potentially leading to overfitting under limited fine-tuning budgets.
Uniform Adaptation: Current approaches typically utilize fixed prompts or adaptation layers that adjust the entire feature set uniformly. They fail to account for the inherent multi-scale characteristics of signals, where different frequency bands correspond to different semantic granularities (e.g., global structure vs. local details).

The authors posit that adjusting image information in the frequency domain can offer a more compact, disentangled representation, facilitating stable optimization and better cross-modal alignment.

2. Methodology: FreqAdapter

The paper proposes FreqAdapter, a framework that performs text-guided, multi-scale fine-tuning of visual embeddings in the frequency domain. The core workflow involves:

2.1 Frequency Domain Transformation

Visual embeddings ( $E_v$ ) and textual embeddings ( $E_t$ ) are first extracted by the CLIP encoder. Both are transformed from the spatial domain to the frequency domain using the Discrete Cosine Transform (DCT), yielding frequency representations $X_v$ and $X_t$ . The authors empirically demonstrate that semantic information is highly concentrated in low-frequency components, allowing for compact adaptation.

2.2 Multi-Scale Adaptation Strategy

To capture diverse receptive fields, FreqAdapter employs a multi-scale strategy. The visual frequency sequence is reshaped and downsampled at $N$ different scales. At each scale $n$ , two lightweight modules operate:

Multi-Scale Global Frequency Adapter (MGFA): A global calibration module that applies a lightweight bottleneck transformation to adjust each frequency channel globally. This ensures stable and consistent adaptation across scales, preventing scale misalignment caused by coupled frequency coefficients.
Multi-Scale Cross-Modal Frequency Adapter (MCFA): A text-guided refinement module. It predicts modulation parameters ( $\gamma, \beta$ ) from the textual embeddings ( $X_t$ ) using a lightweight MLP. These parameters are applied to the visual features via element-wise multiplication and addition ( $C_n = \gamma \odot X_{v,n} + \beta$ ), enabling fine-grained cross-modal alignment.

2.3 Reconstruction

The refined frequency features from all scales are aggregated (averaged) and upsampled to restore the original spatial dimensions. An Inverse Discrete Cosine Transform (IDCT) converts the adapted frequency features back to the spatial domain ( $\tilde{E}_v$ ), which are then fed into the downstream encoder (e.g., for LLaVA) or used for retrieval tasks.

2.4 Training

The adapter is optimized using the CLIP contrastive loss on paired image-text data. The method is plug-and-play, requiring no modification to the backbone model's architecture.

3. Key Contributions

FreqAdapter Framework: The proposal of a novel adapter that adjusts visual information in the frequency domain conditioned on textual information, addressing the redundancy issues of spatial-domain fine-tuning.
Multi-Scale Adaptation Strategy: The design of a strategy that performs adjustments across multiple frequency scales, integrating both global calibration (MGFA) and text-guided refinement (MCFA) to enhance representational capacity.
Empirical Validation: Extensive experiments on multimodal models (CLIP and LLaVA) demonstrating that FreqAdapter significantly improves performance and efficiency with minimal parameter overhead.

4. Experimental Results

The authors evaluated FreqAdapter on image-text retrieval and Visual Question Answering (VQA) tasks.

Image-Text Retrieval (CLIP): On the COCO 2017 and Flickr30K datasets, FreqAdapter achieved state-of-the-art performance among parameter-efficient methods. For instance, on CLIP-L/14 for text-to-image retrieval on COCO, it improved the R@1 score by up to 4.9% compared to baselines like CoOp and MaPLe. It also demonstrated superior zero-shot generalization on Flickr30K.
Visual Question Answering (LLaVA): When integrated into LLaVA (7B and 13B versions), FreqAdapter outperformed baselines (including CLIP-Adapter and prompt-based methods) on the MM-Vet and LLaVA-Bench datasets. Qualitative analysis showed FreqAdapter could accurately identify specific numerical values and perform arithmetic reasoning where other models failed.
Efficiency: FreqAdapter introduces approximately 476k parameters (0.11% of the total model), which is fewer than many existing adapters (e.g., MaPLe). The computational overhead (GFLOPs) remains nearly identical to the baseline, as the DCT/IDCT operations are negligible compared to the backbone.
Convergence: The model converges rapidly, often within one epoch, and exhibits more stable training curves with less overfitting compared to spatial-domain adapters (SpatialAdapter).

5. Significance and Claims

The paper claims that FreqAdapter provides a unique perspective for parameter-efficient fine-tuning by shifting the adaptation space from the redundant spatial domain to the structured frequency domain.

Stability and Generalization: The authors attribute the performance gains to the "disentangled" nature of frequency representations, where low- and high-frequency structures are separated, and noise is reduced. This leads to cleaner parameter updates and a smoother optimization trajectory, resulting in stronger generalization capabilities.
Multi-Scale Integration: By explicitly modeling multi-scale receptive fields in the frequency domain, the method effectively integrates fine-grained and coarse-grained information without sacrificing generalization, a common weakness of traditional spatial adapters.
Modest Limitations: The authors acknowledge that while empirical results are strong, the theoretical depth regarding frequency-domain adaptation requires further exploration. They also note that current studies are limited to CLIP-based settings with relatively small parameter scales, suggesting that future work should investigate scaling these approaches to larger multimodal language models.

In conclusion, the paper argues that frequency-domain adaptation offers a stable inductive bias for cross-modal representation learning, achieving superior fitting performance and faster adaptation speeds compared to spatial-domain alternatives.