Fourier-Attentive Representation Learning: A Fourier-Guided Framework for Few-Shot Generalization in Vision-Language Models

This paper proposes Fourier-Attentive Representation Learning (FARL), a novel framework that enhances few-shot generalization in Vision-Language Models by explicitly disentangling image structure and style via Fourier analysis and a dual cross-attention mechanism to guide robust vision-language alignment.

The Gist

The Big Problem: The "Superficial Student"

Imagine you are teaching a brilliant student (an AI model called a Vision-Language Model, or VLM) to recognize animals. You only have a few photos to show them (this is called "few-shot learning").

The student is smart, but they have a bad habit: they are superficial.

• If you show them a picture of a dog on green grass, the student learns: "Dog = Green Grass."
• If you show them a cat on a red rug, the student learns: "Cat = Red Rug."

When you later show them a dog on a sandy beach, the student gets confused because there is no green grass. They failed to learn what a dog actually looks like (its shape and structure); they only learned the background style (the grass).

In technical terms, the AI is getting "stuck" on the amplitude (colors, textures, lighting) of the image and ignoring the phase (the actual shapes, edges, and geometry).

The Solution: The "Fourier Detective"

The authors propose a new method called FARL (Fourier-Attentive Representation Learning). Think of this as giving the student a special pair of glasses that splits every image into two distinct layers before they look at it.

They use a mathematical trick called the Fourier Transform (which is like a recipe for turning a complex dish back into its individual ingredients). This splits an image into:

1. The "Skeleton" (Phase): This contains the outlines, shapes, and geometry. It's the "what" of the object. (e.g., The shape of a cat's ears).
2. The "Makeup" (Amplitude): This contains the colors, textures, and lighting. It's the "style" of the object. (e.g., The fur texture or the green grass).

How FARL Works: The "Dual-Brain" Strategy

Instead of letting the AI look at the whole messy image at once, FARL forces it to look at the "Skeleton" and the "Makeup" separately, then combine them intelligently.

Here is the step-by-step process using an analogy:

1. The Split (The Kitchen)

Imagine the AI is a chef. Before cooking, it takes a photo of a dish and separates the recipe structure (how the ingredients are arranged) from the plating style (the sauce color and garnish).

• Phase Stream: Looks only at the arrangement (the structure).
• Amplitude Stream: Looks only at the colors and textures (the style).

2. The "Dual-Brain" Attention (The Tasting)

The AI has two special "learning tokens" (think of them as two little detectives).

• Detective A asks the Phase stream: "What is the shape of this object?"
• Detective B asks the Amplitude stream: "What is the texture and color?"

They don't just guess; they actively query these streams to get the best information. This ensures the AI doesn't get distracted by the background grass when trying to identify the dog.

3. The Asymmetric Injection (The Smart Teacher)

This is the most clever part of the paper. The authors realized that you shouldn't treat the "Image Brain" and the "Text Brain" the same way.

• The Text Brain (The Describer): This part needs to be flexible. We inject the "enriched" information (both shape and style) here.

  • Analogy: Imagine the teacher writing a description on a whiteboard. Instead of just writing "A dog," the teacher writes, "A fluffy, white dog with pointy ears." The text description is now customized to the specific image's structure and style. This helps the AI match the text to the image perfectly.

• The Image Brain (The Observer): This part needs to be stable. We do not inject the specific style details here. We only inject generic "learning tokens."

  • Analogy: Imagine the student's eyes. If we tell their eyes, "Look at this specific green grass," they will get confused later. Instead, we tell their eyes, "Just look for shapes and edges, ignore the specific colors for now." This keeps the image recognition robust and prevents it from memorizing the background.

Why This Matters

By separating the "Skeleton" from the "Makeup," FARL solves the "Superficial Student" problem:

• It stops overfitting: The AI stops memorizing that "Dogs always have green backgrounds."
• It learns the truth: The AI learns that "Dogs have four legs and a tail" (the phase/structure).
• It generalizes: When shown a dog on a beach, a desert, or in black and white, the AI still recognizes it because it focused on the shape, not the style.

The Results

The authors tested this on 15 different datasets (like recognizing flowers, cars, and textures).

• The Result: FARL consistently beat other top methods.
• The Proof: When they visualized what the AI was looking at, they saw that the "Phase" detector was focusing strictly on the object's outline, while the "Amplitude" detector was looking at the background. They were working together perfectly, rather than getting confused.

Summary

FARL is like teaching an AI to ignore the "noise" (colors, backgrounds, lighting) and focus on the "signal" (shapes, edges, geometry).

It does this by mathematically splitting images into "structure" and "style," letting the AI learn from both separately, and then using a smart strategy to update its text descriptions with this new knowledge while keeping its image recognition steady. This makes the AI much better at learning new things with very few examples.

Technical Summary

1. Problem Statement

Large-scale Vision-Language Models (VLMs) like CLIP have demonstrated impressive zero-shot and few-shot capabilities. However, existing adaptation methods (e.g., prompt learning, adapters) often suffer from spectral bias in low-data regimes.

• The Core Issue: Neural networks tend to overfit to superficial, domain-specific statistics (texture, color, lighting) rather than robust semantic structures.
• Fourier Perspective: In the frequency domain, these superficial cues are encoded in the amplitude spectrum, while robust geometric structures and shapes are preserved in the phase spectrum.
• Current Limitation: Existing few-shot adaptation methods treat visual features as "holistic" embeddings where amplitude and phase are implicitly entangled. Consequently, models latch onto dominant amplitude statistics of the support set, leading to poor generalization when facing domain shifts or novel classes.

2. Methodology: Fourier-Attentive Representation Learning (FARL)

FARL is a novel framework designed to explicitly disentangle visual representations using Fourier analysis before injecting them into the VLM. The architecture consists of three main stages:

A. Fourier Decomposition & Feature Extraction

The input image $I$ is transformed via the 2D Fast Fourier Transform (FFT) into amplitude ($A$) and phase ($P$) spectra.

• Phase Image ($I_{phase}$): Reconstructed by preserving the original phase spectrum and setting the amplitude to unity. This isolates structural information (edges, shapes, geometry).
• Amplitude Image ($I_{amp}$): Reconstructed by preserving the original amplitude spectrum and setting the phase to zero. This isolates stylistic information (texture, color, lighting).
• These two images are passed through lightweight CNNs to generate feature sets: $F_{phase}$ (structure-focused) and $F_{amp}$ (style-focused).

B. Dual Cross-Attention Mechanism

FARL introduces learnable, modality-agnostic representation tokens ($R$). These tokens act as queries in a dual cross-attention mechanism:

1. Phase Stream: Tokens query $F_{phase}$ to generate structure-aware tokens ($R'_{phase}$).
2. Amplitude Stream: Tokens query $F_{amp}$ to generate style-aware tokens ($R'_{amp}$).
3. Fusion: The two streams are concatenated, processed by a fusion MLP, and combined with the original tokens via a residual connection to produce enriched, disentangled tokens ($R_{fused}$).

C. Asymmetric Injection Strategy

A critical design choice is the asymmetric injection of these tokens into the VLM encoders:

• Text Encoder: The enriched, fused tokens ($R_{fused}$) are injected. This allows the text prompts to dynamically adapt to the specific structural and stylistic properties of the input image (e.g., converting "a photo of a dog" to an instance-specific description).
• Image Encoder: Only the original, generic tokens ($R$) are injected. This acts as a regularization mechanism, preventing the powerful visual backbone from overfitting to the specific amplitude statistics of the few-shot support set, thereby preserving its pre-trained, domain-invariant capabilities.

Loss Function: The model is optimized using a combination of cross-entropy loss for both class and representation features, plus a cosine regularization loss to ensure the class feature path remains close to the original CLIP space.

3. Key Contributions

1. Reframing the Failure Mode: The authors identify and formalize "spectral bias" as the primary cause of few-shot adaptation failure, arguing that holistic adapters overfit to domain-specific amplitude statistics.
2. FARL Framework: The first prompt learning framework to integrate Fourier-based disentanglement directly into the representation learning loop, rather than using it merely for data augmentation.
3. Asymmetric Design: A novel injection strategy that provides rich, instance-specific guidance to the text encoder while enforcing robust regularization on the image encoder.
4. Empirical Validation: Extensive experiments demonstrating that explicit spectral disentanglement leads to superior generalization compared to state-of-the-art (SOTA) methods.

4. Experimental Results

The authors evaluated FARL on 15 datasets (including ImageNet, Caltech101, OxfordPets, Flowers, Food101, etc.) under a 16-shot learning protocol.

• Base-to-Novel Generalization: FARL achieved the best Harmonic Mean (HM) across almost all datasets.
  • Example (ImageNet): FARL achieved 74.53% HM, outperforming the previous SOTA (MMRL) at 74.37%.
  • Example (EuroSAT): Significant improvement, with FARL reaching 88.66% HM compared to MMRL's 81.67%.
• Cross-Dataset Transfer: When trained on ImageNet and tested on other datasets zero-shot, FARL maintained the highest average accuracy, proving the transferability of its disentangled representations.
• Domain Generalization: On out-of-distribution variants (ImageNet-V2, Sketch, A, R), FARL demonstrated superior robustness, attributed to its reliance on phase-derived structural features.
• Ablation Studies:
  • Removing the phase stream caused a massive drop in novel class accuracy, confirming that structure is the primary driver of generalization.
  • Removing the amplitude stream caused a slight drop, indicating it provides necessary context for base classes.
  • Using raw RGB images in a dual-stream architecture (without Fourier decomposition) performed worse than FARL, proving that the explicit frequency decomposition is the key factor, not just the dual-stream architecture.

5. Significance and Conclusion

This paper makes a significant contribution by bridging signal processing principles (Fourier analysis) with deep learning adaptation.

• Theoretical Insight: It moves beyond treating VLMs as black boxes, offering a mechanism to control what the model learns (structure vs. style) via frequency domain manipulation.
• Practical Impact: FARL provides a robust solution for few-shot learning where data is scarce and domain shifts are common. By forcing the model to prioritize geometric structure (phase) over texture (amplitude), it mitigates the "shortcut learning" problem.
• Future Direction: The work suggests that integrating fundamental signal processing directly into the representation learning loop (rather than just as data augmentation) is a promising direction for improving the generalization of foundation models.