💬 NLP

SpecPL: Disentangling Spectral Granularity for Prompt Learning

SpecPL introduces a novel prompt-learning framework that decouples spectral granularity via counterfactual granularity supervision to bridge modality asymmetry in vision-language models, achieving state-of-the-art performance on 11 benchmarks by revitalizing text-oriented baseline models with visual guidance.

Original authors: Jingtao Zhou, Xirui Kang, Feiyang Huang, Lai-Man Po

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingtao Zhou, Xirui Kang, Feiyang Huang, Lai-Man Po

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot (a Vision-Language Model) to recognize different bird species. You show it a few pictures of a "sparrow" and a "hawk".

The Problem: The Robot's "One-Size-Fits-All" Glasses
Current methods for teaching these robots suffer from a strange imbalance. They spend all their time polishing the robot's words (the text descriptions) but leave its eyes (the visual encoder) frozen in a "one-size-fits-all" mode.

Think of it this way: The robot wears thick, foggy glasses that only allow it to see the general shape of a bird (e.g., "it has wings and a beak"). It completely overlooks the fine details (e.g., the specific feather pattern, the texture of the beak, the lighting). Because it cannot recognize these tiny details, it gets confused when the bird looks slightly different or when it tries to distinguish between two very similar species. It is like trying to identify a specific person in a crowd based solely on their silhouette.

The Solution: SpecPL (The "Spectral" Glasses)
The authors introduce a new method called SpecPL. Instead of just polishing the words, they give the robot a pair of "spectral glasses" capable of decomposing an image into two distinct layers, similar to how a sound engineer separates a song into bass (low tones) and treble (high tones).

Here is how SpecPL works, using simple analogies:

1. The "Bass vs. Treble" Separation (Disentangling Granularity)

The robot uses a frozen "teacher" (a pre-trained AI called a VAE) to view an image and split it into two parts:

The "Bass" (Low Frequency/Base): This is the stable, slowly moving part of the image. It captures the big picture: the bird's shape, its general arrangement, and its category. This is the "invariant" part that remains the same whether the bird is in the sun or in the shade.
The "Treble" (High Frequency/Detail): This is the fast-moving, jagged part. It captures the finely graded details: the texture of the feathers, the specific color patterns, and the tiny quirks that make this specific bird unique.

Why this matters: Previous methods tried to learn everything at once and were confused by the noise. SpecPL separates the "stable shape" from the "fickle details."

2. The "Anchor" (Visual Semantic Bank)

To ensure the robot does not get lost in the details, SpecPL creates a Visual Semantic Bank.

Analogy: Imagine the "Bass" layer is a stable anchor thrown into the sea. No matter how hard the waves (the changing details) crash around, the anchor keeps the boat (the text description) steady.
The robot uses the "Bass" part of the image to anchor its text descriptions to a universal, stable truth. This prevents the robot from merely memorizing incorrect things (overfitting) just because it saw a specific bird under specific lighting.

3. The "What-If?" Game (Counterfactual Supervision)

This is the cleverest part. To teach the robot to actually use the "Treble" (the fine details), the researchers play a trick on it.

The Game: They take the "Treble" (feather texture) of a hawk and swap it with the "Bass" (body shape) of a sparrow.
The Lesson: They force the robot to guess: "If I see a sparrow-shaped body with hawk feathers, is it a sparrow or a hawk?"
The Result: The robot realizes that the "Treble" details are so important that they can change the identity of the object. This forces the robot to pay attention to the fine details instead of ignoring them as noise.

The Result
By separating the "Bass" (stable shape) from the "Treble" (fine details) and playing this "What-If?" game, SpecPL corrects the robot's vision.

It is Plug-and-Play: You can connect this method to existing robot brains (like CoOp or MaPLe) without having to rebuild them from scratch.
It works better: In 11 different tests (ranging from flower recognition to aircraft identification), SpecPL helped robots significantly improve their ability to distinguish between similar things.
The Record: It achieved a new peak of 81.51% in a specific test type called "Harmonic Mean Accuracy," which balances how well the robot knows what it has learned versus how well it can guess new things.

Summary:
The work claims that by teaching the robot to separate the "big picture" from the "tiny details" and then forcing it to learn from those details through a swapping game, we can make it much better at recognizing subtle differences without needing to retrain the entire system from the ground up. It bridges the gap between a robot that only sees shapes and one that sees the whole picture.

Technical Summary: SpecPL: Disentangling Spectral Granularity for Prompt Learning

1. Problem Statement

The work identifies a persistent modality asymmetry in existing prompt-learning methods for Vision-Language Models (VLMs) such as CLIP. While these methods effectively optimize text tokens (prompts) to adapt to downstream tasks, they rely on frozen visual encoders that treat images as holistic, static features. This approach neglects the spectral granularity essential for fine-grained discrimination.

Consequently, this asymmetry creates a tension between stability and generalization:

Overfitting: Prompts often adapt too strongly to scarce training distributions.
Loss of Invariance: Models fail to leverage the transferable invariances encoded by the frozen VLM.
Text-Centric Bias: Attempts to mitigate this by importing external knowledge (e.g., Large Language Models) introduce noise and computational costs, offering no principled mechanism for the visual side to provide discriminative granularity.

The authors argue that the missing inductive structure is spectral but must be reinterpreted within deep latent manifolds rather than in pixel space. Specifically, low-frequency components capture stable, cross-category structures (semantic invariances), while high-frequency components encode instance-specific details (textures, lighting, pose) necessary for fine-grained discrimination.

2. Methodology: SpecPL

The authors propose SpecPL (Disentangling Spectral Granularity for Prompt Learning), a framework that reframes prompt learning as spatial-spectral factorization with counterfactual supervision. The method acts as a universal, plug-and-play booster for text-oriented baselines (e.g., CoOp, MaPLe).

Core Components

Spatial-Spectral Proxy (VAE-based Decomposition):
- Instead of pixel-level Fourier transforms, SpecPL utilizes a frozen, pre-trained VAE (e.g., from Stable Diffusion) as a teacher.
- The VAE encoder decomposes visual signals within the latent manifold via a lightweight Spatial-Spectral Proxy into two complementary components:
  - Base Component (Low-Frequency): Captures stable semantic invariances (coarse shape, layout, topology) through local smoothing (moving average pooling).
  - Detail Component (High-Frequency): Encodes granular, instance-specific cues (textures, variations) as a residual ( $z_{detail} = z - z_{base}$ ).
- These components are projected into the CLIP embedding space via lightweight MLP heads ( $Proj_{low}$ and $Proj_{high}$ ).
Visual Semantic Bank (Semantic Anchoring):
- To prevent overfitting and stabilize adaptation, a frozen Visual Semantic Bank is initialized with low-frequency prototypes ( $t_{low}$ ).
- During training, text representations are anchored to these universal low-frequency invariances via soft retrieval mechanisms and compositional refinement. This ensures the model aligns with stable semantic structures rather than noisy training specifics.
Counterfactual Granularity Supervision:
- To force the model to utilize high-frequency details for discrimination, SpecPL introduces a FiLM-like modulation branch (active only during training).
- Factual Supervision: The model is trained to predict class labels using the fused semantic anchor and the high-frequency details of its own instance.
- Counterfactual Supervision: The model is forced to distinguish visual granularity by permuting high-frequency signals across samples. The model must predict the label of the source of the swapped granularity. This prevents the model from relying solely on a "shape bias" and compels it to encode high-frequency evidence as discriminative.
Inference Efficiency:
- During inference, the VAE teacher and the FiLM modulation branch are discarded.
- The system relies exclusively on the Visual Semantic Bank for text refinement and standard CLIP similarity for prediction, incurring no additional inference latency or FLOPs.

3. Main Contributions

The work outlines three primary contributions:

Identification of the Bottleneck: The authors identify the lack of spectral granularity factorization as a key bottleneck behind modality asymmetry in prompt learning.
Mechanism of the Spatial-Spectral Proxy: They introduce a mechanism that effectively disentangles semantic invariance from discriminative details within a frozen VAE manifold, avoiding the semantic gap associated with pixel-level frequency analysis.
Plug-and-Play Framework: They provide a practical framework using counterfactual supervision that revitalizes existing baselines without relying on noisy, LLM-generated priors or external knowledge.

4. Experimental Results

SpecPL was evaluated on 11 benchmarks (including ImageNet, Caltech101, OxfordPets, FGVC-Aircraft, DTD, EuroSAT, etc.) across 16-shot generalization from base to novel classes, cross-dataset transfer, and domain generalization.

Performance: SpecPL achieves a new performance ceiling of 81.51% harmonic mean (HM) accuracy when applied to the MMRL baseline.
Improvements:
- CoOp: +4.86% HM improvement.
- CoCoOp: +2.00% HM improvement.
- MaPLe: +0.80% HM improvement.
- MMRL: +0.31% HM improvement.
Generalization Gap: SpecPL significantly reduces the generalization gap from base to novel classes (e.g., a 31.6% reduction for CoOp), indicating improved transfer to unseen classes.
Fine-Grained Discrimination and Shift Robustness: The method achieves the largest gains on datasets requiring fine-grained discrimination (e.g., FGVC-Aircraft, DTD) and those with strong domain shifts (e.g., EuroSAT), validating the utility of high-frequency granularity.
Efficiency: The method adds negligible inference overhead. Although there are one-time costs during training for building a VAE cache, inference FLOPs remain identical to the baseline.

5. Significance and Claims

The work claims that spectral disentanglement with counterfactual supervision effectively closes the gap in the trade-off between stability and generalization.

Beyond Text-Centric Optimization: The work demonstrates that moving beyond pure text prompt tuning to explicitly model structured visual granularity within deep representations is a promising direction for VLM adaptation.
Principled Visual Guidance: Unlike methods that treat vision holistically or rely on external LLMs, SpecPL offers a controllable, principled mechanism to feed discriminative granularity directly into the visual adaptation process.
Modest Scope: The authors acknowledge that the method is most beneficial for tasks where class identity depends on subtle appearance cues (fine-grained) rather than coarse semantics. They note that for generic object recognition tasks with distinct semantic categories, improvements over strong baselines are marginal.

The code will be released to facilitate reproducibility, and the authors commit to transparency regarding the use of external VAE teachers and the avoidance of new data collection.