💻 computer science

SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

The paper introduces SAMPLe, a sharpness-aware optimizer designed to resolve the performance-generalization dilemma in Vision-Language Model prompt learning by dynamically balancing exploration and exploitation to reduce overfitting and enhance adaptability to unseen data across various frameworks.

Original authors: Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi, Hossein Kashiani, Xiaolong Ma, Fatemeh Afghah

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi, Hossein Kashiani, Xiaolong Ma, Fatemeh Afghah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, pre-trained robot (like CLIP) that has read millions of books and seen millions of pictures. It's already very good at understanding the world. However, you want to teach it a specific new trick, like identifying rare flowers or spotting specific car models.

In the past, people tried to "fine-tune" the whole robot, but that's like trying to rewire a supercomputer just to learn a new card game—it's expensive and the robot often forgets its old skills or gets confused.

Instead, researchers use Prompt Learning. Think of this as giving the robot a specific "cheat sheet" or a set of magic keywords (called prompts) to help it focus on the new task. The robot's brain stays frozen, and we only tweak these few keywords.

The Problem: The "Over-Confident" Student

The paper identifies a major problem with this approach. When we tweak these keywords to get a perfect score on our training examples (the "seen" data), the robot often becomes too confident and too specific.

Imagine a student who memorizes the exact answers to a practice test. They get 100% on the practice test, but if you ask them a slightly different question on the real exam, they fail. In the paper's language, the robot has found a "sharp minimum."

Sharp Minimum: A spot on the map where the score is high, but if you take even one tiny step in any direction, the score crashes. It's like balancing a ball on the very tip of a needle. It's stable only if nothing moves.
Flat Minimum: A spot where the score is high, but the ground is wide and flat. If you take a step, the score stays high. This is like a ball sitting in a wide valley. It's robust and can handle bumps.

The current methods for teaching these prompts tend to put the robot on the "tip of the needle." It works great on the training data but fails miserably when faced with new, unseen data (like a different type of flower or a car in bad weather).

The Solution: SAMPLe (The "Safety-First" Coach)

The authors introduce a new tool called SAMPLe (Sharpness-Aware Minimization Prompt Learning). You can think of SAMPLe as a very smart coach who doesn't just care about the current score; they care about how stable that score is.

Here is how SAMPLe works, using a simple analogy:

1. The "What-If" Test (Exploration vs. Exploitation)
Most coaches just say, "Run faster to get a better time!" (This is called Exploitation). They push the robot to the absolute best spot on the current map.
SAMPLe is different. Before the robot settles on a spot, SAMPLe asks: "Okay, you're at a great spot. But what if you took one small step to the left? Or one step to the right? Would you still be doing well?"

If the answer is "No, I'd fall off a cliff," SAMPLe says, "Okay, this spot is too sharp. Let's move to a flatter area."
If the answer is "Yes, I'm still doing great," SAMPLe says, "Great! This is a safe, flat spot. Let's stay here."

2. The "Two-Step" Dance
The paper explains that SAMPLe does a special dance with the robot's learning steps:

Step A (The Push): It looks at the current data and says, "Go this way to improve your score."
Step B (The Safety Check): It then looks at the entire history of the data to see if that direction is too risky. It calculates a "safety vector" that pushes the robot away from the dangerous, sharp edges of the learning landscape.
The Result: The robot moves in a direction that improves the score but also keeps it in the wide, safe valley.

3. Why It's Better Than Others
The paper compares SAMPLe to other methods (like SAM, F-SAM, and SAGM).

Old Methods: Some are too aggressive and push the robot too hard, making it unstable. Others are too rigid and don't adapt well to the changing "terrain" of the data.
SAMPLe: It's like a seasoned hiker. It knows when to push hard to climb a hill (exploit) and when to wander slightly to find a safer, wider path (explore). It dynamically adjusts its strategy based on how the robot is feeling at that exact moment.

The Results: A More Robust Robot

The authors tested SAMPLe on 11 different image datasets (like recognizing pets, cars, flowers, and aircraft) and compared it against the best existing methods.

The "Base" vs. "New" Balance: In these tests, the robot is trained on some categories (Base) and then tested on new ones it has never seen (New).
The Win: SAMPLe consistently achieved the best balance. It didn't just get a high score on the training data; it kept its high scores when facing new, tricky data.
The Metaphor: If other methods were like a student who memorized the textbook but failed the surprise quiz, SAMPLe was the student who understood the concepts so well that they could answer any question, even the ones not in the book.

Summary

SAMPLe is a new way to teach AI models new tasks without breaking their ability to handle the unexpected. Instead of just chasing the highest possible score on the training data, it looks for a "safe zone" where the model is both accurate and robust. It ensures that the AI doesn't just memorize the answers but learns how to generalize, making it a much more reliable tool for real-world applications where data is always changing.

Technical Summary: SAMPLe: A Sharpness Aware Minimization based Optimizer for Prompt Learning in Vision-Language Models

Problem Statement
Pre-trained Vision-Language Models (VLMs), such as CLIP, serve as powerful foundation models for downstream applications. However, adapting these models via prompt learning presents a significant performance-generalization dilemma. While prompt learning allows for efficient task adaptation by tuning a limited set of learnable parameters (prompts) while keeping the backbone frozen, this constraint often leads to overfitting. The limited parameter space makes the optimization process highly sensitive, frequently causing models to converge to "sharp minima" in the loss landscape. These sharp solutions yield high accuracy on seen training distributions but suffer from poor generalization to unseen data or classes. Existing Sharpness-Aware Minimization (SAM) methods attempt to address this by seeking flatter minima, but they often fail to balance the trade-off between minimizing training loss (exploitation) and exploring flat regions (exploration), particularly when applied to the constrained subspace of prompt learning.

Methodology: SAMPLe
The authors propose SAMPLe (Sharpness-Aware Minimization Prompt Learning), a model-agnostic optimizer designed specifically to enhance the generalizability of VLM prompts. Unlike conventional optimizers or standard SAM variants (e.g., F-SAM, SAGM) that may apply uniform perturbations or rely on heuristic hyperparameters, SAMPLe introduces a dual-objective framework that dynamically adapts to the current optimization state.

The core of SAMPLe involves three strategic mechanisms:

Dual-Objective Optimization: The method optimizes for a loss minimum that is simultaneously low in training loss and flat in the loss landscape. This ensures stability and robustness without sacrificing accuracy on seen data.
Gradient Alignment and Orthogonality: SAMPLe ensures that gradient updates computed at a deviated point (perturbed by SAM) are coherently aligned with Empirical Risk Minimization (ERM) gradients. Crucially, it constrains the SAM gradient updates at the perturbed point to be orthogonal to the full-batch gradient.
- It decomposes the mini-batch gradient into a component aligned with the full-batch gradient and a "batch-specific" component orthogonal to it.
- The objective function includes terms that drive the perturbed gradient toward the batch-specific component (exploration) while resisting the full-batch component (which can lead to over-smoothing and reduced generalization).
Dynamic Adaptation: The optimizer automatically balances exploitation (minimizing ERM loss) and exploration (seeking flat minima) at every iteration based on the local curvature and gradient properties, rather than relying on fixed hyperparameters.

Theoretical analysis in the paper demonstrates that under standard assumptions (bounded gradients and K-Lipschitz continuity), SAMPLe converges with a rate of $O(\frac{\log T}{\sqrt{T}})$ , comparable to SGD and Adam.

Key Contributions

Novel Optimizer: Introduction of SAMPLe, the first SAM-based optimizer specifically tailored for VLM prompt learning, defining a dual-objective framework that adaptively balances low ERM loss and flat minima.
Theoretical Insight: An analysis establishing the limitations of existing sharpness-aware methods in the context of prompt learning's limited parameter space, motivating the need for an adaptive, dual-objective approach.
Comprehensive Evaluation: Extensive experiments integrating SAMPLe into multiple prompt learning frameworks (CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt) across diverse datasets.

Experimental Results
The paper evaluates SAMPLe across three primary settings using 11 image classification datasets and various ImageNet variants:

Base-to-New Generalization: In few-shot learning scenarios, SAMPLe consistently improves the harmonic mean (HM) of base and novel class accuracies compared to baselines and other SAM variants (SAM, F-SAM, SAGM). For instance, on average, CoOp improved from 77.65 (with SAGM) to 78.88 with SAMPLe.
Cross-Dataset Zero-Shot Generalization: When models trained on ImageNet are tested on 10 unseen target datasets without fine-tuning, SAMPLe consistently boosts transfer performance. For CoOp, average accuracy increased from 63.88% to 65.85%.
Cross-Domain Generalization: On ImageNet variants representing distribution shifts (V2, Sketch, Adversarial, Rendition), SAMPLe enhanced robustness. CoOp+SAMPLe achieved an average accuracy of 60.41%, outperforming the standard CoOp (59.28%) and other strong baselines.

The results indicate that SAMPLe achieves a more stable trade-off between specialization on base classes and transfer to unseen classes, often outperforming state-of-the-art methods without drastically degrading performance on seen distributions.

Significance and Claims
The paper claims that SAMPLe establishes a robust, model-agnostic solution for prompt learning by directly addressing the overfitting-prone nature of the limited learnable parameter space in VLMs. By incorporating loss landscape awareness and dynamically balancing exploration and exploitation, SAMPLe preserves the generalization potential of pre-trained models. The authors assert that their approach elevates existing prompt learning frameworks, providing a consistent performance boost across diverse settings and demonstrating that careful optimization of the loss landscape is critical for effective prompt learning in VLMs.

The Problem: The "Over-Confident" Student

The Solution: SAMPLe (The "Safety-First" Coach)

The Results: A More Robust Robot

Summary

More like this