🤖 machine learning

Robust Adaptation of Foundation Models with Black-Box Visual Prompting

This paper proposes BlackVIP, a parameter-efficient framework that adapts large-scale pre-trained models in black-box settings without access to their parameters or intermediate activations by utilizing a Coordinator for input-dependent visual prompts and SPSA-GC for efficient gradient estimation, while also offering a theoretical link to certified robustness.

Original authors: Changdae Oh, Gyeongdeok Seo, Geunyoung Jung, Zhi-Qi Cheng, Hosik Choi, Jiyoung Jung, Kyungwoo Song

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Changdae Oh, Gyeongdeok Seo, Geunyoung Jung, Zhi-Qi Cheng, Hosik Choi, Jiyoung Jung, Kyungwoo Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-genius chef (the "Foundation Model") who has spent years cooking in a massive, high-tech kitchen. This chef knows how to make thousands of dishes perfectly. However, there's a catch:

You can't see the kitchen: The chef works in a locked room. You can't touch the stove, change the recipes, or see the ingredients list (the model's internal parameters are hidden).
You can't bring your own ingredients: You can't walk in and say, "Hey, let's add salt here." You can only hand the chef a plate of raw food and ask, "Can you cook this?"
Memory is tight: The chef's kitchen is so big that trying to remember every single step of every recipe to make changes would crash the building's power grid (too much memory required).

Usually, to teach this chef a new trick (like making a specific type of sushi), you'd need to rewrite their entire recipe book. But since you can't see the book, you're stuck.

Enter "BlackVIP": The Magic Napkin.

This paper introduces a clever new way to teach the chef without ever entering the kitchen or rewriting their recipes. Instead, they use a Magic Napkin (called a "Visual Prompt").

Here is how it works, broken down into simple parts:

1. The Magic Napkin (Visual Prompting)

Instead of trying to change the chef's brain, you draw a picture on a napkin and place it on top of the raw food before handing it to the chef.

Old way: You tried to draw the same static picture on every single plate (like a fixed border).
BlackVIP way: You have a tiny, super-fast robot artist (called the Coordinator) that looks at each specific plate of food and draws a custom picture on the napkin just for that dish.
- Analogy: If you're handing the chef a picture of a cat, the robot draws a "meow" sign on the napkin. If it's a dog, it draws a "woof" sign. The robot adapts the hint based on what it sees.

2. The Blind Taste Test (Black-Box Optimization)

Since you can't see the chef's brain, how do you know if your napkin drawing is good?

You can't ask the chef, "Why did you get this wrong?" (No gradients/parameters).
The Solution: You play a game of "Hot and Cold."
1. You give the chef a plate with a slightly different napkin drawing.
2. You see if the result gets better or worse.
3. You do this thousands of times, slightly tweaking the drawing each time, until you find the perfect napkin art that makes the chef cook the dish correctly.
The Upgrade (SPSA-GC): The paper introduces a special "Smart Guessing" algorithm. Instead of just guessing randomly, it uses a momentum trick (like a skier leaning into a turn) to guess the right direction faster and avoid getting stuck in the mud. This saves you from having to ask the chef thousands of questions.

3. The "Lite" Version (BlackVIP-SE)

The original robot artist (BlackVIP) is great but needs a small computer to run. The authors created a "Lite" version called BlackVIP-SE.

Instead of using a complex robot, this version uses a simple statistical trick (like sorting a deck of cards by color) to figure out what to draw on the napkin.
It's much faster and cheaper, almost as good as the robot, but runs on a toaster instead of a supercomputer.

Why is this a big deal?

It works with locked doors: You can use this on any AI model, even if the company that made it refuses to share the code (like using an API from a big tech company).
It saves memory: You don't need a massive server farm to train it. A regular laptop can do it.
It's robust: The paper shows that this method is surprisingly good at handling "tricky" situations.
- Example: If you train the chef to recognize a "7" only when it's on a pink background, but then you show them a "7" on a blue background, most chefs get confused. BlackVIP's custom napkin helps the chef ignore the background color and focus on the actual shape of the number.

The Big Picture

Think of BlackVIP as a universal translator for AI. It doesn't try to reprogram the AI's brain (which is often impossible). Instead, it learns how to "speak" to the AI by drawing the perfect little hints (prompts) on the input, guiding the AI to do exactly what you want, even if you can't see inside the machine.

It turns a "Black Box" (a mystery machine) into a helpful tool that can be adapted to new tasks quickly, cheaply, and securely.

1. Problem Statement

The paper addresses the limitations of current Parameter-Efficient Transfer Learning (PETL) methods when applied to large-scale Pre-trained Models (PTMs) or Foundation Models in real-world scenarios. Existing methods (e.g., Visual Prompt Tuning, LoRA) rely on two optimistic assumptions that often fail in practice:

Full Parameter Accessibility: They require access to the model's internal weights and architecture to compute gradients via backpropagation. However, many state-of-the-art PTMs are deployed as black-box APIs or proprietary software where parameters are hidden.
Memory Capacity: Even if parameters are accessible, fine-tuning or prompting often requires caching all intermediate activations to compute gradients, which demands prohibitive memory resources for large models.

The authors aim to develop a method that can adapt PTMs to downstream tasks without access to model parameters, architecture details, or large memory buffers, while maintaining robustness against distribution shifts.

2. Methodology: BlackVIP

The proposed framework, BlackVIP (Black-Box Visual Prompting), consists of two core components:

A. Coordinator (Input-Dependent Visual Prompting)

Unlike previous visual prompting methods that learn a static, input-agnostic prompt (e.g., a fixed frame added to images), BlackVIP introduces a Coordinator, a tiny neural network that generates input-dependent visual prompts.

Mechanism: The Coordinator takes an input image $x$ , processes it through a fixed encoder $f(\cdot)$ (or a statistical projector), and outputs a visual prompt $v_\phi(x)$ of the same shape as the image.
Reparameterization: Instead of optimizing raw pixels directly, the method optimizes the parameters $\phi$ of the Coordinator network.
BlackVIP-SE (Statistical feature-guided Efficient prompt): A lightweight variant that replaces the heavy pre-trained encoder with a Principal Component Analysis (PCA) projection. This significantly reduces computational cost and runtime while maintaining competitive performance.
Output: The generated prompt is added to the original image (clipped to valid RGB range) and fed into the black-box target model.

B. SPSA-GC (Simultaneous Perturbation Stochastic Approximation with Gradient Correction)

Since the target model is a black box, true gradients cannot be computed. The authors employ a zeroth-order optimization strategy.

SPSA: Standard SPSA estimates gradients by perturbing parameters in random directions and observing the change in the loss function. It requires only two forward passes (API calls) per iteration regardless of parameter dimension.
Gradient Correction (GC): To address the slow convergence and high variance of standard SPSA in high-dimensional settings, the authors introduce a momentum-based look-ahead mechanism inspired by Nesterov's Accelerated Gradient (NAG).
- The update rule computes the gradient at a "look-ahead" position ( $\phi_i + \beta m_i$ ) rather than the current position.
- This "correction" mitigates the noise inherent in stochastic gradient estimation, leading to faster and more stable convergence.

3. Key Contributions

Black-Box Adaptation Framework: The first work to propose a robust visual prompting method for large PTMs that operates entirely in a black-box setting (no parameter access) with minimal memory requirements.
Input-Dependent Prompting: Introduction of the Coordinator, which dynamically generates prompts based on input features, allowing the model to adapt to diverse semantic contexts rather than using a static prompt.
SPSA-GC Optimizer: A novel optimization algorithm that combines SPSA with gradient correction, significantly improving convergence speed and stability for high-dimensional black-box optimization.
BlackVIP-SE Variant: An efficient variant using PCA for feature extraction, reducing training/inference time and parameter count (down to ~1K parameters) without sacrificing much accuracy.
Theoretical Analysis: The paper provides a theoretical proof connecting visual prompting to Certified Robustness via Randomized Smoothing. It demonstrates that the optimization trajectory of visual prompts asymptotically follows a Gaussian distribution, effectively acting as an adaptive randomized smoothing classifier. This explains why the method improves generalization and robustness against $L_2$ noise and distribution shifts.

4. Experimental Results

The authors evaluated BlackVIP on 19 datasets, covering diverse domains (natural images, satellite imagery, fine-grained recognition, medical imaging) and tasks.

Performance: BlackVIP and BlackVIP-SE consistently outperformed baselines (Zero-Shot, Black-Box Adversarial Reprogramming (BAR), and standard Visual Prompting with SPSA) on 13 out of 14 standard benchmarks.
- On CLEVR (visual reasoning), BlackVIP significantly outperformed BAR, which struggled with complex reasoning tasks.
- On Biased MNIST (correlation shift), BlackVIP showed superior robustness, successfully ignoring spurious correlations (e.g., background color) that misled other models.
Efficiency:
- Memory: BlackVIP requires minimal memory (e.g., ~2.4 GB peak for ViT-B vs. ~21 GB for full Fine-Tuning) as it does not store intermediate activations.
- Parameters: It learns only 9K (BlackVIP) or 1K (BlackVIP-SE) parameters, compared to millions in full fine-tuning.
- Query Efficiency: It achieves significant performance gains with a relatively low number of API calls (approx. 10k calls), offering a favorable cost-performance trade-off.
Robustness: The method demonstrated strong robustness against $L_2$ noise and real-world distribution shifts (WILDS datasets: iWildCam, FMoW, Camelyon17).
Compatibility: The visual prompts are compatible with black-box text prompting methods (e.g., CuPL, LLM BBO), further boosting performance when combined.

5. Significance

Democratizing Foundation Models: BlackVIP enables users with limited computational resources and no access to model weights to effectively adapt state-of-the-art foundation models to specific downstream tasks.
Robust AI: By theoretically linking visual prompting to randomized smoothing, the paper offers a new perspective on how to build robust AI systems that are less susceptible to distribution shifts and adversarial noise without explicit robust training.
Practical Deployment: The introduction of BlackVIP-SE and the compatibility with quantized models make this approach highly suitable for edge devices and cost-sensitive commercial applications where inference latency and memory are critical constraints.

In summary, BlackVIP bridges the gap between the theoretical potential of foundation models and the practical constraints of real-world deployment, offering a memory-efficient, parameter-light, and robust solution for black-box transfer learning.