💻 computer science

Prompt-based Adaptation in Large-scale Vision Models: A Survey

This survey presents a unified "Prompt-based Adaptation" framework that systematically distinguishes between pixel-level Visual Prompting and token-level Visual Prompt Tuning, categorizing them by generation mechanisms while reviewing their applications across diverse domains, benchmarks, and future challenges.

Original authors: Xi Xiao, Yunbei Zhang, Lin Zhao, Yiyang Liu, Xiaoying Liao, Zheda Mai, Xingjian Li, Xiao Wang, Hao Xu, Jihun Hamm, Xue Lin, Min Xu, Qifan Wang, Tianyang Wang, Cheng Han

Published 2026-03-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xi Xiao, Yunbei Zhang, Lin Zhao, Yiyang Liu, Xiaoying Liao, Zheda Mai, Xingjian Li, Xiao Wang, Hao Xu, Jihun Hamm, Xue Lin, Min Xu, Qifan Wang, Tianyang Wang, Cheng Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot chef who has spent years learning to cook millions of dishes using a massive library of recipes. This robot is incredibly talented, but it's also very rigid. If you ask it to cook a specific regional dish it's never seen before, it might struggle because it's used to following its original, massive recipe book exactly.

Traditionally, to teach this robot a new dish, you'd have to rewrite its entire recipe book (a process called Full Fine-Tuning). This is expensive, slow, and risks making the robot forget how to cook its original favorite dishes.

This paper introduces a smarter, lighter way to teach the robot: Prompt-based Adaptation (PA). Instead of rewriting the whole book, you just give the robot a tiny, specific note or a visual cue to guide its cooking. The paper surveys two main ways to do this: Visual Prompting (VP) and Visual Prompt Tuning (VPT).

Here is the breakdown of the paper using simple analogies:

1. The Two Main Strategies: The "Sticky Note" vs. The "Secret Handshake"

The authors divide these techniques into two categories based on where the instruction is given to the robot.

Visual Prompting (VP): The "Sticky Note" on the Ingredient

How it works: You modify the input (the raw ingredients) before the robot even starts cooking. You might draw a circle around the onion, add a colored sticker to the image, or overlay a pattern.
The Analogy: Imagine you are showing the robot a picture of a cat. Instead of changing the robot's brain, you draw a red box around the cat's ears on the photo. You are saying, "Hey, look right here!"
Types:
- Fixed: You use a pre-drawn box or a human points with a finger (like in interactive apps). No learning needed.
- Learnable: The robot learns to draw the best possible box or pattern itself to get the right answer.
- Generated: A tiny helper robot draws a unique, custom note for every single photo it sees, adapting instantly to the lighting or angle.
Best for: When you can't touch the robot's brain (it's a "black box" API) or when you need to interact with the image directly (like clicking on a tumor in a medical scan).

Visual Prompt Tuning (VPT): The "Secret Handshake" inside the Brain

How it works: You don't touch the photo. Instead, you inject tiny, invisible "tokens" (like secret codes) directly into the robot's internal thinking process. The robot's main brain stays frozen, but these tiny codes whisper instructions to it as it processes the image.
The Analogy: Imagine the robot is reading a book. You don't change the book pages. Instead, you slip a tiny, magical bookmark into the book that says, "When you see a dog, think 'Golden Retriever'." The robot reads the same pages but interprets them differently because of your bookmark.
Types:
- Learnable: You train a small set of these "bookmarks" to be the perfect hint for a specific task.
- Generated: A small helper creates a unique set of bookmarks for every single image, tailoring the advice to the specific situation.
Best for: When you have access to the robot's internal layers and need to teach it complex, deep concepts (like understanding the difference between two very similar types of clouds or recognizing objects in 3D space).

2. Why is this a Big Deal? (The "Efficiency" Factor)

The paper explains that teaching the whole robot (Full Fine-Tuning) is like trying to remodel a skyscraper just to add a new door. It's costly and risky.

PA is like adding a smart doorbell. It's cheap, fast, and doesn't risk collapsing the building.
Memory: It saves a massive amount of computer memory because you aren't saving new weights for the whole robot, just the tiny "notes" or "bookmarks."
Data: It works even if you only have a few examples of the new task (Few-Shot Learning), because the robot already knows how to cook; it just needs a nudge on what to cook.

3. Where is this being used? (The "Real World" Applications)

The survey shows these techniques are being used everywhere:

Medicine: Helping doctors spot tumors in X-rays by drawing a "prompt" around the suspicious area without retraining the whole AI.
Robotics: Teaching a robot arm to understand 3D objects (like a pile of tools) by translating 2D images into 3D instructions.
Self-Driving Cars: Helping cars recognize roads in heavy rain or fog by "prompting" the system to ignore the rain and focus on the lane.
Industrial Factories: Spotting tiny defects on metal sheets by teaching the AI to look for specific "flaws" without needing thousands of photos of broken parts.

4. The Challenges (The "Gotchas")

Even though this is a great tool, the paper warns of some hurdles:

Safety: Just like you can trick a human with a misleading sign, bad actors could use "prompts" to trick the AI into doing something harmful or biased.
Instability: Sometimes, changing the prompt just a tiny bit (like moving a dot by one pixel) can make the robot fail completely. It can be a bit fragile.
Speed: Adding these extra "notes" or "bookmarks" takes a tiny bit of extra time to process, which matters if you need the robot to react instantly.
Real-World Testing: Most tests are done on clean, perfect datasets. The paper urges researchers to test these methods in the messy, chaotic real world (like a rainy street or a dusty factory) to see if they really hold up.

Summary

This paper is a guidebook for researchers. It says: "Stop trying to rebuild the whole AI brain every time you want to teach it something new. Instead, use these clever, lightweight 'prompts' (either as visual notes on the image or secret codes in the brain) to guide the AI. It's cheaper, faster, and works almost as well as the old way."

It's the difference between rewriting the entire dictionary to add a new word versus just writing a sticky note on the page that says, "This new word means X." The paper maps out exactly how to write those sticky notes for the future of AI.

1. Problem Statement

Large-scale vision models (e.g., Vision Transformers, Swin Transformers) have revolutionized computer vision through the "pretrain-then-finetune" paradigm. However, as model scales increase, conventional full fine-tuning (FT) becomes prohibitively expensive in terms of computation, storage, and memory, while also risking the erosion of valuable pre-trained knowledge (catastrophic forgetting).

While Parameter-Efficient Fine-Tuning (PEFT) methods have emerged to address this, the field of Prompt-based Adaptation (PA) suffers from conceptual ambiguity. The terms "Visual Prompting" (VP) and "Visual Prompt Tuning" (VPT) are frequently used interchangeably in literature, despite representing distinct mechanisms with different injection granularities and application scopes. There is a lack of a unified framework to systematically distinguish, categorize, and analyze these methods, particularly regarding their theoretical underpinnings, efficiency trade-offs, and applicability across diverse domains and constraints.

2. Methodology: A Unified Taxonomy

The authors propose a unified framework termed Prompt-based Adaptation (PA), which categorizes methods based on two primary axes: Injection Granularity and Generation Mechanism.

A. Injection Granularity

The survey distinguishes between two core paradigms based on where the prompt interacts with the model:

Visual Prompting (VP) – Pixel Level:
- Mechanism: Modifies the input image before tokenization/feature extraction via a prompt function $u(x; \theta)$ .
- Sub-types:
  - VP-Fixed: Predefined prompts (e.g., points, boxes, masks in SAM) provided by users or rules. No learnable parameters.
  - VP-Learnable: Optimizes pixel-space patterns (overlays, residuals, frequency cues) while keeping the backbone frozen.
  - VP-Generated: Uses a lightweight generator network to synthesize instance-adaptive prompts for each input image.
- Use Case: Ideal for black-box scenarios, interactive segmentation, and input-space rectification.
Visual Prompt Tuning (VPT) – Token Level:
- Mechanism: Injects learnable prompt tokens directly into the token/feature sequence inside the network layers (e.g., prepending to the input sequence or inserting between Transformer blocks).
- Sub-types:
  - VPT-Learnable: Static learnable tokens optimized via gradient descent (e.g., VPT-Shallow at the first layer, VPT-Deep at all layers).
  - VPT-Generated: A generator produces dynamic, instance-specific prompt tokens conditioned on the input.
- Use Case: Ideal for deep semantic adaptation, domain shifts, and tasks requiring internal feature recalibration.

B. Generation Mechanism

Methods are further classified by how prompts are obtained:

Fixed: Immutable, zero-shot friendly, but limited by the frozen backbone's inherent capabilities.
Learnable: Static vectors optimized for a dataset; effective for domain adaptation but lack instance-level flexibility.
Generated: Dynamic prompts conditioned on the input; offer high flexibility and robustness to intra-class variations but incur inference latency.

3. Key Contributions

First Unified Taxonomy: The paper establishes the first structured framework that disentangles VP and VPT, clarifying their distinct algorithmic designs, injection locations, and use cases.
Comprehensive Survey: It covers a vast landscape of PA applications, including foundational tasks (segmentation, restoration, compression), domain-specific applications (medical, remote sensing, robotics, industrial), and constrained learning paradigms (few-shot, test-time adaptation, federated learning).
Efficiency Analysis: The authors provide a detailed breakdown of memory and computational costs, highlighting that while both methods reduce parameter/optimizer memory, VP offers superior black-box compatibility and training-free options, whereas VPT offers deeper semantic adaptation at the cost of activation memory.
Theoretical Insights: The survey explores foundational questions, such as how PA alters model attention (via GradCAM), what prompts actually learn (re-mapping label spaces vs. learning independent representations), and the non-linear relationship between prompt length and performance.
Trustworthy AI & Challenges: It addresses PA's role in robustness, fairness, and privacy, and identifies critical challenges like safety alignment, training instability, inference latency, and the gap between academic benchmarks and real-world deployment.

4. Key Results and Findings

Performance vs. Efficiency:
- VP is superior for black-box scenarios and interactive tasks (e.g., medical segmentation with SAM) where internal gradients are inaccessible. It effectively acts as a "preprocessing" layer.
- VPT generally outperforms VP in tasks requiring deep semantic realignment (e.g., remote sensing, 3D point clouds) where the pre-trained feature space is misaligned with the target domain.
- Memory: VPT reduces parameter storage significantly (<0.5% of total params) but does not significantly reduce activation memory during training. VP-Fixed eliminates prompt-side memory entirely.
Domain Adaptation:
- In Medical Imaging, VP dominates segmentation (using points/boxes), while VPT excels in report generation and multi-modal reasoning.
- In Remote Sensing, VP is preferred for instance segmentation (guiding models through clutter), while VPT is used for scene classification and cross-modal retrieval.
- In Adverse Environments (underwater, fog), VP provides spatial anchors for degraded boundaries, while VPT enforces feature invariance against environmental noise.
Constrained Learning: PA is highly effective in Few-Shot Learning (preventing overfitting), Test-Time Adaptation (TTA) (adapting to distribution shifts online), and Federated Learning (minimizing communication overhead).
Prompt Length: Empirical studies show a non-linear relationship between prompt length and accuracy; diminishing returns occur quickly, suggesting that a small number of high-quality prompts are often sufficient.

5. Significance

This survey serves as a critical roadmap for researchers and practitioners by:

Clarifying Confusion: It resolves the terminological ambiguity between VP and VPT, enabling more precise communication and method selection.
Guiding Deployment: It provides practical guidelines for selecting between VP and VPT based on constraints (e.g., black-box access, memory limits, need for deep semantic adaptation).
Identifying Future Directions: It highlights urgent needs for improving training stability, reducing inference latency, ensuring safety/alignment, and validating methods on real-world, heterogeneous datasets rather than just academic benchmarks.
Bridging Theory and Practice: By connecting theoretical analysis (e.g., attention mechanisms, mutual information) with practical applications across diverse industries, it accelerates the adoption of PA in real-world AI systems.

In conclusion, the paper positions Prompt-based Adaptation not just as a parameter-saving technique, but as a fundamental shift in how large vision models are adapted, offering a flexible, modular, and efficient alternative to full fine-tuning that is essential for the next generation of scalable and trustworthy AI.