💻 computer science

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

ZOMP is a query-efficient, fully forward-only method that optimizes deep prompts in both vision and text branches of frozen CLIP models using simultaneous perturbation stochastic approximation, outperforming existing zeroth-order approaches across multiple benchmarks by leveraging cross-modal low-rank reparameterization, gradient-correction momentum, and a dynamic rank schedule.

Original authors: Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh, Ramtin Pedarsani

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh, Ramtin Pedarsani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that has read the entire internet and learned to understand both pictures and words. It's like a genius librarian who can instantly tell you what a photo is about just by looking at it, or describe a scene using perfect sentences. This robot is called a Vision-Language Model, and it's incredibly powerful. But here's the catch: usually, to teach this robot a new trick—like recognizing a specific type of rare flower or a weird medical scan—you have to let it "rewire" its own brain. This process is heavy, like trying to reorganize a massive library while it's still open to the public, and it requires a lot of computing power and memory.

Sometimes, though, you can't do that. Maybe the robot lives on a tiny, battery-powered device in your pocket, or maybe it's a secret service you can only talk to through a chat window that doesn't let you see how it thinks. In these cases, you can't rewire the brain; you can only whisper new instructions to it. This is called "prompt tuning." Instead of changing the robot's brain, you give it a special set of hints or "prompts" to help it solve a new puzzle. The problem is, finding the perfect hints usually requires a lot of trial and error, and if you can't see the robot's internal thoughts (gradients), it's like trying to find a needle in a haystack while blindfolded. You might need thousands of guesses just to get it right, which is too slow and expensive for real life.

This is where a new method called ZOMP comes in. Think of ZOMP as a clever detective who solves the "blindfolded needle" problem by changing how they search. Instead of guessing every single detail of the hint all at once, ZOMP uses a smart strategy to guess the most important parts first, then slowly fills in the rest.

Here's how it works, using a few fun metaphors:

The "Shared Blueprint" Trick
Usually, when you try to teach the robot new things, you have to write separate hints for its "eyes" (vision) and its "voice" (text). If you try to write deep, complex hints for both at the same time, the number of possibilities becomes so huge that the blindfolded search gets lost. ZOMP changes the game by saying, "Let's use a shared blueprint." Imagine that the hints for the eyes and the voice are both built from the same small set of Lego bricks. You only have to figure out how to arrange those few bricks, and they automatically build the right hints for both the eyes and the voice at the same time. This keeps the search space small and manageable, even though the hints are deep and complex.

The "Budget-Indexed" Expansion
Even with the shared blueprint, trying to arrange all the bricks at once is still too hard when you are blindfolded. ZOMP uses a "budget" system. Imagine you have a limited number of guesses (queries) to find the best hint. ZOMP starts by only allowing itself to play with the very first, most important brick. It finds the right direction for that one piece. Once it's sure it's on the right track, it "unlocks" the next brick, then the next, slowly expanding the search only when it has a reliable direction. It's like learning to drive: you start in an empty parking lot (a tiny search space), get comfortable, and then slowly move to the neighborhood streets, and finally the highway. You don't try to drive on the highway on your very first day.

The "Momentum" Memory
Because the search is blindfolded, the clues it gets are often noisy and shaky, like trying to walk on a boat in choppy water. ZOMP adds a "momentum" memory. Think of this as a surfer who doesn't just react to every single wave, but remembers the general direction the ocean is pushing. Even if a wave knocks them sideways, they keep moving forward in the right direction because they remember where they were going. This helps the robot ignore the noise and stick to the path that actually works.

The Results
The researchers tested this method on 13 different challenges, from identifying flowers to recognizing vehicles in satellite images. They gave ZOMP and other methods the exact same number of guesses (5,000 queries). The results showed that ZOMP consistently found better hints than the other methods. It didn't just get slightly better; it was often significantly more accurate, especially on tricky tasks where the robot usually struggles.

What's really cool is that ZOMP didn't just get better at the specific task it was trained on. It also got better at handling new, weird situations it had never seen before (like recognizing a sketch of an object instead of a photo). This suggests that by searching carefully and efficiently, ZOMP learned a smarter way to use the robot's existing knowledge, rather than just memorizing the training examples.

In short, ZOMP proves that you don't need to rewire the robot's brain to make it smarter. By using a shared blueprint, expanding the search slowly, and keeping a steady memory of the direction, you can teach a powerful AI new tricks even when you can only whisper to it and have a limited number of tries. It's a practical, efficient way to get the most out of these giant models without needing a supercomputer in your pocket.

Technical Summary: ZOMP (Zeroth-Order Multi-Modal Prompt Tuning)

Problem Statement

Fine-tuning large vision-language models (VLMs) like CLIP typically relies on backpropagation (BP) through the entire model. However, this is infeasible in two increasingly common scenarios: memory-constrained edge devices where the backward computation graph cannot fit, and proprietary model deployments where weights are inaccessible via inference APIs. In these settings, tuning must rely solely on forward passes (zeroth-order optimization).

Existing backpropagation-free (BP-free) prompt tuning methods attempt to address this by estimating gradients via random perturbations (e.g., evolutionary strategies or Zeroth-Order Stochastic Gradient Descent). However, these methods face a critical trade-off:

Shallow or Single-Modality: To maintain query efficiency, prior methods like BAR, BlackVIP, and ZIP restrict tuning to shallow prompts or single modalities (e.g., text only). They avoid deep, multi-modal prompting because the variance of zeroth-order gradient estimates grows with the dimensionality of the search space.
Convergence Limits: Deep multimodal prompting (injecting learnable vectors into multiple layers of both image and text encoders) is known to yield superior accuracy under BP, but the high dimensionality makes it impractical under realistic query budgets when using zeroth-order methods.

The core problem is reconciling the expressiveness of deep, multi-modal prompting with the query-efficiency constraints of zeroth-order optimization.

Methodology: ZOMP

ZOMP (Zeroth-Order Multi-Modal Prompt tuning) proposes a fully forward-only method that enables deep, multi-modal prompt tuning by restructuring the parameter space and the optimization trajectory. The method relies on three key components:

1. Cross-Modal Low-Rank Reparameterization
Instead of optimizing prompt matrices directly, ZOMP adopts a factorized form inspired by LoRA and MMLoP. For each layer $l$ , the visual prompt $P_v^{(l)}$ and text prompt $P_t^{(l)}$ are generated as:
$P_v^{(l)} = U^{(l)} V_v^{(l)}, \quad P_t^{(l)} = U^{(l)} V_t^{(l)}$
Here, $U^{(l)}$ is a small, shared factor common to both modalities, while $V_v^{(l)}$ and $V_t^{(l)}$ are modality-specific factors.

Benefit: This ties the two encoders together, ensuring that a single perturbation of the shared factor $U^{(l)}$ moves both prompts coherently. It reduces the effective search dimensionality and allows the rank $r$ to be treated as a tunable "knob."

2. Budget-Indexed Progressive Rank Schedule
To manage the variance of zeroth-order estimates, ZOMP does not optimize all rank components simultaneously from the start. Instead, it employs a schedule indexed by the query budget fraction.

Mechanism: The optimizer begins by searching only a small, low-rank subspace (e.g., rank-1). As the query budget is consumed, the rank is progressively unlocked (e.g., from rank 1 to the full rank $r$ ).
Initialization: All modality-specific factors $V$ are initialized to zero. This ensures that newly unlocked components contribute nothing initially, making the transition between ranks function-preserving and preventing "initialization shock" that would waste queries.

3. Gradient-Correction Momentum
Zeroth-order estimates are inherently noisy. ZOMP incorporates a momentum term based on SPSA-GC (Simultaneous Perturbation Stochastic Approximation with Gradient Correction).

Mechanism: The momentum buffer accumulates past gradient estimates, smoothing the trajectory.
Synergy: The low-rank early phase provides a low-noise direction for the momentum to lock onto. As the rank expands and variance increases, the momentum buffer preserves the descent direction found in the low-variance phase, preventing the optimizer from being derailed by noise.

Algorithm Flow:
The algorithm (Algorithm 1) iterates through the query budget. In each step, it samples perturbation directions masked to the currently active rank components, computes the SPSA gradient estimate, applies momentum correction, and updates the parameters. The rank mask is updated based on the cumulative query count.

Key Contributions

Deep Multimodal Prompting under Zeroth-Order: ZOMP is the first BP-free method to successfully implement deep prompting across both vision and text encoders. It achieves this by using cross-modal low-rank factorization to make the high-dimensional search space tractable.
Synergistic Training Mechanisms: The paper introduces and validates a budget-indexed rank schedule paired with gradient-correction momentum. Ablation studies show that neither mechanism is sufficient alone; the schedule provides a low-noise starting point for the momentum, while momentum allows the optimizer to survive the variance spike when the search space widens.
State-of-the-Art Performance: ZOMP demonstrates superior few-shot accuracy, query efficiency, and generalization capabilities compared to existing BP-free baselines (BAR, BlackVIP, BPT-VLM, and ZIP).

Experimental Results

The authors evaluated ZOMP on 13 vision-language benchmarks under a matched 5,000-query budget using a frozen CLIP ViT-B/16 backbone.

Few-Shot Classification: ZOMP achieved an average accuracy of 68.2% across 13 datasets, outperforming the strongest prior method (ZIP, 62.2%) by 6.0 points. It surpassed all baselines on every individual dataset, with particularly large gains on specialized domains (e.g., +15.9 on EuroSAT, +10.3 on CLEVR).
Base-to-New Generalization: In settings where prompts are trained on base classes and tested on new classes, ZOMP achieved a harmonic mean (HM) of 71.2%, exceeding ZIP (67.5%) by 3.7 points. This indicates the method does not overfit to base classes at the expense of new ones.
Cross-Dataset & Out-of-Distribution (OOD): Prompts trained on ImageNet were transferred to 12 other datasets and 4 ImageNet variants without further tuning. ZOMP achieved the best average performance in both CDT (56.9%) and OOD (59.1%) settings, demonstrating robust generalization.
Query Efficiency: Despite having the highest number of trainable parameters (46.2k) among the methods tested, ZOMP reached target accuracy in the fewest queries. This contradicts the expectation that zeroth-order optimization degrades with parameter count, proving that the low-rank schedule effectively mitigates this issue.
Component Ablation: Removing either the rank schedule or momentum resulted in significant performance drops, confirming their complementary nature. Removing the cross-modal sharing (using independent $U$ factors) also reduced performance, validating the benefit of shared factors.

Significance and Claims

The paper claims that ZOMP closes the gap between the theoretical expressiveness of deep multimodal prompting and the practical constraints of backpropagation-free tuning. By jointly exploiting multimodality and low-rank structure, ZOMP provides a practical route to efficient prompt tuning for edge devices and proprietary APIs.

The authors assert that their results demonstrate that deep, multi-modal prompting is not inherently incompatible with zeroth-order optimization; rather, it requires a specific reparameterization and optimization schedule to manage variance. The work suggests that the "ceiling" previously observed in BP-free methods was a consequence of search space design rather than a fundamental limitation of the zeroth-order approach itself.