💬 NLP

Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

This paper introduces Multimodal Prompt Optimization (MPO), a unified framework that extends prompt optimization beyond text to jointly optimize textual and non-textual inputs (such as images, videos, and molecules) using alignment-preserving updates and Bayesian selection, thereby outperforming existing text-only methods and unlocking the full potential of Multimodal Large Language Models.

Original authors: Yumin Choi, Dongki Kim, Jinheon Baek, Sung Ju Hwang

Published 2026-02-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yumin Choi, Dongki Kim, Jinheon Baek, Sung Ju Hwang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant (a Multimodal Large Language Model, or MLLM) that can see pictures, watch videos, and even understand chemical structures. You want it to solve a tricky problem, like identifying a specific bird from a photo or predicting if a new drug will work.

To get the robot to do its best, you need to give it instructions, called a prompt.

The Problem: The "Text-Only" Blindfold

For a long time, people trying to teach these robots only used text prompts. They would write long, complicated sentences like: "Look at the bird. It has a pale bill, dark feathers, and lives in the North Pacific. It might be a Black-footed Albatross, but check if the bill is white..."

The paper argues this is like trying to describe a delicious pizza to someone who has never seen one, using only words. You might say "round, red sauce, melted cheese," but they might imagine a square pie with ketchup. It's inefficient and often confusing.

The authors realized: Why not just show the robot a picture of the pizza?

The Solution: MPO (The Multimodal Prompt Optimizer)

The team created a new system called MPO. Think of MPO as a super-tutor that doesn't just write better instructions; it also creates better visual aids to go with them.

Here is how MPO works, using a simple analogy:

1. The "Alignment-Preserving" Team (The Co-Pilots)

Imagine you are trying to fix a broken car.

Old Way: You have one mechanic who only talks. He says, "The engine sounds weird," and you have to guess what to do.
MPO Way: You have a Team of Co-Pilots. One is the Text Mechanic, and the other is the Visual Mechanic.
- If the car fails a test, they don't just argue about words. They look at the engine together.
- The Text Mechanic says, "We need to check the spark plugs."
- The Visual Mechanic immediately draws a diagram showing exactly where the spark plugs are.
- Crucially: They update their instructions together. The text and the picture always match. If the text says "red," the picture shows red. This prevents confusion (misalignment).

To do this, MPO uses three special tools (Operators):

Generation: "Let's draw a brand new picture from scratch to explain this."
Edit: "Let's take the old picture and just fix the blurry part."
Mix: "Let's take the best parts of Picture A and Picture B and combine them into one perfect guide."

2. The "Smart Selector" (The Experienced Coach)

Now, imagine MPO is trying out thousands of different text-and-picture combinations. It can't test them all; that would take forever. It needs a way to pick the winners quickly.

Old Way: The coach picks a random player to try out, sees if they fail, and tries another random one. It's slow and wasteful.
MPO Way: The coach uses Experience.
- The paper noticed that if a parent prompt (the original instruction) was good, its "children" (new variations) are likely to be good too.
- MPO uses a mathematical trick called Bayesian UCB. Think of it as a coach who says: "This new idea looks a lot like the one that worked great yesterday, so let's give it a chance first!"
- This allows MPO to skip the bad ideas and focus on the promising ones, saving about 42% of the time and money compared to older methods.

The Results: Why It Matters

The team tested MPO on three very different worlds:

Birds: Identifying specific bird species from photos.
Driving: Watching videos of cars and understanding what the driver should do.
Molecules: Looking at chemical structures to predict if a drug will be absorbed by the body.

The Outcome:
In every single test, MPO beat the best "text-only" experts.

Text-only was like trying to describe a complex 3D object with a 2D sketch.
MPO was like handing the robot a 3D model and a manual.

The Big Takeaway

This paper is a wake-up call. We built robots that can see, hear, and feel, but we've been teaching them using only one sense: reading.

Multimodal Prompt Optimization is the realization that to unlock the full power of these AI super-brains, we need to stop just talking to them and start showing them, too. By optimizing both the words and the visuals together, we can get much smarter, more accurate, and more efficient results.

1. Problem Definition

The paper addresses a critical gap in the current landscape of Multimodal Large Language Models (MLLMs). While MLLMs can process text, images, videos, and molecules, existing Automatic Prompt Optimization (APO) techniques remain confined to the textual modality.

Limitation: Current APO methods (e.g., APE, OPRO, SEE) optimize only text prompts. This forces MLLMs to rely on verbose, potentially ambiguous textual descriptions to convey visual or structural information, failing to leverage the models' inherent capacity to process rich non-textual context.
Goal: The authors define a new problem: Multimodal Prompt Optimization (MPO). The objective is to discover an optimal prompt pair $p = (t, m)$ , where $t$ is a textual prompt and $m$ is a non-textual prompt (e.g., an image, video frame, or molecule structure), to maximize task performance $f(MLLM(t, m, q), a)$.
Challenges:
1. Cross-Modal Alignment: Updating text and non-text components independently risks semantic misalignment (e.g., text describing a bird that the image does not show).
2. Search Space Complexity: The combinatorial space of multimodal prompts is vast, making high-quality prompts sparse and difficult to identify efficiently without wasting evaluation budgets.

2. Methodology: Multimodal Prompt Optimizer (MPO)

The authors propose MPO, a unified framework consisting of two core components: Alignment-Preserving Exploration and Prior-Inherited Bayesian UCB Selection.

A. Alignment-Preserving Exploration

To navigate the multimodal space while ensuring consistency between modalities, MPO employs a joint update strategy:

Cohesive Backpropagation: Instead of treating errors in text and non-text inputs separately, MPO identifies a failure set and generates a unified feedback signal ( $\nabla_p$ ) in textual form. This single signal encodes cross-modal weaknesses, guiding updates for both modalities simultaneously to prevent overfitting to one.
Joint Multimodal Update: The MLLM uses the feedback to generate an updated text prompt ( $t'$ $t^{'}$ ) and a modality-specific condition ( $c$ $c$ ). This condition $c$ $c$ directs a modality-specific generator ( $g$ $g$ ) to create or revise the non-textual prompt ( $m'$ $m^{'}$ ).
- Mechanism: $m' = g(c, \text{context})$ . This ensures the non-textual update is strictly guided by the textual reasoning derived from the failure analysis.
Exploration Operators: To diversify the search, MPO utilizes three operators:
- Generation: Creates entirely new non-textual prompts from scratch (useful for escaping local optima).
- Edit: Refines existing non-textual prompts (e.g., adjusting textures or specific features) while retaining useful structures.
- Mix: Combines complementary strengths from multiple parent prompts to create hybrid solutions.

B. Prior-Inherited Bayesian UCB Selection

To address the "cold-start" problem and the sparsity of high-quality prompts in the enlarged search space, MPO introduces a selection strategy based on Bayesian Upper Confidence Bound (UCB) with informative priors.

Hypothesis: The performance of a "child" prompt is positively correlated with its "parent" prompt.
Mechanism:
- Instead of initializing new candidates with a uniform prior (uninformative), MPO initializes the Beta distribution parameters ( $\alpha, \beta$ ) of a child prompt based on the posterior mean performance of its parent.
- Formula: $\alpha_i = \hat{\mu}_{par(i)} \cdot S + 1$ , where $S$ is a prior strength hyperparameter.
Benefit: This "warm-starts" the evaluation process, allowing the algorithm to rapidly eliminate low-potential candidates and focus resources on promising regions of the search space, significantly reducing the evaluation budget required.

3. Key Contributions

Problem Formulation: First to formally define and tackle Multimodal Prompt Optimization, expanding the optimization space beyond text to include images, videos, and molecules.
Unified Framework (MPO): Proposes a novel architecture that jointly optimizes text and non-text prompts via alignment-preserving updates, ensuring semantic consistency across modalities.
Efficient Selection Strategy: Introduces Prior-Inherited Bayesian UCB, which leverages parent-child performance correlations to reduce evaluation costs by 42% compared to prior-free baselines.
Comprehensive Evaluation: Validates the approach across 10 diverse datasets spanning images (classification, VQA), videos (action recognition, anomaly detection), and molecules (property prediction).

4. Experimental Results

The authors benchmarked MPO against leading text-only APO methods (APE, OPRO, EvoPrompt, PE2, ProTeGi, SEE) and manual prompting baselines.

Performance Gains: MPO consistently outperformed all baselines across all modalities.
- Image Tasks: Achieved state-of-the-art results on PlantVillage (76.4% vs. 69.0% for SEE) and CUB-200 (78.6% vs. 71.6%).
- Video Tasks: Improved accuracy on Drive&Act (58.3% vs. 51.7%) and VANE-Bench (71.2% vs. 57.9%).
- Molecule Tasks: Surpassed baselines on Absorption, BBBP, and CYP inhibition tasks (e.g., 64.3% F1 on BBBP vs. 62.3% for SEE).
Efficiency: MPO achieved the same performance as uniform allocation strategies using only 30% of the evaluation budget, demonstrating a 70% reduction in resource cost.
Ablation Studies:
- Alignment: Removing alignment-preserving updates (sequential optimization) significantly degraded performance, proving the necessity of joint updates.
- Operators: Combining Generation, Edit, and Mix operators yielded the best results, confirming their complementary roles.
- Prior Strength: An intermediate prior strength ( $S$ ) was found optimal, balancing the benefit of parent knowledge with the flexibility to adapt to new data.
Generalizability: MPO remained robust across different base models (Qwen, Gemma, InternVL), optimizer models (GPT-4o, Gemini), and lightweight modality-specific generators (SANA1.5).

5. Significance and Conclusion

This paper establishes Multimodal Prompt Optimization as a crucial paradigm shift for unlocking the full potential of MLLMs.

Theoretical Insight: It demonstrates that text-only prompts are suboptimal for MLLMs because they force the model to "hallucinate" or infer visual/structural details that could be directly provided.
Practical Impact: By providing a framework that efficiently searches the multimodal space, MPO enables MLLMs to achieve higher accuracy in complex real-world tasks (e.g., medical diagnosis, drug discovery) with fewer evaluation trials.
Future Direction: The work suggests that future prompt engineering must move beyond linguistic instructions to include dynamic, optimized non-textual contexts, fundamentally changing how we interact with and optimize multimodal AI systems.