💬 NLP

CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning

CMAP introduces a parameter-efficient framework for multi-domain task-incremental learning that leverages CLIP's unexploited text embedding space for robust task routing, confidence estimation, and encoder adaptation, achieving state-of-the-art performance on the MTIL benchmark without external data or task identity information.

Original authors: Sriram Mandalika

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sriram Mandalika

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart art critic who has already read millions of books and seen millions of paintings. This critic (the AI model) knows how to describe a "cat" or a "car" perfectly based on text alone. However, you want to teach this critic to recognize new, specific types of objects (like a specific breed of dog or a rare flower) one by one, without ever showing them the old examples again.

The big problem in this scenario is forgetting. If you teach the critic about "Golden Retrievers," they might forget how to recognize "Siamese cats." If you teach them too aggressively, they might get confused when they see a new type of bird they haven't learned yet.

Current methods try to solve this by looking only at the pictures. They ask, "Does this new photo look like the photos I've seen before?" The paper argues this is like trying to identify a person in a crowd by only looking at their shoes, ignoring their face and voice. It's inefficient and prone to errors, especially when you don't have many photos to learn from.

The authors propose a new system called CMAP (Cross-Modal Adaptive Prompting). Here is how it works, using simple analogies:

1. The "Text-Based ID Card" (Text-Space Task Routing)

The Old Way: When a new photo arrives, the old system tries to match it against a blurry cloud of previous photos it has seen. If the photo is slightly different (like a dog in a different pose), the system gets confused about which "task" (category) it belongs to.
The CMAP Way: Instead of guessing based on the photo's appearance, CMAP looks at the text description of the categories. It asks, "Does this photo match the idea of a 'Golden Retriever' or a 'Siamese Cat'?"

The Analogy: Imagine you have a library of books. Instead of trying to guess which book a person is looking for by the color of their shirt (visual), you ask them, "Which book title sounds like what you are looking for?" (text). Since the book titles (text prototypes) never change, this method is super stable, even if you only have one or two photos to learn from. It costs nothing extra to set up.

2. The "Double-Check System" (Multi-Prototype Confidence)

The Old Way: The system assumes every object looks exactly the same (like a perfect circle). If a photo is a bit weird or blurry, the system gets unsure and might guess wrong.
The CMAP Way: CMAP realizes that a "dog" can look many different ways (running, sleeping, from the side). It creates multiple "mental snapshots" (prototypes) for each category.

The Analogy: Instead of having one single rule for "What is a dog?", the system keeps a folder with three different sketches of dogs: one running, one sleeping, and one sitting. When a new photo comes in, it checks against all three sketches.
The Twist: It doesn't just check the sketches; it also checks the text description. It asks, "Does this photo look like the sketch AND does it match the word 'dog'?" If both agree, the system is very confident. If they disagree, it knows to be careful.

3. The "Synchronized Gatekeepers" (Symmetric Cross-Modal Gating)

The Old Way: The system has two doors: one for the image and one for the text. In the past, if the image door was closed (because the photo was weird or unknown), the text door would still stay wide open. This caused a mismatch, like a translator speaking a language the listener doesn't understand.
The CMAP Way: CMAP installs a linked mechanism. If the image door closes because the photo is confusing, the text door closes at the exact same time.

The Analogy: Imagine a security guard at a museum. If the guard sees a suspicious painting (image), they immediately lock the audio guide (text) so the visitor doesn't get confused by mixed signals. This ensures the "picture" and the "words" stay in perfect harmony, even when the system encounters something it has never seen before.

The Results

The authors tested this on a massive challenge involving 11 different datasets (from cars to flowers to handwritten numbers) with over 1,200 different categories.

Performance: CMAP beat the previous best methods by a significant margin (improving accuracy by about 3 to 5 percentage points).
Efficiency: It did all this while only adding a tiny amount of "brain power" (about 2.5 million adjustable parameters), which is roughly the size of a small app, rather than retraining the whole massive model.
Data Scarcity: It worked especially well when the system was given very few examples (only 16 photos per category) to learn from, proving that using the "text ID cards" is a smarter way to learn than just staring at more photos.

In short: CMAP teaches an AI to learn new tasks by listening to its own internal "text voice" as much as it looks at the pictures. This keeps the AI from forgetting old lessons, helps it handle weird new inputs, and does it all without needing a massive computer or a huge library of past photos.

Technical Summary: CMAP – Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning

1. Problem Definition

The paper addresses Multi-Domain Task-Incremental Learning (MTIL), a setting where a model must sequentially learn tasks from visually diverse domains without access to past data and without knowing the task identity at inference time. This scenario presents two primary challenges:

Catastrophic Forgetting: The tendency of a model to overwrite previously acquired knowledge when adapting to new tasks.
Forward Forgetting: The degradation of the model's ability to generalize to unseen tasks when adapting too aggressively to seen tasks.

While parameter-efficient fine-tuning (PEFT) methods based on frozen Vision-Language Models (VLMs) like CLIP have shown promise, existing approaches suffer from a fundamental limitation: they rely exclusively on visual features for task routing, confidence estimation, and encoder adaptation. This ignores the rich, semantically stable cross-modal text embedding space inherent to CLIP, treating a cross-modal model as if it were a visual-only one.

2. Methodology: CMAP

The authors propose CMAP (Cross-Modal Adaptive Prompting), a framework that integrates cross-modal awareness into every stage of the inference and adaptation process. The method builds upon the IAP baseline (which uses per-layer Hard Gumbel gating and visual Gaussian confidence) by introducing three specific cross-modal mechanisms:

2.1 Text-Space Task Routing

Existing methods route tasks by computing the log-probability of visual features under task-specific visual Gaussians. CMAP replaces this with cosine similarity to frozen CLIP text prototypes.

Mechanism: For each task $t$ , a text prototype $\tau_t$ is computed by averaging the frozen text embeddings of all class names in that task's label set.
Inference: At inference, the visual feature of the input image is compared against all $\tau_t$ via cosine similarity to select the most likely task.
Advantage: This requires no additional learnable parameters, is invariant to the order of task arrival, and remains robust even with scarce training data, as it relies on pre-trained semantic knowledge rather than learned visual statistics.

2.2 Multi-Prototype Visual-Textual Confidence (MPVTC)

Prior work models class confidence using a single visual Gaussian and fixed global thresholds. CMAP introduces a more robust confidence estimation strategy:

Multi-Prototype Visuals: Instead of a single centroid, each class is represented by $K$ visual cluster centroids (via K-means) to handle multi-modal visual distributions (e.g., different object angles).
Cross-Modal Alignment: The confidence score for a class combines the maximum similarity to visual prototypes with the similarity to the class's text embedding.
Task-Adaptive Thresholds: Rather than using fixed global thresholds, CMAP calibrates upper and lower confidence thresholds automatically based on the 80th and 20th percentiles of the training distribution for each specific task.
Output: A dynamic prompting weight is derived from this joint confidence score, determining whether to apply prompts (high confidence), suppress them (low confidence), or apply them proportionally.

2.3 Symmetric Cross-Modal Gating

In the baseline, only the image encoder is gated. If the image gate closes (e.g., on out-of-distribution inputs), the text encoder still receives task-specific prompts, breaking the cross-modal alignment CLIP was trained to maintain.

Mechanism: CMAP extends the Hard Gumbel gating mechanism symmetrically to the text encoder.
Conditioning: The gating decision for the text encoder is conditioned on the mean image features of the batch.
Result: Both the image and text encoders open or close their prompt injection mechanisms simultaneously. This ensures that cross-modal coherence is preserved even when the model encounters out-of-distribution inputs.

3. Key Contributions

The paper identifies three specific gaps in existing MTIL literature and addresses them as follows:

Text-Space Task Routing: Replaces visual Gaussian routing with cosine similarity to frozen text prototypes, achieving order-independent routing with zero additional learnable parameters.
Multi-Prototype Visual-Textual Confidence (MPVTC): Combines K-means visual prototypes with cross-modal alignment scores and task-adaptive thresholds to provide reliable instance-level confidence estimation.
Symmetric Cross-Modal Gating: Extends per-layer gating to the text encoder conditioned on image features, ensuring consistent co-adaptation and co-suppression of both modalities.

4. Experimental Results

The method was evaluated on the MTIL benchmark, spanning 11 datasets and 1,201 classes, under two task orderings (Order-I and Order-II) and a data-scarce 16-shot setting.

Performance (Order-I): CMAP achieved 73.6% Transfer, 81.4% Average, and 88.9% Last. This surpasses the previous state-of-the-art (IAP) by 4.4, 4.6, and 3.2 percentage points, respectively.
Data Efficiency: In the 16-shot setting, the Transfer metric gap over the visual-Gaussian baseline widened to 6.2%, demonstrating that text prototypes scale gracefully where visual statistics degrade due to limited samples.
Parameter Efficiency: The method introduces only 0.09M additional parameters (for text gating), bringing the total trainable parameters to 2.5M. This is significantly lower than full fine-tuning (211M) or MoE-Adapter (59.8M) while outperforming them.
Ablation Studies:
- Removing text-space routing caused the largest drop in Transfer performance.
- Removing symmetric gating caused the largest drops in Average and Last metrics, highlighting its role in knowledge retention.
- Combining visual and textual confidence signals yielded better results than using either modality alone, confirming they capture complementary information.

5. Significance and Claims

The paper claims that the primary source of untapped potential in current MTIL methods is the failure to utilize the cross-modal text embedding space of pre-trained VLMs. By treating CLIP as a visual-only model, existing methods miss the semantic stability and zero-shot generalization capabilities inherent in the text encoder.

The significance of CMAP lies in demonstrating that exploiting pre-trained cross-modal alignment is a more parameter-efficient adaptation strategy than simply scaling model capacity or relying on complex memory buffers. The authors assert that their approach achieves state-of-the-art performance across all metrics with minimal trainable parameters and no external data, proving that cross-modal signals are critical for robust, incremental learning in data-scarce and diverse domain settings.