Beyond Prompt Degradation: Prototype-guided Dual-pool Prompting for Incremental Object Detection

This paper proposes PDP, a novel prompt-decoupled framework for Incremental Object Detection that utilizes a dual-pool prompting paradigm to separate task-general and task-specific knowledge while employing a prototypical pseudo-label generation module to mitigate prompt drift, thereby achieving state-of-the-art performance on MS-COCO and PASCAL VOC benchmarks.

The Gist

Imagine you are a detective trying to learn how to identify new types of suspects (objects) while remembering all the old ones you've already caught. This is the challenge of Incremental Object Detection (IOD).

The problem is that as you learn new tricks, you often start forgetting the old ones, or you get confused and mix up your old rules with new ones. This is called "catastrophic forgetting."

Recently, scientists tried solving this using "Prompts." Think of a prompt like a sticky note you stick on your detective's hat. It tells your brain, "Hey, we are looking for dogs right now," or "We are looking for cars."

However, the old way of using these sticky notes had two big flaws, which this paper calls Prompt Degradation:

1. The "Crowded Desk" Problem (Prompt Coupling): Imagine you have one single, tiny desk where you throw every sticky note you've ever made. As you add more notes for new suspects, they start crumpling, overlapping, and smudging the old notes. Your brain gets confused because the "Dog" note is now stuck to the "Cat" note.
2. The "False Accusation" Problem (Prompt Drift): In this detective game, when you learn about a new suspect (say, a "Robot"), the computer tells you to treat all the old suspects (like "Dogs") as just "background noise" (like a tree or a rock). This confuses your brain. It starts thinking, "Wait, I thought Dogs were important? Now I'm told they are just background?" So, your "Dog" sticky note gets rewritten and drifts away from the truth.

The Solution: PDP (Prototype-guided Dual-pool Prompting)

The authors of this paper built a new system called PDP to fix these messes. They used two clever ideas:

1. The "Two-Drawer Desk" (Dual-Pool Decoupling)

Instead of one messy desk, they gave the detective two separate drawers:

• The Shared Drawer (The General Knowledge): This drawer holds sticky notes for things that are true for everyone. For example, "Look for edges," "Look for colors," or "Look for movement." These are general skills that help you spot any object. This drawer is constantly updated to get better at these general skills.
• The Private Drawer (The Specific Knowledge): This drawer is for the specific suspects. When you learn about "Dogs," you get a private, locked drawer just for Dogs. When you learn about "Cars," you get a separate, locked drawer for Cars.

Why this works: The "Dog" notes never get mixed with the "Car" notes. The general skills help you find the object, and the private notes help you identify exactly what it is. They work together but don't get in each other's way.

2. The "Mugshot Album" (Prototype-Guided Pseudo-Labeling)

Remember the "False Accusation" problem? When the computer says, "Treat the old Dog as background," the detective gets confused.

To fix this, PDP creates a Mugshot Album (called a Prototype Space).

• Every time the detective successfully identifies a "Dog," they take a perfect photo of it and put it in the album. This photo represents the "true essence" of a Dog.
• Later, when the computer says, "That looks like background," the detective checks the album. "Wait," the detective says, "This blurry shape looks 90% like the Dog in my album. Even though the computer says it's background, I know it's a Dog!"
• The system then creates a "Pseudo-Label" (a fake but reliable note) saying, "Actually, this is a Dog," and uses that to teach the model.

Why this works: It stops the detective from forgetting what a Dog looks like just because the computer told them to ignore it. It uses the "Mugshot" to keep the memory alive.

The Results

The paper tested this system on famous object detection datasets (like MS-COCO and PASCAL VOC).

• The Result: The new system (PDP) became the best in the world (State-of-the-Art).
• The Improvement: It improved detection accuracy by nearly 10% on some tests.
• The Takeaway: By separating general skills from specific memories (the Two-Drawer Desk) and using a "Mugshot Album" to double-check the truth (the Prototype), the AI can learn new things without forgetting the old ones.

In short: PDP is like giving a detective a better filing system and a reliable photo album, so they can solve new cases without losing their memory of the old ones.

Technical Summary

1. Problem Statement

Incremental Object Detection (IOD) aims to continuously learn new object categories from sequential data streams while retaining the ability to detect previously learned classes without access to past data. This task faces the stability-plasticity dilemma: the model must be plastic enough to learn new concepts but stable enough to avoid catastrophic forgetting.

While recent prompt-based methods have gained popularity for their parameter efficiency and replay-free nature, the authors identify that they suffer from Prompt Degradation, which manifests in two specific ways:

1. Prompt Coupling: Existing methods typically use a single prompt pool to store both task-general (transferable) and task-specific (discriminative) prompts. This forces prompts with different functional roles to compete for limited parameter space, leading to interference and degraded performance.
2. Prompt Drift: In IOD, when learning a new task, objects from old classes are often relabeled as "background" because they lack annotations in the current dataset. This inconsistent supervision forces the model to update prompts optimized for old tasks, causing them to drift toward incorrect semantic directions. Furthermore, existing pseudo-labeling methods rely on static confidence thresholds, which fail to adapt to the varying confidence distributions across different classes, exacerbating the drift.

2. Methodology: The PDP Framework

The authors propose PDP (Prototype-guided Dual-pool Prompting), a fully end-to-end trainable framework designed to decouple prompts and maintain supervisory consistency. PDP consists of two core modules:

A. Decoupled Dual-Pool Prompting (DDP)

To solve prompt coupling, PDP introduces a dual-pool architecture that explicitly separates prompts into two distinct pools:

• Shared Pool: Contains learnable prompt tokens designed to capture task-general knowledge. This pool is accessible to all tasks and is continuously refined to facilitate stable forward knowledge transfer.
• Private Pool: Contains prompt parameters specific to the current task. These are privatized (frozen for previous tasks) to preserve task-specific discriminative features, preventing interference from new tasks.
• Inter-Pool Diversity Constraint: To ensure the two pools learn complementary and orthogonal representations, a directional decoupled loss ($L_{DDL}$) is applied. This loss maximizes the angular separation between vectors in the shared and private pools, penalizing any overlap.

B. Prototypical Pseudo-Label Generation (PPG)

To solve prompt drift caused by inconsistent supervision, PDP replaces static confidence thresholds with a dynamic, prototype-guided approach:

• Class Prototype Space: The model maintains a prototype (feature anchor) for each learned class in the embedding space. These prototypes are updated only at the end of each task to ensure stability.
• Hierarchical Validation:
  1. Easy Samples: High-confidence predictions from the teacher model are accepted directly.
  2. Hard Samples: For low-to-mid confidence candidates, the model computes the similarity between the candidate's feature and the frozen class prototype. If the similarity exceeds a threshold, the sample is retained as a valid pseudo-label, even if its raw confidence is low.
• Benefit: This mechanism ensures consistent supervision for old classes (preventing them from being treated purely as background) and generates high-quality pseudo-labels without relying on ill-calibrated fixed thresholds.

3. Key Contributions

1. Dual-Pool Decoupling: The first framework for IOD to explicitly decouple task-general and task-specific prompts using a shared and private pool paradigm, significantly improving the stability-plasticity balance.
2. Prototype-Guided Pseudo-Labeling: A novel PPG module that utilizes class prototypes in the embedding space to generate reliable pseudo-labels, effectively mitigating prompt drift caused by inconsistent supervision.
3. State-of-the-Art Performance: The method achieves superior results on standard benchmarks, demonstrating robustness against catastrophic forgetting while maintaining high adaptability to new classes.

4. Experimental Results

The authors evaluated PDP on MS-COCO and PASCAL VOC datasets under various incremental settings (e.g., multi-step, 40+40, 70+10).

• MS-COCO (Multi-step IOD):
  • PDP achieved a final mAP@A of 59.4%, representing a 9.2% improvement over the previous best method (MD-DETR).
  • It showed the lowest forgetting rate (highest mAP@P) and strong plasticity (mAP@C).
• PASCAL VOC:
  • Across three settings (10+10, 15+5, 19+1), PDP consistently outperformed all competitors.
  • Notable improvements included +3.3% mAP@A in the 19+1 setting compared to the second-best method.
• Ablation Studies:
  • Removing the Private Pool led to significant forgetting.
  • Removing the Shared Pool reduced the model's ability to transfer general knowledge.
  • Replacing PPG with fixed-threshold pseudo-labeling resulted in a drop of 3.3% mAP@A in the final task, confirming the necessity of prototype-guided supervision.

5. Significance

This work addresses a fundamental limitation in current prompt-based continual learning: the conflation of general and specific knowledge leading to degradation. By decoupling these knowledge types and introducing a semantic-stable pseudo-labeling mechanism, PDP provides a robust solution for open-world object detection. It demonstrates that explicit structural design (dual pools) combined with semantic guidance (prototypes) is more effective than relying solely on parameter efficiency or static thresholds. The code and datasets have been released to facilitate further research in incremental learning.