DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

1. Problem Statement

While Multi-Modal Large Language Models (MLLMs) have demonstrated strong zero-shot capabilities in vision-language tasks (e.g., image captioning, VQA), they struggle with Few-Shot Object Detection (FSOD) on out-of-distribution (OOD) classes, tasks, and imaging modalities (e.g., X-rays, thermal, aerial imagery) not present in their pre-training data.

Key challenges identified by the authors include:

• Ineffective In-Context Learning (ICL): Contrary to expectations, providing MLLMs with few-shot visual examples alongside text prompts often degrades detection accuracy compared to using class names and text instructions alone. The authors attribute this to rigid post-training prompt structures that prevent models from effectively leveraging additional visual context.
• Access Limitations: Frontier MLLMs (e.g., Gemini 3 Pro) are typically accessible only via APIs (black-box), while state-of-the-art open-weights models are too expensive to fine-tune on consumer hardware. This necessitates gradient-free optimization strategies.
• Confidence Calibration: MLLMs often over-predict bounding boxes and lack reliable per-box confidence scores, leading to high false positive rates.

2. Methodology: Detection Prompt Optimization (DetPO)

The authors propose DetPO, a black-box, gradient-free, test-time optimization framework that refines text-only prompts to improve object detection performance without modifying model weights.

Core Components:

1. Contrastive Prompt Refinement:

   • Instead of feeding raw images into the prompt, DetPO uses few-shot examples to iteratively generate and refine natural language descriptions of the target class.
   • Positive Summarization: The model analyzes ground-truth examples to generate an initial class definition.
   • Negative Contrast: The model is shown examples of other classes (negative examples) and asked to explicitly distinguish the target class from them, focusing on discriminative features.
   • Error-Driven Iteration: The system runs inference on the few-shot training set. It identifies the most severe False Positives (FP) and False Negatives (FN).
     • For FPs: The prompt is updated to explicitly exclude the visual features of the false positive.
     • For FNs: The prompt is updated to explicitly include the visual features of the missed instance.
   • This loop continues until performance converges or a maximum iteration limit is reached.

2. Confidence Score Estimation:

   • MLLMs do not natively output calibrated confidence scores. DetPO addresses this in two ways:
     • Self-Reported Scores: Prompting the model to output a confidence score per box, which already improves performance over a default score of 1.0.
     • VQA Score (Optional Post-Processing): To further calibrate confidence, the system uses a Visual Question Answering (VQA) approach. For each predicted bounding box, the model is asked, "Is there an instance of class {CLS} inside this bounding box?" The normalized probability of the "Yes" token is used as the final confidence score. This effectively down-weights false positives.

3. Key Contributions

• Empirical Finding: The paper demonstrates that naive multi-modal in-context learning (showing images to the model) hurts detection accuracy on OOD tasks, whereas optimizing the textual description of the class yields significant gains.
• DetPO Framework: Introduction of a novel, gradient-free prompt optimization technique that iteratively refines class definitions based on detection errors (FP/FN) and contrastive learning.
• Confidence Calibration: A simple yet effective method to improve detection precision by using VQA-based scoring to recalibrate bounding box confidence, significantly reducing false positives.
• State-of-the-Art Performance: Establishing a new benchmark for black-box few-shot object detection, outperforming both specialist detectors and prior prompt optimization methods.

4. Experimental Results

The method was evaluated on Roboflow20-VL (20 diverse domains including medical, aerial, and industrial) and LVIS Rare 50.

• Performance Gains:
  • Gemini 3 Pro + DetPO: Achieved 26.4 mAP, outperforming the best specialist detector (LLMDet at 17.2 mAP) by 9.2%.
  • Qwen3-VL (30B-A3B) + DetPO: Achieved 21.6 mAP, a 9.7% improvement over its baseline.
  • Comparison to Other Optimizers: DetPO significantly outperformed general-purpose prompt optimizers like GEPA and MIPROv2, which often failed to improve or even degraded performance on this specific task.
• Error Analysis:
  • DetPO drastically reduced class confusion (e.g., distinguishing "Juvenile" vs. "Adult" boars).
  • It mitigated aggressive overconfidence in defect detection (e.g., misclassifying non-defective parts as defective).
  • While VQA scoring slightly increased false negatives (missed detections) due to stricter filtering, the overall reduction in false positives led to a net gain in mAP.
• Efficiency: DetPO is more token-efficient than GEPA, reducing total token usage by 81% and execution time by 17%.

5. Significance and Conclusion

• Bridging the Gap: DetPO successfully bridges the gap between zero-shot generalization and task-specific adaptation without requiring gradient-based fine-tuning. This is crucial for closed-source models where weights cannot be accessed.
• Practicality: It offers a viable alternative to fine-tuning for scenarios involving proprietary APIs or expensive open-weight models, enabling generalist MLLMs to adapt to novel classes and modalities encountered in real-world applications.
• Future Outlook: The authors note that while current black-box prompting still trails behind "white-box" fine-tuning of specialist models, the rapid performance gains of frontier closed-source models (like Gemini 3 Pro) suggest that DetPO applied to next-generation MLLMs may eventually surpass fine-tuned specialists.

In summary, DetPO redefines few-shot object detection for MLLMs by shifting the focus from feeding raw visual examples to the model, to optimizing the textual instructions that guide the model's visual reasoning, resulting in robust, high-accuracy detection across diverse and challenging domains.