FewMMBench: A Benchmark for Multimodal Few-Shot Learning

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in handling interleaved image-text data, their few-shot learning capabilities remain under-explored and poorly understood. Existing benchmarks primarily focus on zero-shot evaluation or isolated reasoning skills, failing to systematically test MLLMs in In-Context Learning (ICL) settings with interleaved image-text streams and Chain-of-Thought (CoT) prompting.

Key open questions include:

• Do instruction-tuned models benefit from few-shot demonstrations, or do they rely solely on pre-training?
• Does retrieving semantically similar examples improve performance over random selection?
• Does CoT prompting enhance multimodal reasoning, or does it introduce noise and hallucinations in visual tasks?

2. Methodology: FEWMMBENCH

The authors introduce FEWMMBENCH, a comprehensive benchmark designed to evaluate MLLMs under controlled few-shot conditions.

Dataset Curation

• Task Diversity: The benchmark covers 9 distinct tasks ranging from low-level perception to high-level reasoning:
  1. Attribute Recognition
  2. Visual Object Recognition
  3. Plurality Recognition (Singular vs. Plural)
  4. Object Counting
  5. Spatial Relations Understanding
  6. Action Recognition
  7. Commonsense Reasoning
  8. Temporal Reasoning (video/sequence understanding)
  9. Coreference Resolution
• Data Construction:
  • Selection Strategy: Instead of random clustering, the authors use a Graph Cut-based submodular selection strategy. This algorithm balances relevance (semantic similarity to the query) and diversity (minimizing redundancy) to construct representative support sets.
  • CoT Generation: Since source datasets lack reasoning traces, CoT rationales were generated using Qwen2.5-VL-7B-Instruct. An iterative correction process was employed: if the model's initial prediction was incorrect, the ground truth was injected, and the rationale was regenerated to ensure factual accuracy.
  • Validation: Human annotators validated a subset of CoT examples, achieving a mean reasoning quality score of 4.01/5 and 82.2% answer correctness.

Experimental Setup

• Models Evaluated: 26 open-weight MLLMs from 6 major families (Qwen, InternVL, Idefics, Phi, LLaVA, xGen-MM), ranging from 0.5B to 9B parameters. Both instruction-tuned and non-instruction-tuned variants were tested.
• Evaluation Settings:
  1. Zero-Shot: No examples provided.
  2. Random Few-Shot: 4 or 8 examples selected randomly.
  3. Similar Few-Shot: 4 or 8 examples selected via retrieval based on visual/textual similarity.
  4. Similar + CoT: Similar examples augmented with Chain-of-Thought reasoning.
• Metrics: Primary metric is Accuracy. To mitigate format bias, a perplexity-based pairwise ranking method was also used, where the model selects the option with the lowest perplexity. Uncertainty quantification was performed via probability distribution analysis.

3. Key Contributions

1. Controlled Framework: A unified benchmark supporting both random and semantically similar demonstration selection, utilizing a graph-cut algorithm for optimal support set construction.
2. Explicit CoT Rationales: The first large-scale multimodal benchmark providing detailed, verified CoT traces for few-shot analysis, enabling the study of reasoning-guided inference.
3. Robust Evaluation Protocol: Extends beyond simple accuracy to include perplexity-based pairwise comparisons and uncertainty quantification, addressing order-sensitivity issues in ICL.
4. Diagnostic Insights: Reveals specific failure modes of current MLLMs in few-shot and CoT settings, challenging the assumption that "more examples" or "reasoning steps" always help.

4. Key Results and Findings

The evaluation of 26 models yielded four critical findings:

• Finding 1: Limited Gains for Instruction-Tuned Models.

  • Instruction-tuned models (e.g., Qwen2.5-VL, Phi-4) already exhibit strong zero-shot performance.
  • Adding few-shot demonstrations (random or similar) yields marginal or negative gains. In many cases, performance regresses slightly, suggesting these models have internalized task strategies and do not rely on in-context examples.
  • Conversely, non-instruction-tuned (base) models show significant improvements (often double-digit) with few-shot prompting.

• Finding 2: Retrieval-Based Selection Offers No Consistent Advantage.

  • Using semantically similar examples (retrieved via Graph Cut) did not consistently outperform random selection.
  • This suggests current MLLMs struggle to leverage fine-grained distributional cues from retrieved examples, or their generation is driven more by the query than the context.

• Finding 3: More Demonstrations Do Not Guarantee Better Performance.

  • Increasing the shot count from 4 to 8 often resulted in stagnation or performance drops.
  • This indicates context dilution, where additional examples introduce noise or distract the model from the core task, particularly for tasks requiring precise visual grounding.

• Finding 4: CoT Prompting Degrades Performance.

  • Contrary to text-only LLMs, CoT prompting consistently reduced accuracy across all tested MLLMs and task types.
  • Failure Modes: CoT often led to hallucinated visual details, irrelevant elaboration, or "reasoning loops" that distracted from the visual grounding. The models struggled to maintain coherent step-by-step reasoning in multimodal contexts.

5. Significance and Conclusion

FEWMMBENCH serves as a rigorous diagnostic tool that exposes a critical gap in current MLLM capabilities: the inability to effectively generalize from few-shot examples or utilize CoT reasoning in multimodal settings.

• Implications: The results suggest that the "scaling laws" and prompting strategies successful in text-only domains do not directly transfer to multimodal tasks. Instruction tuning appears to saturate the model's ability to learn from context, while current CoT strategies are misaligned with the needs of visual reasoning.
• Future Directions: The paper calls for new architectural or training approaches that specifically address:
  • Modality-aware CoT prompting.
  • Better mechanisms for leveraging semantic similarity in few-shot settings.
  • Improved visual grounding to prevent hallucinations during reasoning.

The benchmark and data are publicly available, providing a standardized testbed for diagnosing and advancing the few-shot learning capabilities of future multimodal models.