Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation

1. Problem Statement

While Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in general multimodal perception and reasoning (e.g., GPT-4o, InternVL, Qwen), their performance on fine-grained image tasks remains largely unexplored. Fine-grained tasks require distinguishing between subordinate categories (e.g., specific bird species, car models, or plant varieties) that differ only by subtle visual attributes.

• The Gap: Existing benchmarks (like MMBench or LVLM-eHub) focus on holistic or specialized tasks (OCR, charts) but lack systematic evaluation of LVLMs on fine-grained discrimination.
• The Challenge: It is unclear whether LVLMs can effectively leverage their visual encoders and language bases to handle the subtle discriminative patterns required for fine-grained recognition, or if they suffer from knowledge biases, poor feature alignment, and vulnerability to perturbations in these specific domains.

2. Methodology: The FG-BMK Benchmark

To address this gap, the authors introduce FG-BMK, a comprehensive benchmark comprising 1.01 million questions and 0.28 million images sourced from 12 established fine-grained datasets (e.g., CUB-200-2011, Stanford Dogs, Food-101, iNat2021).

The evaluation is conducted through two distinct paradigms:

A. Human-Oriented Evaluation

This paradigm assesses the model's ability to engage in dialogue and answer queries about visual content. It includes:

• Attribute Recognition: True/False and Multiple-Choice questions regarding visual attributes (color, pattern, shape, length, size).
• Knowledge Bias Estimation: True/False questions designed to detect if models perform significantly better on certain categories due to training data imbalances.
• Hierarchical Granularity Recognition: Questions at varying levels of taxonomy (Class, Genus, Species) to test the model's ability to reason at different levels of detail.

B. Machine-Oriented Evaluation

This paradigm directly measures the quality of the visual feature representations extracted by the LVLMs, independent of the language generation component.

• Image Retrieval: Measuring the similarity of visual features to retrieve images of the same subordinate category (Metric: mAP).
• Image Classification: Recognizing images into fine-grained categories, both within a single meta-category and across multiple meta-categories (Metric: Top-1 Accuracy).
• Robustness Testing: Introducing feature perturbations using Projected Gradient Descent (PGD) to test stability.

3. Key Contributions & Experimental Setup

• Scale: The benchmark covers 12 diverse domains (Birds, Cars, Dogs, Aircraft, Plants, Products, etc.) with 12 evaluated models, including 9 open-source LVLMs (e.g., InternVL3, LLaVA-1.5, Qwen2.5-VL), 2 closed-source models (GPT-4o, Gemini-2.0-flash), and a pure vision model (DINOv2).
• Controlled Variables: The study isolates factors such as training paradigms (Contrastive vs. Generative vs. Reconstruction), vision encoder sizes, and the impact of image-text alignment granularity.

4. Key Results & Findings

A. Training Paradigms Matter

• Contrastive Learning Wins: Models trained with contrastive paradigms (e.g., EVA-CLIP, InternVL, DINOv2) significantly outperform those using generative or reconstruction-based paradigms (e.g., Qwen, BEiT3) in fine-grained discrimination.
• Generative Limitations: Generative and reconstruction losses tend to yield weaker discriminability for fine-grained features, as they prioritize semantic coherence over subtle visual distinctions.

B. The "Alignment Penalty"

• Granularity Mismatch: Aligning visual features with textual features often impairs fine-grained discriminability. This is attributed to the mismatch in granularity between image-text pairs in training data (e.g., a specific bird species image paired with a coarse "bird" description).
• Evidence: When visual features were aligned to fine-grained text (retrained), performance improved, but original (un-aligned) visual features often remained superior for classification tasks.

C. Knowledge Bias and Data Imbalance

• Inconsistent Performance: LVLMs exhibit high variance in accuracy across different fine-grained categories (e.g., 90% accuracy on some species vs. 30% on others).
• Root Cause: This bias is primarily caused by uneven representation of categories in the training data rather than the inherent difficulty of the visual task. Fine-tuning on balanced datasets significantly reduces this variance.

D. Robustness and Perturbations

• Vulnerability: LVLMs are significantly more susceptible to feature perturbations in fine-grained tasks compared to generic tasks (e.g., accuracy on CUB-200-2011 dropped from ~89% to ~25% under perturbation, whereas CIFAR-100 dropped less severely).
• Comparison: Pure vision models trained with cross-entropy on high-quality data (like ImageNet) show superior robustness compared to current LVLMs.

E. Reasoning vs. Perception

• Perception: LVLMs are strong at recognizing visual appearances (e.g., "Is the bird red?").
• Reasoning: They struggle with fine-grained category reasoning that requires combining visual attributes with domain knowledge (e.g., distinguishing between two very similar species based on subtle attribute combinations).
• Gap: Despite advancements, LVLMs still lag behind specialized fine-grained tailored models (e.g., CAP, SR-GNN) in pure classification tasks.

5. Significance and Future Directions

• Diagnostic Tool: FG-BMK provides a critical diagnostic framework to identify specific weaknesses in current LVLM architectures, particularly regarding feature alignment and training data quality.
• Design Guidance:
  • Data Construction: Future training data must ensure granularity alignment between images and text to preserve visual discriminability.
  • Training Objectives: Incorporating contrastive learning objectives during the alignment stage is crucial for maintaining fine-grained feature quality.
  • Hybrid Approaches: There is a need to develop models that retain the general reasoning capabilities of LVLMs while integrating specialized fine-grained recognition modules.
• Open Source: The authors have open-sourced the code, data splits, and evaluation scripts to facilitate further research in this domain.

In conclusion, the paper argues that while LVLMs are powerful generalists, they are not yet "fine-grained experts." Their performance is currently bottlenecked by training data granularity mismatches and the limitations of generative training paradigms in preserving subtle visual distinctions.