Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games

1. Problem Definition

The paper addresses the challenge of open-ended, multi-granularity object recognition in the context of AI-powered educational tools (specifically interactive educational cameras). The core problem is balancing two conflicting requirements:

• Open-Endedness: The system must recognize a vast, undefined vocabulary of objects encountered in real-world environments (homes, parks, classrooms) without a fixed category set.
• Multi-Granularity: Different objects require different levels of detail.
  • General Objects: Common items (e.g., backpacks, LEGO bricks) only need coarse-grained recognition.
  • Specialized Objects: Educational subjects like animals and plants require fine-grained recognition (e.g., distinguishing a "Reticulated Giraffe" from a generic "Giraffe") to provide meaningful learning value.

Current Limitations:

• Multimodal Large Language Models (MLLMs): Excel at open-ended, coarse-grained recognition but struggle with fine-grained distinctions (e.g., specific species).
• CLIP-style Models: Excel at fine-grained recognition within specific domains but lack broad coverage for general open-world objects and are not inherently open-ended.
• Fine-Tuning Trade-off: Naively fine-tuning an MLLM on fine-grained datasets improves performance on that specific domain but significantly degrades its generalization capabilities on out-of-distribution (OOD) data.

2. Methodology: The HyMOR Framework

The authors propose HyMOR (Hybrid Multi-granularity open-ended Object Recognition), a framework that integrates an MLLM and a CLIP model to leverage the strengths of both.

Architecture & Workflow

1. MLLM as Router and Coarse Recognizer:

   • The MLLM (e.g., Qwen2.5-VL or Gemma-3) processes the input image.
   • It performs two tasks:
     1. Routing: Classifies the object into a general category: plant, animal, or other.
     2. Coarse Recognition: Generates a coarse object name (e.g., "Dog", "Flower").
   • Crucial Design Choice: The MLLM is frozen (not fine-tuned on fine-grained data) to preserve its broad generalization and open-ended capabilities.

2. Conditional Fine-Grained Retrieval (CLIP):

   • Trigger: If the MLLM classifies the object as plant or animal, the system activates the fine-grained pipeline. If the object is other, the system outputs the MLLM's coarse label directly.
   • Feature Extraction: A CLIP vision encoder (SigLIP) extracts image embeddings.
   • Vector Database Retrieval: The system compares the image embedding against a pre-computed vector database containing centroids of thousands of specific animal and plant species (derived from FGVR datasets and the TreeOfLife-10M dataset).
   • Decision Logic:
     • If the similarity score between the image and the nearest centroid exceeds a predefined threshold, the system outputs the fine-grained label (e.g., "Reticulated Giraffe").
     • If the score is below the threshold, it falls back to the coarse label generated by the MLLM.

3. Key Contributions

1. HyMOR Framework: A novel hybrid architecture that successfully bridges the gap between the open-ended world knowledge of MLLMs and the precise visual discrimination of CLIP models. It achieves this without fine-tuning the MLLM, thereby avoiding the "catastrophic forgetting" of general capabilities.
2. TBO Dataset (TextBook Objects): The authors introduce a new, education-oriented benchmark dataset to fill the gap in existing object recognition benchmarks.
   • Source: Extracted from 2,623 PDF textbooks (primary to high school) from a national educational platform.
   • Scale: Contains 20,942 images annotated with 8,816 unique object names.
   • Purpose: Specifically designed to evaluate models on objects relevant to educational curricula.
3. Comprehensive Evaluation: Extensive experiments demonstrating that HyMOR outperforms standalone MLLMs and CLIP models across both fine-grained and coarse-grained tasks.

4. Experimental Results

The evaluation utilized standard Fine-Grained Visual Recognition (FGVR) datasets (Dog-120, Bird-200, Flower-102, Pet-37) and coarse-grained datasets (ImageNet-1K, ObjectNet-313, TBO-8K). Metrics included Exact Match (EM) and Sentence-BERT (SBert) similarity.

• Fine-Grained Performance:

  • HyMOR significantly closes the gap with specialized CLIP models.
  • HyMOR-Gemma achieved an average SBert of 95.7 on FGVR datasets, narrowing the performance gap to the best-performing CLIP variant (SigLIP-FT) to just 0.2%.
  • This is a massive improvement over the baseline MLLM (Gemma-3-4B), which scored 56.9 SBert.

• Coarse-Grained Performance:

  • HyMOR maintained and slightly improved upon the baseline MLLM's performance on general objects.
  • HyMOR-Gemma improved the average SBert score by 2.5% over the Gemma-3-4B baseline on general datasets (ImageNet, ObjectNet, TBO).
  • In contrast, pure CLIP models (even fine-tuned ones) performed poorly on coarse-grained tasks (e.g., <1% EM on ImageNet-1K) due to lack of general category coverage.

• Overall Impact:

  • HyMOR-Gemma achieved a 23.2% improvement in average SBert score across all evaluated datasets compared to the baseline MLLM.
  • Routing Accuracy: The MLLM router correctly identified specialized vs. general objects with 99.8% accuracy on FGVR data and 95.6% on general data.

5. Significance and Applications

• Educational AI: The framework provides a robust perceptual foundation for "AI Cameras" used in education. It allows children to interact with their environment, receiving accurate feedback on specific species (e.g., identifying a specific flower) while still understanding general objects (e.g., a chair).
• Closed-Loop Learning: The system supports gamified learning (e.g., earning virtual coins for identifying objects to nurture a virtual pet), driving engagement through accurate, multi-granularity recognition.
• Efficiency: By freezing the MLLM and only fine-tuning the CLIP component for retrieval, the system remains lightweight and suitable for edge deployment while avoiding the computational cost of training massive MLLMs on specific fine-grained datasets.
• Future Directions: The authors suggest extending the framework to include object grounding/localization and multi-object scene understanding to further enhance interactive game and learning environments.

In summary, HyMOR effectively solves the "open-ended vs. fine-grained" dilemma by using an MLLM as a smart router that selectively engages a specialized CLIP retrieval system only when high-precision biological identification is required.