Technical Summary: Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent

1. Problem Statement

Image clustering, the task of grouping visually similar images, faces a fundamental fragmentation in current methodologies. Existing approaches are typically optimized for specific scenarios, creating gaps that prevent universal application:

• Task Specificity: Methods optimized for general object categorization often fail at fine-grained distinctions (e.g., distinguishing bird species vs. car models).
• Distribution Bias: Algorithms designed for balanced distributions struggle with long-tail data where many clusters contain very few samples.
• Criteria Rigidity: Techniques for single-perspective clustering cannot handle multiple simultaneous criteria (e.g., grouping by both color and species), nor can they easily adapt to abstract or composite textual guidelines without retraining.
• Training Overhead: Most deep clustering methods require task-specific training, limiting their flexibility in dynamic, real-world environments where guidelines change.

The authors identify a critical need for a universal framework that can accept natural language guidelines to define clustering criteria, operate without task-specific training, and handle diverse scenarios ranging from general to fine-grained, and balanced to long-tail distributions.

2. Methodology

The proposed framework, the Guideline-Driven Image Clustering Agent, operates as a training-free hybrid system comprising two main stages: Generative Concept Proxy Modeling (GCPM) and MST-based LLM Traversal.

2.1. Generative Concept Proxy Modeling (GCPM)

To incorporate complex textual guidelines without fine-tuning encoders, the authors propose GCPM. This approach addresses the challenge that direct encoding of images with guidelines often leads to attribute overshadowing (where visually dominant but irrelevant features mask the guideline's specific attributes).

• Concept Proxy Extraction: A Multimodal Large Language Model (MLLM) acts as a captioning model ($f_{caption}$) to extract textual descriptions of images that explicitly focus on the attributes specified in the guideline ($A \subseteq G$). This process disentangles attributes, surfacing them in text form.
• Instruction-Aware Embedding: These generated "concept proxy captions" are then encoded using instruction-aware embedding models ($f_{embed}$) alongside the guideline text. This creates embeddings ($h_i$) that are semantically aligned with the specific clustering criteria.
• Flexibility: This method supports both comprehensive clustering (considering all attributes) and dimension-specific clustering (focusing on specific attributes) by adjusting the input to the embedding model.

2.2. MST-based LLM Traversal

While GCPM generates guideline-aware embeddings suitable for standard algorithms, the authors observe that density-based algorithms like HDBSCAN often produce consistent small clusters but fail to merge homogeneous groups in complex or long-tail scenarios. Conversely, using Large Language Models (LLMs) for all pairwise comparisons is computationally prohibitive ($O(M^2)$).

To balance semantic reasoning with efficiency, the authors introduce MST-based LLM Traversal:

1. Initial Clustering: HDBSCAN is applied to the GCPM embeddings to generate an initial set of clusters.
2. Distance Computation: Ward distance is computed between all cluster pairs based on their centroids.
3. MST Construction: A Minimum Spanning Tree (MST) is constructed from the distance matrix. This orders cluster pairs by similarity, prioritizing the most promising merge candidates.
4. Selective LLM Reasoning: Instead of querying the LLM for all pairs, the algorithm iteratively traverses the MST. For each edge (cluster pair), the LLM is queried to decide whether to merge them based on the guideline.
5. Efficiency Optimization:
   • The MST structure ensures that only the most likely merges are evaluated first.
   • The number of LLM calls is reduced from $O(M^2)$ to an expected $O(M \log M)$.
   • Decisions are cached to avoid redundant queries for previously evaluated pairs.

2.3. Guideline Handling

The framework accepts explicit textual guidelines (e.g., "group by brand and intended activity"). In scenarios where explicit guidelines are unavailable, the system employs heuristic prompts to leverage the LLM's internal knowledge for automatic guideline generation or uses unsupervised data substructure exploration to identify fundamental criteria.

3. Key Contributions

The paper outlines four primary contributions:

1. Universal Framework: The first clustering framework capable of handling composite and abstract textual guidelines without task-specific training, unifying general, fine-grained, multiple, and long-tail clustering tasks.
2. Generative Concept Proxy Modeling (GCPM): A training-free embedding approach that uses MLLMs to extract concept proxies, effectively disentangling attributes and incorporating guidelines into embeddings where direct multimodal encoding fails.
3. MST-based LLM Traversal: A novel algorithm that significantly minimizes expensive LLM invocations by intelligently selecting which clustering decisions require deep semantic reasoning, reducing complexity from quadratic to near-linear relative to cluster counts.
4. Comprehensive Evaluation: Extensive experiments across four distinct clustering tasks demonstrating superior generalization and performance compared to specialized, training-based methods.

4. Experimental Results

The framework was evaluated on a wide range of datasets including CIFAR-10, STL-10, ImageNet-10 (General Clustering), Fruit/Cards/CIFAR10-MC (Multiple Clustering), CUB Birds/Stanford Dogs/Cars/Oxford Flowers (Fine-Grained Clustering), and a proposed ABO-LC (Long-Tail E-commerce) dataset.

• General Clustering (GC): On ImageNet-10, the GCPM-G (MLLM-based) variant with K-Means achieved 98.8% accuracy, surpassing the previous best training-based method (IDCTCL) by 1.6%.
• Multiple Clustering (MC): The method achieved 99.9% NMI on the Fruit dataset, significantly outperforming Multi-Sub (98.5%).
• Fine-Grained Clustering (FC): While improvements were more modest due to the difficulty of subtle distinctions, the method achieved state-of-the-art results (e.g., 89.9% NMI on CUB Birds) without training.
• Long-Tail Clustering (LC): On the ABO-LC dataset (78.7% of clusters have $\le$ 2 samples), HDBSCAN alone struggled. However, adding MST Traversal improved the Adjusted Rand Index (ARI) from 23.5% to 37.9% (for GCPM-G), demonstrating effectiveness in real-world, imbalanced distributions.
• Efficiency: The MST Traversal reduced LLM calls significantly compared to naive pairwise comparison. For example, on ImageNet-10, the ratio of LLM calls to samples was 0.86, whereas naive approaches would require orders of magnitude more calls.

Embedding Model Analysis: The study found that MLLM-based embeddings (GCPM-G) generally outperformed LLM-based (GCPM-E) and regular instruction-aware encoders (GCPM-I). However, in cases of high visual entanglement (e.g., playing cards where number and suit are visually mixed), the text-based concept proxy approach proved superior to direct image embedding.

5. Significance and Claims

The authors claim that this work fundamentally bridges the gap between rigid, task-specific clustering algorithms and the flexible requirements of real-world applications. By leveraging textual guidelines, the framework eliminates the need for retraining when clustering criteria change.

The significance lies in the hybrid design: it combines the computational efficiency of traditional embedding-based clustering with the sophisticated semantic reasoning of LLMs, but only where necessary. This allows the system to handle:

• Composite Criteria: Grouping by multiple attributes simultaneously.
• Abstract Semantics: Understanding guidelines that require reasoning beyond simple visual similarity.
• Long-Tail Distributions: Adapting to datasets with extreme class imbalance without prior knowledge of cluster counts.

The paper concludes that this guideline-driven paradigm offers a scalable, training-free solution for diverse image clustering scenarios, outperforming specialized methods that require dedicated training for each new task or dataset.