💬 NLP

Hierarchical Textual Knowledge for Enhanced Image Clustering

This paper proposes Knowledge-Enhanced Clustering (KEC), a training-free method that leverages large language models to construct hierarchical concept-attribute textual knowledge, thereby significantly improving unsupervised image clustering performance and robustness across diverse datasets compared to existing visual-only and zero-shot approaches.

Original authors: Yijie Zhong, Yunfan Gao, Weipeng Jiang, Haofen Wang

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yijie Zhong, Yunfan Gao, Weipeng Jiang, Haofen Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive, chaotic attic filled with thousands of photos. Some photos look almost identical at first glance—a picture of a Golden Retriever and a picture of a Labrador might look very similar. A computer trying to sort these photos usually just looks at the colors and shapes (the "visual" clues). If the dogs look alike, the computer might accidentally put them in the same pile, even though they are different breeds.

This is the problem the paper "Hierarchical Textual Knowledge for Enhanced Image Clustering" (or KEC for short) is trying to solve.

Here is the simple breakdown of their solution, using some everyday analogies:

1. The Problem: The "Blind" Sorter

Traditional methods are like a blind sorter who only feels the texture of the photos. If two photos feel the same (same fur, same shape), they get grouped together. They miss the meaning.

The Flaw: They can't tell the difference between a "Shiba Inu" and an "Akita" just by looking at the pixels because the pixels are too similar.

2. The Old Solution: The "Dictionary" Approach

Some recent methods tried to help the computer by giving it a list of words (like a dictionary). They might say, "This photo is a 'Dog'."

The Flaw: This is too vague. Telling the computer "Dog" is like telling a librarian to sort books by "Animal." It's true, but it doesn't help distinguish a "Cat" from a "Dog." Furthermore, some methods just grabbed random words from a thesaurus, leading to a messy list of synonyms (e.g., "Canine," "Pup," "Doggy") that confused the computer even more.

3. The KEC Solution: The "Expert Librarian"

The authors propose a new method called KEC. Instead of just looking at the photo or reading a simple label, they use a Large Language Model (LLM)—think of it as a super-smart, well-read librarian—to build a structured knowledge map.

Here is how they do it, step-by-step:

Step A: The "Filing Cabinet" (Concept Abstraction)

Imagine you have a messy pile of sticky notes with words like "Shiba Inu," "Akita," "Corgi," "Poodle," and "Dog."

What KEC does: It asks the smart librarian to group these messy notes into neat folders.
The Result: Instead of 500 random words, the librarian creates 20 clear folders called "Dog Breeds," "Birds," "Vehicles," etc. This removes the clutter and redundancy.

Step B: The "Detective's Clues" (Discriminative Attributes)

This is the secret sauce. Just having the folder name "Dog Breeds" isn't enough to tell a Shiba from an Akita. The librarian needs to write down specific clues that tell them apart.

The Analogy: Imagine you are trying to tell two identical twins apart. You don't just say "They are twins." You say, "One has a mole on his left cheek, and the other has a scar on his right knee."
What KEC does: It asks the librarian: "What are the specific visual differences between a Shiba Inu and an Akita?"
- Answer: "The Akita has longer legs, a curlier tail, and ears that tilt forward."
- Answer: "The Shiba has a straighter tail and smaller ears."
The computer then uses these clues (attributes) to sort the photos. Even if the dogs look 90% similar, that 10% difference in "tail curl" or "ear tilt" is enough to put them in the right pile.

Step C: The "Hybrid Vision" (Knowledge Grounding)

Finally, the computer looks at the photo again. But this time, it doesn't just look at the pixels. It asks: "Does this photo have the 'curly tail' clue?"

It combines the visual look (the pixels) with the textual clues (the tail, the ears, the size).
This creates a "super-feature" that is much harder to get wrong.

Why is this better?

It's Automatic: You don't need to manually write down rules for every animal or car. The AI figures out the rules for you.
It's Robust: If you just throw random words at a computer, it might get confused and do worse than before. But because KEC builds a hierarchical structure (Big Categories -> Specific Clues), it is very stable and reliable.
It Works Without Training: The best part is that this method works immediately on new datasets without needing to "train" the computer for days. It just uses the librarian's brain to organize things on the fly.

The Bottom Line

Think of KEC as upgrading a computer from a blind sorter (who only sees shapes) to a smart detective (who sees shapes and understands the specific details that make things unique). By teaching the computer to look for "clues" like tail shapes and ear positions, rather than just "dog" or "car," it can organize a messy attic of photos with incredible accuracy.

1. Problem Statement

Image clustering aims to group unlabeled images into semantically meaningful clusters without supervision. While traditional methods rely on visual features (geometric priors, deep learning representations), they often struggle to distinguish between visually similar but semantically distinct classes (e.g., different dog breeds like Akita vs. Shiba Inu).

Recent approaches attempt to integrate textual knowledge using Vision-Language Models (VLMs) like CLIP. However, existing methods suffer from two main limitations:

Shallow Knowledge: They rely on coarse class labels, simple nouns (e.g., from WordNet), or pseudo-labels, which lack deep semantic hierarchies and discriminative attributes.
Redundancy and Noise: Directly using generated captions or raw noun sets introduces semantic redundancy and inconsistent granularity, which can degrade clustering performance rather than improve it. Naively incorporating text often fails to provide clear cluster boundaries.

The core challenge is: How to construct structured, discriminative, and hierarchical textual knowledge automatically to guide image clustering without manual annotation or expensive per-image generative modeling?

2. Methodology: Knowledge-Enhanced Clustering (KEC)

KEC proposes a fully automated pipeline that leverages Large Language Models (LLMs) to construct a hierarchical concept-attribute knowledge structure. This structure is then instantiated for input images to generate "knowledge-enhanced features."

The process consists of four main stages:

A. Initializing Image-Text Mapping

Visual Clustering: Input images are clustered using K-Means on visual features (CLIP embeddings).
Noun Retrieval: For each visual cluster, the top 5 most similar nouns from a predefined set (e.g., WordNet) are retrieved based on cosine similarity.
Goal: This creates an initial, albeit redundant, mapping between images and textual semantics.

B. Representative Concept Abstraction

Redundancy Reduction: The initial noun lists contain synonyms and varying granularities. KEC merges similar visual/textual clusters into connected components.
LLM Abstraction: An LLM is prompted to condense each merged noun set into a Representative Concept (e.g., converting "Shiba Inu," "Akita," "Husky" into "Dog Breeds") and a concise Description.
Output: A set of abstract concepts ( $C$ ) and their descriptions ( $D$ ), which are encoded into feature vectors using CLIP.

C. Discriminative Attribute Mining

To resolve ambiguity between similar concepts, KEC extracts two types of attributes via LLMs:

Uni-Concept Attributes: For each concept, the LLM identifies visual attributes that distinguish instances of this concept from all others (e.g., "Waxy texture" for a specific flower).
Bi-Concept Attributes: For pairs of semantically similar concepts (identified via feature similarity), the LLM extracts attributes that specifically differentiate the two (e.g., "Leg length" to distinguish Akita from Shiba Inu).

Result: A structured knowledge base where each concept is linked to a set of discriminative attributes.

D. Knowledge Grounding & Feature Enhancement

Instantiation: For a specific input image, the system calculates attention weights between the image and the concepts.
Attribute Grounding: Instead of using static attribute text, KEC "instantiates" attributes by combining the image visual features with the attribute text embeddings ( $\hat{\xi} = x \odot \xi$ ). This grounds the textual knowledge in the specific visual context of the image.
Feature Fusion: The final Knowledge-Enhanced Feature ( $\kappa$ ) is a weighted sum of the concept features and the instantiated attribute features.
Clustering: These enhanced features are concatenated with original visual features and fed into standard clustering algorithms (e.g., K-Means, TURTLE, TAC).

3. Key Contributions

Hierarchical Concept-Attribute Framework: Unlike previous methods using flat noun lists, KEC constructs a structured hierarchy: Nouns $\to$ Abstract Concepts $\to$ Discriminative Attributes (Uni & Bi).
Automated Knowledge Construction: The method is fully automated, requiring no manual annotation. It uses LLMs to distill redundant nouns and extract fine-grained, visual-grounded attributes.
Plug-and-Play Compatibility: KEC is model-agnostic and can be integrated into various downstream clustering strategies (training-free or training-based) as an additional feature space.
Robustness: The paper demonstrates that naive text integration harms performance, whereas KEC's structured approach consistently improves robustness and accuracy.

4. Experimental Results

The method was evaluated on 20 diverse datasets (ranging from 8 to 1000+ classes), including fine-grained tasks (Pets, Flowers, Aircraft) and domain-specific tasks (Satellite, Medical).

Performance Gains:
- Training-Free: KEC (without training) outperforms Zero-shot CLIP on 14 out of 20 datasets.
- Comparison: It achieves an average improvement of ~3% in NMI over existing state-of-the-art methods (SIC, TAC) and surpasses training-based methods by 3% on average.
- Specific Wins: Notable improvements on fine-grained datasets like Pets (+10% ARI) and Flowers.
Ablation Studies:
- Removing the hierarchical structure (using only concepts or only nouns) significantly drops performance.
- Both Uni-concept and Bi-concept attributes are necessary; they complement each other to refine cluster boundaries.
Efficiency: While LLM calls are involved, KEC is more efficient than per-image captioning methods (like VLM Caption + Cluster) because it constructs knowledge once per dataset, not per image.
Robustness: KEC performs consistently across different CLIP backbones (ResNet, ViT) and various LLMs (from 0.6B to flagship models), proving its generalizability.

5. Significance and Impact

Bridging the Gap: KEC successfully bridges the gap between visual similarity and semantic distinctiveness, solving the "visually similar but semantically different" problem that plagues traditional clustering.
Paradigm Shift: It shifts the focus from "noun selection" to "structured textual space modeling." It proves that the quality and structure of textual knowledge matter more than the sheer volume of text.
Interpretability: The generated concepts and attributes provide human-interpretable reasons for clustering decisions (e.g., "These images are grouped because they share 'curly tails' and 'forward-tilting ears'"), enhancing the explainability of unsupervised learning.
Future Direction: The work highlights the potential of human-in-the-loop systems where users can guide the granularity of concepts for specific domains, suggesting a path toward more adaptive and controllable AI clustering.

In summary, KEC demonstrates that leveraging LLMs to construct hierarchical, discriminative textual knowledge is a powerful, scalable, and robust strategy for enhancing unsupervised image clustering.