🤖 AI

HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning

HERMAN is a novel Class-Incremental Learning framework for CLIP that leverages LLMs to generate hierarchical textual descriptors and adaptively matches them to multi-level visual representations, thereby achieving state-of-the-art performance by better capturing fine-grained distinctions and mitigating catastrophic forgetting.

Original authors: Zhen-Hao Xie, Yan Wang, Lan Li, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhen-Hao Xie, Yan Wang, Lan Li, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, well-traveled librarian (the AI) how to recognize new animals. You start with a Cat and a Dog. The librarian learns to tell them apart easily.

Then, you introduce a Wolf. This is tricky because a wolf looks a lot like a dog. If you just tell the librarian, "Hey, remember that wolf has pointy ears," and force them to update their entire mental library at once, they might accidentally mess up their definition of "Dog." Suddenly, they might start thinking a Golden Retriever is a wolf because of those pointy ears, and they might forget how to tell a cat from a dog entirely. This is called Catastrophic Forgetting—learning something new ruins what you already knew.

This paper introduces a new system called HERMAN to solve this problem. Here is how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Mistake

Current AI methods are like a librarian who only has one giant, messy shelf. They try to fit the broad difference between "Cat" and "Dog" on the same shelf as the tiny difference between "Dog" and "Wolf." When they try to add the "Wolf" details, they accidentally knock over the "Cat vs. Dog" books. The system gets confused because it's trying to do everything at the same level of detail.

2. The Solution: A Multi-Level Library (Hierarchical Representation)

HERMAN changes the library structure. Instead of one messy shelf, it builds a multi-level hierarchy:

The Top Shelf (Coarse Level): This holds big, broad categories like "Animal," "Vehicle," or "Fruit." It's very stable.
The Middle Shelf: This holds medium details like "Has fur," "Has wheels," or "Is red."
The Bottom Shelf (Fine Level): This holds tiny, specific details like "Pointy ears," "Rusty hubcaps," or "A small stem."

How HERMAN builds this:
Instead of just asking the librarian, "What is a Wolf?", HERMAN uses a super-smart assistant (an LLM) to write a whole list of descriptions for every animal, ranging from the very general to the very specific.

General: "It's a canine."
Specific: "It has a long snout and thick fur."

The AI then matches these written descriptions to different layers of its own "brain" (the visual layers). The top layers of the brain look at the big picture, and the bottom layers look at the tiny details. This keeps the "Cat vs. Dog" rules safe on the top shelf while the "Dog vs. Wolf" rules get their own specific space on the bottom shelf.

3. The Smart Manager: The Adaptive Router

Now, how does the AI know which shelf to look at?

If you show it a Car vs. a Bicycle, it only needs the "Top Shelf" (wheels vs. no wheels).
If you show it a Sparrow vs. a Finch, it needs the "Bottom Shelf" (tiny color patterns).

HERMAN uses a Smart Manager (Router). This manager looks at the image and decides, "Okay, for this specific picture, I need to listen to the Top Shelf 80% and the Bottom Shelf 20%." It dynamically adjusts the volume for each level of detail depending on what it's looking at.

4. The Safety Net: Projection-Based Updates

Here is the tricky part: When the Smart Manager learns a new rule for a new task, how do we make sure it doesn't forget the old rules?

Imagine the Manager's brain is a room full of furniture (knowledge).

Old Way: When you bring in new furniture (new knowledge), you just shove it in, knocking over the old chairs.
HERMAN's Way: Before bringing in new furniture, the Manager takes a "snapshot" of the empty space where the old chairs sit. When adding new knowledge, it is forced to fit the new items into the empty spaces or stack them in a way that doesn't touch the old chairs.

Technically, this is called a projection-based strategy. It ensures that the new learning happens in a direction that is "orthogonal" (at a right angle) to the old knowledge, so the old knowledge stays perfectly intact.

The Result

By organizing knowledge into layers (hierarchy), using a smart manager to pick the right level of detail, and using a safety net to protect old memories, HERMAN allows the AI to learn new classes (like adding 100 new types of cars or birds) without forgetting the old ones.

The paper claims this method beats all previous methods on standard tests, achieving the best results (SOTA) in keeping the AI smart and forget-free. It does this without needing to store old photos, just by using these clever mathematical tricks to organize the AI's memory.

Technical Summary: HERMAN for Class-Incremental Learning

Problem Statement

Class-Incremental Learning (CIL) aims to continuously acquire new classes while preserving knowledge of previously learned categories. A primary challenge in this setting is catastrophic forgetting, where learning new information degrades performance on old tasks. While recent approaches leverage pre-trained models like CLIP to provide strong cross-modal alignment and prior knowledge, existing CLIP-based CIL methods suffer from two critical limitations:

Brittle Knowledge Representation: Most methods rely on single-level, coarse-grained alignment between the final visual layer and fixed textual templates (e.g., "a photo of a [CLASS]"). This forces fine-grained updates (e.g., distinguishing a "wolf" from a "dog") into the same semantic space used for coarse separations (e.g., "canine" vs. "feline"). Consequently, updates for new details can inadvertently shift decision boundaries, degrading performance on old classes.
Underutilization of Hierarchical Semantics: CLIP's vision and text encoders inherently capture multi-level features and descriptors ranging from coarse concepts to fine-grained attributes. However, existing methods typically ignore these intermediate representations, focusing solely on the final layer.

Methodology: HERMAN

The authors propose HERMAN (HiErarchical Representation MAtchiNg), a framework designed to structure, align, and adaptively route hierarchical semantics within CLIP-based CIL. The method consists of three core components:

1. Hierarchical Representation Matching

Instead of using a single fixed prompt, HERMAN leverages Large Language Models (LLMs) to generate a set of $M$ textual descriptors for each class, spanning from coarse-grained identity cues to fine-grained visual details.

Textual Generation: For a class $i$ , an LLM recursively generates descriptors $T_i = \{t_{i,1}, \dots, t_{i,M}\}$ . For example, for "Aston Martin Virage," descriptors range from "sports car" (coarse) to "sculpted side panels" (fine).
Multi-Level Alignment: These descriptors are encoded via the CLIP text encoder to form a hierarchical textual space. The model explicitly aligns these textual embeddings with intermediate visual layers of the CLIP vision encoder (not just the final layer).
Top-K Aggregation: To reduce noise, the model selects the top- $K$ most aligned descriptors for each visual layer and aggregates them into a compact textual embedding $h_i^b$ for layer $b$ .

2. Adaptive Hierarchy Routing

Recognizing that the importance of different hierarchical levels varies by input and task, HERMAN introduces an input-conditioned adaptive router.

Dynamic Weighting: A lightweight linear layer generates non-negative weights $\beta_i$ based on the final visual representation. These weights dynamically balance the contributions of the unified hierarchical textual embeddings from different layers to form the final representation $\tilde{h}_i$ .
Projection-Based Update (Anti-Forgetting): To prevent the router from forgetting policies learned for previous tasks, the router weights are updated using a projection-based strategy.
- The weight matrix $W_r$ is decomposed via Singular Value Decomposition (SVD).
- A projection matrix $P_{old}$ is constructed to preserve the subspace containing the routing patterns of previous tasks.
- New updates are decomposed into components within this preserved subspace (stability) and orthogonal directions (plasticity). The final update combines these to adapt to new tasks without disrupting established knowledge.

3. Cross-Modal Matching and Training

The framework employs a lightweight visual adapter and a contrastive objective.

Mixed Embeddings: The final prediction utilizes a convex interpolation of the aggregated hierarchical embedding $\tilde{h}_i$ and the standard template embedding $e_i$ , controlled by a factor $\lambda$ .
Generative Replay: To further mitigate forgetting without storing raw images, the method uses feature-level generative replay. It models the distribution of the final visual token ( $x_{cls}^B$ ) for each class as a Gaussian distribution, sampling pseudo-features during incremental training to stabilize past knowledge.

Key Contributions

Hierarchical Semantic Structuring: The paper introduces a method to generate and align multi-level textual descriptors with intermediate visual features, moving beyond single-layer alignment to create a structured semantic space.
Adaptive Routing Mechanism: A novel router dynamically balances hierarchical contributions per instance, allowing the model to prioritize coarse or fine cues as needed.
Projection-Based Continual Adaptation: A projection-based update strategy for the router ensures that learning new tasks does not overwrite the routing policies essential for old tasks, effectively mitigating catastrophic forgetting.
State-of-the-Art Performance: The integration of these components results in a framework that enhances discrimination and robustness in continual learning scenarios.

Experimental Results

The authors evaluated HERMAN on nine benchmark datasets (including CIFAR100, ImageNet-R, CUB200, StanfordCars, and SUN397) under various incremental settings (e.g., Base0 Inc10, Base50 Inc10).

Performance: HERMAN consistently outperforms state-of-the-art methods (including L2P, DualPrompt, CODA-Prompt, RAPF, and PROOF) by 1% to 5% in terms of average accuracy ( $\bar{A}$ ) and final accuracy ( $A_B$ ).
Robustness: The method demonstrates stability across different random seeds for class partitioning and remains robust to variations in hyperparameters (e.g., Top-K descriptors, mixing factor $\lambda$ , and projection parameters $\delta, \rho$ ).
Efficiency: HERMAN requires less training time than competing methods while maintaining superior performance, making it computationally efficient for large-scale tasks.
Ablation Studies: Experiments confirm that each component (LLM descriptors, adaptive router, and projection updates) contributes positively to the final performance. The projection mechanism is specifically shown to maintain high cosine similarity between visual and textual embeddings of old classes over time, unlike methods without projection which suffer rapid degradation.

Significance and Claims

The paper claims that HERMAN addresses the fundamental brittleness of existing CLIP-based CIL methods by decoupling learning across semantic granularities. By isolating fine-grained updates to specific levels of the hierarchy while anchoring coarse separations, the method prevents destructive boundary shifts. The authors assert that their approach effectively leverages the inherent hierarchical capacity of pre-trained models, transforming them into more robust continual learners capable of adapting to new categories without forgetting prior knowledge. The work highlights the potential of combining LLM-generated semantic richness with structured, adaptive alignment in vision-language models.