💻 computer science

Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation

This paper introduces HyRo, a hyperbolic fine-tuning framework that decouples hierarchical and semantic alignment in the Poincaré ball model to achieve state-of-the-art performance in open-vocabulary semantic segmentation.

Original authors: Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot how to look at a photo and point out exactly where every object is (like a person, a chair, or a tree). This is called "semantic segmentation." The robot already knows a lot about words and pictures because it was trained on millions of them, but it's bad at looking at a single photo and saying, "That pixel is a chair, and that pixel is a person."

The problem is that the robot's internal "map" of the world is a bit messy. It knows that "furniture" is a big category and "chair" is a smaller one inside it, but when it tries to look closely at a specific photo, it gets confused. It might think a person sitting on a chair is just one giant "chair" blob because it can't tell the difference between the two.

The Old Way: Just Changing the "Zoom"

Previous researchers tried to fix this by using a special kind of geometry called Hyperbolic Space. Think of this space like a tree.

The top of the tree (the trunk) represents big, general ideas like "furniture."
The branches represent smaller ideas like "chair."
The leaves represent specific items like "that specific wooden chair."

In this tree-like map, the distance from the center (the trunk) tells you how specific an idea is. The closer to the center, the more general; the further out, the more specific.

A previous method called HyperCLIP tried to fix the robot's confusion by simply stretching or shrinking the distance from the center. It was like zooming in or out on the tree to make the "chair" branch the right size for the photo. But there was a catch: it only changed the distance. It didn't fix the direction.

The Analogy: Imagine you are holding a compass. If you only change how far you walk from the center of a room, but you keep walking in the wrong direction, you'll still end up in the wrong place. The old method fixed the "how far" but ignored the "which way."

The New Way: HyRo (The "Spin" Fix)

The authors of this paper propose a new method called HyRo (Hyperbolic Rotation). They realized that to fix the robot's confusion, you need to do two things at once:

Adjust the Distance (Radius): Make sure the idea is at the right level of detail (general vs. specific).
Adjust the Direction (Angle): Make sure the idea is pointing in the right semantic direction.

The Creative Metaphor:
Imagine the robot's understanding of the world is a globe (like the Earth).

Latitude (Distance from the center): This tells you if you are talking about a broad concept (like "Animal") or a specific one (like "Dog").
Longitude (The Angle): This tells you which animal you are talking about.

The old method (HyperCLIP) was like moving a sticker up or down the globe to change its latitude, but it never rotated the sticker to the correct longitude. So, a "Dog" sticker might end up at the right distance from the center, but it's stuck on the "Cat" line of longitude.

HyRo adds a new step: Rotation.
It keeps the sticker at the perfect distance (so it knows it's a specific animal), but it spins the globe to move the sticker to the correct longitude. This ensures that "Dog" is actually pointing to "Dog," and "Chair" is pointing to "Chair," even if they are sitting right next to each other in the picture.

How It Works in Simple Steps

Map the World: The robot takes its standard picture and text knowledge and maps it onto this special "tree-like" globe (the Poincaré ball).
Zoom In/Out: It first adjusts the distance of the words from the center to match the level of detail needed for the photo (e.g., making sure "chair" is specific enough).
Spin the Globe: This is the magic part. It uses a mathematical "spin" (an orthogonal rotation) to turn the words so they align perfectly with the picture. Crucially, this spin does not change the distance. It only changes the direction.
The Result: The robot can now clearly see that the "person" and the "chair" are two different things, even though they are close together, because their "directions" in the robot's mind have been corrected.

What They Found

The authors tested this on many standard photo datasets.

The Result: Their method (HyRo) beat all previous methods, including the one that only adjusted the distance.
Why it matters: It proved that to really understand a picture, you can't just worry about how "specific" a word is; you also have to make sure the word is pointing in the right direction. By fixing both the distance and the angle, the robot became much better at finding and separating objects in complex scenes.

In short, HyRo teaches the robot to not just know how big an idea is, but also exactly where it belongs in the big picture.

Technical Summary: HyRo – Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation

1. Problem Statement

Open-vocabulary semantic segmentation (OVSS) aims to assign semantic categories to every pixel in an image, including classes unseen during training. While Vision-Language Models (VLMs) like CLIP provide rich semantic alignments, they are pre-trained for image-level representation, making the adaptation to dense, pixel-level predictions challenging.

Existing approaches face two primary limitations:

Hierarchical vs. Semantic Mismatch: Recent works utilizing hyperbolic geometry (e.g., HyperCLIP) successfully model hierarchical relationships by adjusting the hyperbolic radius of embeddings to match pixel-level granularity. However, they often overlook semantic misalignment among embeddings within the same hierarchical level.
Geometric Constraints: In hyperbolic space, hierarchy is encoded by radial distance, while semantic similarity is reflected in angular orientation. Prior methods that only adjust the radius fail to correct incorrect angular relationships, leading to scenarios where semantically distinct concepts (e.g., a "person" and a "chair") are embedded at similar radii but incorrect angles, causing the model to misclassify them as a single object.

2. Methodology: HyRo (Hyperbolic Rotation)

The authors propose HyRo, a fine-tuning framework that explicitly decouples hierarchical alignment and semantic refinement within the Poincaré ball model. The core insight is that hierarchy and semantics correspond to distinct geometric properties: radius for hierarchy and angle for semantics.

2.1. Theoretical Foundation

The method operates in the Poincaré ball model ( $D^m_c$ ), where:

Conformality: Angles measured at the origin are preserved from Euclidean space.
Orthogonal Transformations: The authors prove that applying an orthogonal matrix $R$ $R$ to a point in the Poincaré ball acts as an ideal rotation. It adjusts the angular relationship (semantic alignment) without altering the hyperbolic radius (hierarchical depth).
- Let $x$ be an embedding. Its tangent vector at the origin is $v_x = \log_0(x)$ .
- Applying an orthogonal matrix $R$ yields $v'_x = R v_x$ . Since $R$ preserves the Euclidean norm ( $\|v'_x\| = \|v_x\|$ ), the hyperbolic radius remains unchanged after mapping back via the exponential map: $x' = \exp_0(R v_x) = R x$ .

2.2. Architecture and Workflow

The HyRo pipeline (illustrated in Fig. 3) consists of four stages:

Mapping to Hyperbolic Space: Euclidean feature embeddings ( $z$ ) from CLIP are mapped to the Poincaré ball using the exponential map: $h = \exp_0^{\mathbb{D},c}(z)$ .
Hierarchical Adjustment (Radius): A learnable diagonal matrix $S$ (with block-diagonal structure for efficiency) scales the hyperbolic radius to align embeddings with the appropriate pixel-level granularity: $q = S \otimes_c h$ .
Semantic Refinement (Rotation): An orthogonal matrix $R$ $R$ is applied to refine angular relationships: $v = R q$ $v = R q$ .
- Parameterization: To ensure $R$ is strictly orthogonal, the authors use the Cayley transform on a learnable skew-symmetric matrix $\Theta$ : $R = (I + A)(I - A)^{-1}$ , where $A = \Theta - \Theta^\top$ .
- Efficiency: To handle high-dimensional CLIP embeddings (e.g., 768-dim) and differing modality dimensions, $R$ is decomposed into independent block-diagonal matrices, reducing computational cost from $O(d^3)$ to $O(d^3/n^2)$ .
Decoding: The refined hyperbolic embeddings are mapped back to Euclidean space via the logarithmic map and fed into a Cost Aggregation Decoder (based on CATSeg) to generate pixel-level predictions.

2.3. Training Strategy

Frozen Encoders: The CLIP encoders remain frozen to preserve generalization capabilities.
Fine-Tuning: Only the hyperbolic transformation parameters (radius scaling matrices $S$ and rotation matrices $R$ ) are fine-tuned using pixel-wise cross-entropy loss.

3. Key Contributions

Decoupled Alignment: The paper introduces a framework that explicitly separates hierarchical alignment (radius) from semantic refinement (angle) in hyperbolic space, addressing a critical gap in prior OVSS methods.
Orthogonal Rotation Strategy: A theoretically justified method using orthogonal transformations (via Cayley parameterization) to adjust semantic angles without distorting hierarchical structure.
State-of-the-Art Performance: The proposed method achieves superior results on standard benchmarks, validating the geometric approach.

4. Experimental Results

The authors evaluated HyRo on standard OVSS benchmarks: ADE20K (A-847, A-150), PASCAL-Context (PC-459, PC-59), and PASCAL VOC (PAS-20, PAS-20b).

Quantitative Performance: HyRo achieved the best performance on four out of six benchmarks. Notably, it improved upon the previous hyperbolic state-of-the-art (HyperCLIP) by:
- +0.1 mIoU on A-847 (12.0 vs 11.9)
- +0.7 mIoU on PC-459 (18.9 vs 18.2)
- +0.2 mIoU on PC-59 (57.3 vs 57.1)
- +0.1 mIoU on PAS-20 (95.0 vs 94.9)
Qualitative Improvements: Visualizations show HyRo effectively resolves semantic ambiguities where prior methods fail. For instance, in scenes with a person lying on a chair, HyperCLIP often misclassifies both as "chair," whereas HyRo correctly disentangles the objects. Attention maps confirm that HyRo concentrates attention on target regions and suppresses background noise.
Ablation Studies:
- Component Importance: Combining both radius scaling and rotation yields the best results. Rotation is identified as the primary driver for open-vocabulary generalization, while radius scaling ensures fine-grained separability.
- Curvature ( $c$ ): A "gentler" curvature ( $c=0.01$ ) was found to be optimal, preserving CLIP's zero-shot generalization while allowing necessary hierarchical adjustments.
- Block Size: A block size of $n=256$ provided the best balance of capacity and generalization.

5. Significance and Claims

The paper claims that HyRo highlights the importance of jointly modeling hierarchy and semantics in non-Euclidean spaces. By demonstrating that angular relationships are as critical as radial distances for dense prediction, the work opens new directions for geometric representation learning in vision-language tasks.

The authors position their work as a solution to the "semantic misalignment" problem inherent in previous hyperbolic approaches. They argue that simply adjusting the granularity (radius) is insufficient; explicit optimization of the semantic orientation (angle) is required to achieve robust open-vocabulary generalization. The method is presented as a significant step forward in adapting foundation models for dense, pixel-level understanding without sacrificing their ability to recognize unseen categories.

Future work is suggested to extend this hyperbolic geometric approach to complex open-vocabulary video segmentation settings (e.g., MOSE, MeViS), though the current scope is limited to static images.