HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models

Technical Summary: HANCLIP – A Family of Hyperbolic Angular Negation Vision Language Models

Problem Statement
Vision-Language Models (VLMs), despite their strong performance on standard retrieval and classification tasks, exhibit significant brittleness when processing negation. Current models often rely on shallow word-object co-occurrences rather than deep logical reasoning, leading them to assign high similarity scores to both affirmative and negated captions (e.g., failing to distinguish between "a cat next to an apple tree" and "a cat next to a chair, not an apple tree"). Existing solutions that attempt to address this via specialized datasets and contrastive objectives typically require large-scale retraining, which risks catastrophic forgetting of previously acquired knowledge and degrades performance on standard benchmarks. Furthermore, many approaches struggle to model the compositional nature of visual negation, which requires reasoning over object relations and alternatives rather than simple lexical inversion.

Methodology
The authors introduce HANCLIP (Hyperbolic + Angular + Negation), a framework designed to restructure the embedding space to explicitly encode "what an image is not" alongside "what it is." The approach is model-agnostic and can be applied to pre-trained backbones like CLIP, LongCLIP, SmartCLIP, and HiMo-CLIP using a compact dataset of 20,000 image-text quadruplets $(I, P, P', N)$, where $I$ is the image, $P$ is the positive caption, $P'$ is the negated-positive caption, and $N$ is the true negative caption.

The methodology relies on two core components:

1. Hyperbolic Contrastive Objective (HCO):
   Utilizing the Poincaré ball model of hyperbolic geometry, HCO leverages the property that distances grow exponentially with radius to model hierarchical and asymmetric semantic relations. Unlike standard Euclidean contrastive learning which treats all non-matching pairs as implicit negatives, HCO explicitly incorporates true negative captions ($N$) into the loss denominator.

   • Mechanism: It enforces a hierarchy where the image $I$ and its affirmative/negated-positive captions ($P, P'$) are embedded closer to the center of the hyperbolic space, while semantically incompatible negatives ($N$) are pushed toward the boundary.
   • Loss Terms: The objective consists of three terms: aligning $P'$ with $I$; ensuring $I$ is closer to $P'$ than to any $N$; and aligning $P'$ with $P$ while separating them from $N$.

2. Angular Triplet Negation Loss (ATNL):
   This component operates on relative directions rather than absolute distances to enforce geometric consistency among the triplet $(P, P', N)$.

   • Positive-Alignment ($L_{pos}$): Using the negative caption $N$ as an anchor, this term minimizes the angle between the vectors $\vec{NP}$ and $\vec{NP'}$. This ensures that the negated-positive caption $P'$ lies in a similar semantic direction to the true positive $P$ when viewed from the negative anchor.
   • Negative-Separation ($L_{neg}$): Using the negated-positive caption $P'$ as an anchor, this term maximizes the angle between vectors $\vec{P'P}$ and $\vec{P'N}$, pushing the true positive away from the negative in feature space.
   • Total Loss: The final training objective is a weighted sum: $L_{final} = L_{hc} + \alpha L_{at}$.

3. Forgetting Mitigation via LoRA:
   To prevent the degradation of pre-trained capabilities, the framework employs Low-Rank Adaptation (LoRA). The pre-trained backbone weights are frozen, and task-specific adaptation is routed through small, low-rank branches. This modular design allows the model to accumulate negation-aware skills without overwriting the subspace used for general retrieval and classification.

Key Contributions

• Hyperbolic Negation Modeling: The paper proposes the first application of hyperbolic geometry specifically to solve negation understanding in VLMs, leveraging non-Euclidean metrics to better separate affirmative and negative semantic clusters.
• Angular Triplet Negation Loss: A novel bifold loss function that simultaneously pulls true positives and negated-positives together while pushing negatives away, based on angular alignment rather than just distance.
• Data Efficiency and Performance: The framework achieves significant improvements using only 20,000 training samples (a small subset of CC-Neg), avoiding the need for massive retraining.
• Model Agnosticism: The method is successfully applied to four distinct VLM architectures (CLIP, LongCLIP, SmartCLIP, HiMo-CLIP) without large-scale retraining.

Experimental Results
Extensive experiments were conducted on the NegBench benchmark (covering Negative Retrieval and Zero-shot Multiple Choice Questions) and standard retrieval/classification datasets (ImageNet, COCO, Flickr30k, Urban1k, ShareGPT4V).

• Negation Performance: HANCLIP delivers consistent gains on NegBench.
  • Negative Retrieval: The framework improves the overall rSum score by approximately 5%–10% across all tested backbones. For instance, SmartCLIP-L improved its rSum from 427.49 to 438.51.
  • Multiple Choice Questions (MCQ): There is an average boost of nearly 35% in MCQ tasks. The framework shows particular strength in "Hybrid" questions (e.g., "A but not B"), where it improves performance significantly more than on pure "Negated" questions, suggesting better relative reasoning capabilities.
• Preservation of General Capabilities:
  • Classification: Performance on ImageNet, CIFAR-10, and CIFAR-100 remains largely unchanged (variations within $\pm 0.3\%$), with some variants showing slight improvements, indicating that the geometric reshaping does not harm base recognition.
  • Retrieval: Standard short-text and long-text retrieval tasks generally see maintained or improved performance. Notably, Image-to-Text (I2T) retrieval improves consistently, while Text-to-Image (T2I) shows mixed but generally stable results.
• Ablation Studies: The Hyperbolic Contrastive Objective (HCO) is identified as the primary driver for improved negation handling, while the Angular Triplet Negation Loss (ATNL) provides further refinement, particularly for MCQ tasks.
• LoRA Impact: Adapting the text encoder alone yields the strongest negation gains, while joint visual-text adaptation ("V+T") offers the best balance between negation performance and long-text retrieval stability.

Significance
The paper claims that HANCLIP demonstrates that a carefully designed geometric objective can substantially extend the reasoning capabilities of existing VLMs without requiring large-scale retraining or causing catastrophic forgetting. By injecting logical structure directly into the geometry of the embedding space, the framework offers a principled way to improve negation sensitivity while preserving the global structure of pretrained representations. The results suggest that non-Euclidean geometries and angular constraints are effective tools for addressing the "brittleness" of VLMs in handling complex logical operations like negation.