💻 computer science

Robust Hyperbolic Learning with Curvature-Aware Optimization

This paper proposes a curvature-aware optimization framework, featuring a new Riemannian AdamW derivation and a fine-tunable scaling approach, to improve the stability, generalization, and computational efficiency of hyperbolic deep learning across various tasks.

Original authors: Ahmad Bdeir, Johannes Burchert, Lars Schmidt-Thieme, Niels Landwehr

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmad Bdeir, Johannes Burchert, Lars Schmidt-Thieme, Niels Landwehr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive, sprawling family tree. If you try to draw this tree on a flat piece of paper (which is like Euclidean space, the math we use for everyday life), you’ll quickly run out of room. As the family grows, the branches get too crowded, and you have to squish them together, making it impossible to see who is related to whom.

However, if you draw that family tree on a saddle-shaped surface (which is Hyperbolic space), you have much more "room." The surface naturally curves and expands, allowing the branches to spread out beautifully without overlapping. This is why scientists use hyperbolic math for things like social networks, biological hierarchies, and complex images.

The Problem: The "Wobbly Saddle" Dilemma

While hyperbolic space is great, it is incredibly difficult to work with. Imagine trying to build a high-tech skyscraper on a giant, wobbly saddle that is constantly changing its shape.

Current AI researchers face three big headaches:

The Shape-Shifter Problem: Researchers want the AI to "learn" the best curvature (the exact shape of the saddle) for the data. But if the saddle changes shape while the AI is mid-step, the math breaks, the "building" collapses, and the computer crashes (this is called instability).
The Edge of the World: In hyperbolic math, if a data point moves too far from the center, it hits a "mathematical cliff" where the numbers become impossible to calculate (this is numerical instability).
The Heavy Lifting: Doing math on a curved surface is much slower and more "expensive" for a computer than doing math on a flat floor.

The Solution: The "Smart Architect" Approach

The authors of this paper have essentially designed a better set of tools for the architect building on this wobbly saddle. Here is how they fixed it:

1. The "Staggered Update" (Curvature-Aware Optimization)

Instead of changing the shape of the saddle and the position of the building at the exact same time (which causes the crash), they use a staggered approach.

The Analogy: Imagine you are dancing on a trampoline. If the trampoline suddenly changes its tension while your feet are in mid-air, you’ll fall. Instead, the authors' method says: "First, let's adjust the tension of the trampoline, then let's adjust where your feet land." By mathematically "mapping" the old position to the new shape before moving, they keep the training smooth and stable.

2. The "Safety Tether" (Maximum Distance Rescaling)

To prevent data points from flying off the "mathematical cliff," they introduced a new way to keep everything in bounds.

The Analogy: Think of it like a flexible bungee cord. Instead of a hard wall that stops a point abruptly (which can jerk the math and cause errors), they use a smooth, "squishy" boundary. As a data point gets too far out, the bungee cord gently pulls it back toward the center, ensuring it stays in the "safe zone" where the math works perfectly.

3. The "Hyperbolic AdamW" (Better Regularization)

In AI, "overfitting" is when a model memorizes the answers instead of learning the patterns (like a student memorizing a practice test instead of learning math).

The Analogy: They created a new "optimizer" (a specialized math engine) called Riemannian AdamW. Think of this as a smart coach that doesn't just tell the AI how to move, but also constantly reminds it to "stay simple" and "don't overthink it," which helps the AI perform better on new, unseen data.

4. The "Shortcut" (Efficient Implementation)

Finally, they found a way to make the math faster.

The Analogy: Instead of recalculating the entire curved universe every time they want to move a pixel, they found a mathematical shortcut that lets them use the fast, "flat-earth" tools that modern computer chips (GPUs) are already incredibly good at, while still getting the benefits of the curved space.

The Result: Faster, Stronger, Smarter

The researchers tested their "Smart Architect" tools on everything from brain waves (EEG) to image generation. The results?

It’s faster: They achieved massive speed-ups (in some cases, 42 times faster!).
It’s more accurate: The AI became better at recognizing patterns in complex data.
It’s more efficient: It uses much less computer memory, meaning we can run powerful hyperbolic AI on smaller, more accessible machines.

Technical Summary: Robust Hyperbolic Learning with Curvature-Aware Optimization

1. Problem Statement

Hyperbolic deep learning is highly effective for modeling hierarchical and tree-like data structures due to the negative curvature of hyperbolic manifolds, which allows for exponentially growing distances. However, current approaches face four critical challenges:

Instability in Curvature Learning: When the manifold curvature ( $K$ ) is treated as a learnable parameter, updating it creates geometric inconsistencies. If the curvature changes without simultaneously re-aligning the hyperbolic parameters and their gradients, mathematical operations (like $acosh$) can receive invalid inputs, leading to NaN values and model crashes.
Overfitting: The high representational capacity of hyperbolic spaces makes models prone to overfitting, especially in data-scarce scenarios (e.g., EEG classification).
Numerical Instability: Hyperbolic embeddings can drift toward the edge of the manifold's representational radius, where floating-point precision errors lead to inaccurate distance calculations and training divergence.
Computational Complexity: Hyperbolic operations are more expensive than Euclidean ones, and the lack of optimized CUDA implementations for hyperbolic convolutions leads to high memory usage and slow training.

2. Methodology

The authors propose a comprehensive framework to stabilize and optimize learning in the Lorentz (Hyperboloid) model. Their approach consists of three main pillars:

A. Curvature-Aware Optimization Schema
To solve the instability caused by changing curvature, the authors propose a staggered update mechanism. Instead of updating all parameters simultaneously, the optimizer:

Updates hyperbolic parameters ( $\theta_L$ ) using the old curvature.
Updates Euclidean parameters ( $\theta_E$ ) and the curvature parameter ( $\theta_K$ ).
Tangent-Based Mapping: Uses a mapping function to project the old hyperbolic parameters and their gradients/momentums onto the new manifold. By projecting parameters onto the tangent space at the origin via the logarithmic map and then projecting them back via the exponential map, they preserve the relative distances and angles (hierarchy levels) between points.

B. Riemannian AdamW (RAdamW)
To combat overfitting, the authors derive a hyperbolic version of the AdamW optimizer. While standard Adam applies L2-regularization via the loss function, AdamW applies weight decay directly during the update step. The authors reformulate this for the Lorentz manifold by defining regularization as a weighted Lorentzian centroid between the current parameter and the manifold origin, ensuring that regularization is geometrically consistent with the hyperbolic space.

C. Maximum Distance Rescaling
To prevent numerical instability, they introduce a flexible scaling function (a modified $tanh$ function). This function constrains hyperbolic vectors within a "safe" representational radius (determined by the precision of the floating-point format, such as FP16 or FP32) without the non-smooth gradient issues caused by traditional "clipping."

D. Efficient Hyperbolic Convolutions
To improve speed, they implement a "convolution trick" that decomposes hyperbolic convolutions into a Lorentz boost and a rotation. By parameterizing the rotation as a norm-preserving transformation of standard weights, they can leverage highly optimized, existing CUDA-based Euclidean convolution kernels, significantly reducing runtime.

3. Key Contributions

New Riemannian Optimizer Schema: A method to stabilize the learning of curvature by mapping parameters and gradients through the tangent space at the origin.
Riemannian AdamW: A derivation of the AdamW optimizer specifically for the Lorentz manifold to improve generalization through direct weight decay.
Flexible Scaling Function: A smooth, controllable rescaling mechanism to keep embeddings within the stable numerical radius of the manifold.
Efficient Implementation: A method to utilize standard CUDA convolutions for hyperbolic learning, drastically reducing computational overhead.

4. Results

The authors demonstrated consistent improvements across five diverse domains:

Hierarchical Metric Learning (LHIER): Improved Recall@1 by up to 4.71% compared to existing methods, proving the effectiveness of curvature adaptation.
EEG Classification (HyperMAtt): Achieved state-of-the-art results on small datasets (SSVEP, ERN) and achieved a massive 42x speed-up per epoch compared to previous models.
Computer Vision (HCNN+): In ResNet-50 experiments, the proposed model outperformed hybrid models, reduced VRAM usage by ~48%, and reduced runtime by ~66%.
Image Generation (VAE): Improved both reconstruction and generation FID scores while being 2.5x more memory-efficient and 3x faster.
Graph Embedding: Achieved superior node classification performance by combining learnable curvature, RAdamW, and tangent-based scaling.

5. Significance

This paper provides a robust mathematical and practical foundation for scaling hyperbolic deep learning. By solving the fundamental issues of geometric inconsistency and numerical instability, the authors move hyperbolic learning from a niche, unstable research area toward a practical tool capable of handling large-scale, high-dimensional datasets in computer vision and signal processing with high efficiency.