🤖 machine learning

Hyperbolic Multimodal Continual Learning

This paper addresses the underexplored challenges of continual learning in hyperbolic multimodal spaces by establishing a theoretical foundation for preventing forgetting through cross-modal invariance and proposing a principled framework that preserves both relational structure and hierarchical geometry.

Original authors: Jiahong Liu, Ming Shen, Xiaohao Liu, Rex Ying, Menglin Yang, Tat-Seng Chua, Irwin King

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahong Liu, Ming Shen, Xiaohao Liu, Rex Ying, Menglin Yang, Tat-Seng Chua, Irwin King

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. Right now, most robots learn by looking at pictures and reading words, then trying to match them up. They usually do this in a "flat" mental space, like a giant, endless sheet of graph paper. On this flat paper, everything is just a dot, and the distance between dots tells the robot how similar two things are. But the real world isn't flat; it's full of layers. A "dog" is a specific kind of "animal," which is a type of "living thing." This kind of hierarchy—where big, general ideas sit at the top and tiny, specific details branch out below—is hard to draw on flat paper without everything getting squished and messy.

To fix this, scientists have started using a different kind of geometry called "hyperbolic space." Think of this not as a flat sheet, but as a giant, expanding coral reef or a funnel that gets wider the further you go. In this shape, you can fit a massive amount of complex, layered information without it getting crowded. It's perfect for organizing ideas from the most general (like "animal" near the center) to the most specific (like "exhausted golden retriever" far out on the edge). But here's the tricky part: what happens when the robot learns new things over time? If you keep adding new lessons to this coral reef, the whole shape can warp and twist, causing the robot to forget everything it learned yesterday. This is called "catastrophic forgetting," and it's a huge problem for building smart, ever-learning AI.

This paper tackles that exact problem. The researchers, Jiahong Liu and their team, asked: "How do we teach a robot new things in this curved, coral-like world without breaking the shape and losing old memories?" They discovered that to stop the robot from forgetting, you can't just update its brain however you want. Instead, you have to move its knowledge in very specific, rigid ways that keep the coral reef's shape intact. They built a new method called HMCL (Hyperbolic Multimodal Continual Learning) that acts like a strict architect, ensuring that every new lesson fits perfectly into the existing structure without warping it. Their experiments show that this method works much better than older techniques, reducing the robot's forgetfulness by a huge margin—up to 88% less forgetting in some cases—while still letting it learn new tasks effectively.

The Story of the Curved Brain

The Setup: A Robot Learning in a Funnel
Imagine you are building a library for a robot. Most libraries are built on flat floors, where books are just placed in rows. But our robot's library is built inside a giant, magical funnel (that's the hyperbolic space). In this funnel, the most important, general books (like "Science" or "Art") sit near the narrow top, and as you go down and out, the shelves expand to hold millions of specific books (like "Quantum Physics" or "Impressionist Painting"). This shape is amazing because it lets the robot see how everything connects: a specific book is always "under" its general category.

The robot learns by looking at pictures and reading text, trying to match them up in this funnel. But life doesn't stop. New books arrive every day. The robot has to learn them one by one, without ever seeing the old books again. This is "continual learning."

The Problem: The Funnel Warps
Here is the disaster that usually happens. When the robot tries to learn a new book, it pushes the shelves around to make room. In a normal, flat library, this is fine. But in our magical funnel, pushing the shelves around changes the shape of the whole building. The "Science" section might get squished, or the "Art" section might stretch out too far.

Because the robot's memory relies entirely on the shape of the funnel to know what things are, when the shape warps, the robot gets confused. It might look at a picture of a dog and suddenly think it's a cat, or forget that a "poodle" is a type of "dog." The old knowledge doesn't just fade away; it gets distorted and broken. The researchers call this "catastrophic forgetting," and it's like the robot waking up with amnesia every time it learns something new.

The Discovery: The Golden Rule of Movement
The team realized that the reason the funnel was breaking was that the robot was moving its knowledge in the wrong directions. They asked a deep question: "What kind of movement keeps the funnel's shape perfect?"

They found a surprising answer. To keep the shape safe, the robot can't just wiggle its knowledge around randomly. Instead, every time it learns something new, it has to rotate its entire memory structure in a very specific, synchronized way. Imagine holding a spinning globe. If you want to add a new sticker without tearing the map, you have to rotate the whole globe so the sticker fits perfectly into the existing grid. You can't just stretch the rubber.

The paper proves that for the robot to remember everything, it must apply a "shared rotation" to all its memories. This means the relationship between a picture and its description (like a dog and the word "dog") must stay exactly the same distance apart, and the hierarchy (dog under animal) must stay exactly the same depth. If the robot tries to learn in any other way, the funnel breaks.

The Solution: HMCL
Based on this discovery, the team built a new training method called HMCL. Think of HMCL as a strict bouncer at the robot's brain. Every time the robot tries to learn a new task, HMCL checks the proposed changes.

"Hey, you're trying to stretch the 'dog' category? No, that breaks the funnel!"
"You're trying to rotate the whole thing perfectly? Okay, that's allowed."

HMCL forces the robot to only make updates that act like these perfect rotations. It doesn't just guess; it uses math to calculate the exact direction where the robot can learn without breaking the shape. It's like giving the robot a set of training wheels that only let it move in directions that preserve the geometry of the funnel.

The Results: Less Amnesia, More Learning
The team tested this on a bunch of different robot brains (called backbones) using real-world tasks like identifying animals in photos or matching images to sentences. They compared their new HMCL method against older methods that didn't know about the funnel's shape.

The results were clear. The old methods were terrible at remembering. They forgot almost everything they learned before, with "forgetting scores" that were very negative (like -6.46 on a scale where 0 is perfect). But with HMCL, the forgetting dropped dramatically. On one test, the forgetting score improved from -6.46 to just -0.74. That's an 88.5% reduction in forgetting!

The robot didn't just remember better; it also learned new things faster. Because the shape of the funnel stayed stable, the robot could keep adding new books without the whole library collapsing. The researchers showed that this worked for different types of robots and different kinds of tasks, proving that keeping the geometry right is the secret to a smart, ever-learning AI.

Why This Matters
This paper doesn't just say "this works." It explains why it works. It tells us that if we want AI to keep learning forever without losing its mind, we can't just throw data at it. We have to respect the shape of its knowledge. By treating the robot's brain like a curved, expanding world rather than a flat sheet, and by forcing it to move in ways that respect that curve, we can build machines that truly grow smarter over time, remembering their past while embracing their future.

=== SUMMARY ===

Technical Summary: Hyperbolic Multimodal Continual Learning (HMCL)

1. Problem Statement

Multimodal Contrastive Learning (MCL) has established itself as a dominant paradigm for aligning heterogeneous data sources (e.g., images and text) within a shared representation space. However, existing approaches predominantly embed these representations in high-dimensional Euclidean spaces, implicitly assuming a flat geometric structure. This assumption fails to capture the inherent hierarchical nature of multimodal semantics, where abstract concepts subsume specific instances, creating asymmetric and partially ordered structures.

While Hyperbolic geometry offers a principled space for encoding such hierarchies (via geodesic distances for similarity and entailment cones for hierarchy), existing hyperbolic multimodal models are developed under static learning assumptions. They do not account for Continual Learning (CL), where data distributions evolve over time as new tasks emerge. In hyperbolic spaces, standard continual updates violate geometric invariants, leading to catastrophic forgetting. Specifically, standard updates distort the semantic similarity (geodesic distances) and the semantic hierarchy (distance to the origin), causing the collapse of previously learned structures across tasks.

The core challenge addressed is: What geometric conditions must be preserved during parameter updates to prevent forgetting in hyperbolic multimodal representations?

2. Methodology: The HMCL Framework

The paper proposes HMCL (Hyperbolic Multimodal Continual Learning), a principled framework derived from a geometric analysis of representation preservation.

2.1 Theoretical Foundations

The authors establish that preventing forgetting in hyperbolic space requires satisfying three geometric preservation conditions for old-task representations ( $t-1$ ) when learning task $t$ :

Intra-modal preservation: Maintaining the Lorentzian inner products within each modality.
Inter-modal preservation: Maintaining the Lorentzian inner products between modalities (cross-modal associations).
Hierarchical preservation: Maintaining the spatial norms (distance from the origin) of embeddings, which encode the level of semantic abstraction.

Theorem 1 (Geometric Characterization): The authors prove that these three conditions hold if and only if the transformation applied to old-task representations across tasks is a shared hyperbolic isometry. Specifically, the update must act as a rotation within the spatial components ($SO(d)$) while leaving the time-like coordinate invariant. This implies that to preserve knowledge, the model must update parameters such that old embeddings are rotated but not distorted or shifted radially.

Corollary 1 (First-Order Hierarchy Preservation): At the first order, preserving the hierarchy (spatial norm) requires that the update vector $\Delta z$ be orthogonal to the spatial direction of the current embedding ( $z_{[1:d]}^\top \Delta z_{[1:d]} = 0$ ). This restricts admissible updates to the tangent space of the manifold, suppressing drift along the time-like dimension.

2.2 Admissible Parameter Updates

Based on these theorems, the authors derive a closed-form update rule for the learnable transformation layers (Lorentz transformation layers) in the model.

Constraint: The update to the spatial weight matrix ( $\Delta W_s$ ) must be orthogonal to the subspace spanned by the old-task spatial representations.
Time-like Invariance: The update to the time-like weight vector ( $\Delta w_0$ ) is constrained to be zero.
Implementation (Algorithm 1):
1. Compute a low-rank orthonormal basis $V$ of the old-task spatial representations (via SVD or PCA).
2. Project the unconstrained gradient update $\delta_s$ onto the orthogonal complement of this subspace: $\Delta W_s = \delta_s (I - VV^\top)$ .
3. Set $\Delta w_0 = 0$ .
4. Update parameters using this projected direction.

This approach introduces no additional learnable parameters and maintains constant memory overhead relative to the number of tasks, as it only requires a covariance summary of the old-task subspace.

3. Key Contributions

Theoretical Foundation: The paper provides the first geometric study of continual learning in hyperbolic multimodal spaces. It establishes that representation preservation requires cross-modal invariance under a shared hyperbolic isometry.
Geometric Analysis of Forgetting: It identifies that forgetting in this setting is driven by two factors: semantic relation drift (violation of inter/intra-modal distances) and hierarchy-related distortion (violation of radial distances).
Principled Framework (HMCL): The authors derive a geometry-consistent update rule that restricts parameter changes to directions preserving the essential geometric structure (isometry and hierarchy) while allowing adaptation to new tasks.
Empirical Validation: The framework is validated across diverse benchmarks, showing significant reductions in catastrophic forgetting compared to Euclidean-based continual learning methods adapted to hyperbolic space.

4. Experimental Results

The authors evaluate HMCL on continual multimodal benchmarks involving 15 datasets (classification and cross-modal retrieval) across three hyperbolic backbones: MERU-L, MERU-B, and HyCoCLIP-B.

Performance Improvement: HMCL consistently achieves the highest average accuracy and retrieval performance (R@K) compared to baselines (Vanilla, EWC, GEM, C-FLAT). For instance, on the MERU-L backbone, HMCL improved the overall performance metric by +8.2% over the Vanilla baseline.
Reduction in Forgetting: HMCL significantly mitigates catastrophic forgetting, measured via Backward Transfer (BWT). On MERU-L, the classification BWT improved from $-6.46$ (Vanilla) to $-0.74$, representing an 88.5% relative reduction in forgetting.
Drift Analysis: Quantitative analysis of representation drift (radial, angular, cross-modal, and paired-distance) shows HMCL reduces all four types of drift by approximately 60-74% compared to standard continual training.
Case Studies: Qualitative image traversals demonstrate that HMCL preserves the semantic hierarchy (e.g., moving from specific concepts like "hat" to generic ones like "fashion" towards the root), whereas baseline models drift into irrelevant concepts, indicating a collapse of the hierarchical structure.

5. Significance and Claims

The paper claims to provide the first systematic study of continual learning for hyperbolic multimodal representations. Its primary significance lies in shifting the perspective of continual learning from a purely optimization problem to a geometric problem.

The authors argue that standard continual learning methods (like EWC or GEM) are insufficient in hyperbolic spaces because they are "geometry-agnostic"; they do not explicitly preserve the shared cross-modal isometries or hierarchy-related constraints inherent to the Lorentz manifold. By deriving update rules that strictly adhere to the geometric invariants of hyperbolic space, HMCL achieves a balance between stability (preserving old knowledge) and plasticity (learning new tasks) that Euclidean-based methods cannot replicate in this setting.

The work concludes that preserving the geometric structure of the representation space is not merely beneficial but necessary for stable multimodal continual learning, offering a principled path forward for systems where data distributions evolve over time.