Technical Summary: GeoStack – A Framework for Quasi-Abelian Knowledge Composition in VLMs

1. Problem Statement

Vision-Language Models (VLMs), particularly those based on CLIP, face significant challenges when adapting to multiple domains or learning new tasks sequentially. Traditional approaches suffer from two primary limitations:

• Catastrophic Forgetting: Sequential fine-tuning or joint training on disparate data distributions often causes the model to overwrite foundational knowledge, leading to a degradation in performance on previously learned tasks.
• Inefficiency of Existing Composition Methods: While adapter-based methods (e.g., Task Arithmetic, BiCLIP) allow for independent training, they lack robust mechanisms for stable composition. Naive stacking of domain-specific adapters often distorts the decision boundaries of previous domains, and linear summation of task vectors (Task Arithmetic) fails to preserve the geometric integrity of the feature space as the number of tasks increases.

The paper identifies a gap in the literature: there is no existing framework that allows for the independent training of domain experts, their modular composition into a unified model, and the preservation of foundational knowledge without requiring joint retraining or incurring additional inference latency.

2. Methodology: GeoStack Framework

GeoStack (Geometric Stacking) is a modular framework designed to compose independently trained domain experts into a single multi-expert model. It builds upon BiCLIP, which uses a learnable geometric transformation matrix $W$ to adapt image features for specific domains. GeoStack extends this by imposing strict geometric constraints to ensure stability during composition.

2.1 Geometric Constraints and Stackability

To enable stable composition, GeoStack restricts the learnable adapter matrices ($W$) to the set of upper-triangular matrices ($U$). Since the set of upper-triangular matrices is closed under multiplication, the product of any number of adapters remains a valid upper-triangular matrix.

The framework relies on two key theoretical pillars:

1. Perturbation Prior: Each adapter is initialized as the identity matrix ($I$) and learned as a small perturbation ($W = I + \Delta$). This ensures that the transformation is a minor deviation from the original CLIP feature space.
2. Quasi-Additive and Quasi-Abelian Composition: When perturbations are small, the multiplicative composition of adapters approximates an additive operation ($W_{total} \approx I + \sum \Delta_i$). Crucially, this property renders the composition Quasi-Abelian, meaning the order of stacking experts has negligible impact on the final performance, eliminating the need for combinatorial optimization.

2.2 The GeoLayer and COA Loss

The core component of GeoStack is the GeoLayer, an adapter trained with a dual objective to balance domain adaptation and stability:

• Alignment Objective: Uses InfoNCE contrastive loss to align image and text features for the specific domain.
• Stackability Objective: Minimizes the Orthogonality Loss ($L_{ortho} = \|W^T W - I\|_F^2$), which bounds the spectral norm of the perturbation ($\|\Delta\|_2$).

These are combined into a Convex Orthogonality Alignment (COA) Loss:
$$L_{COA} = (1 - \lambda)L_{align} + \lambda L_{ortho}$$
Where $\lambda$ controls the trade-off between domain-specific performance and the preservation of foundational knowledge.

2.3 Weight Folding and Inference Efficiency

A distinct feature of GeoStack is the Weight-Folding Property. Since the visual projection head in CLIP is a matrix $P$, and each GeoLayer is a matrix $W$, the entire stack of $n$ experts can be pre-computed into a single effective projection matrix:
$$P_{eff} = P \cdot \prod_{k=0}^{n-1} W_k$$
This allows the composed model to perform inference with constant time complexity ($O(1)$), regardless of the number of integrated experts, incurring zero additional latency or memory footprint during deployment.

3. Key Contributions

1. The GeoStack Framework: A modular architecture for VLM knowledge composition that enables independent training of domain experts and their stable integration without joint retraining.
2. Theoretical Foundations of Stackability: The derivation of geometric constraints (upper-triangular closure and perturbation minimization) that ensure margin stability and prevent the erosion of foundational knowledge.
3. Quasi-Abelian Property: The demonstration that GeoStack composition is largely order-invariant, allowing for flexible integration of experts without combinatorial search.
4. Weight-Folding Mechanism: The proof that multiple adapters can be folded into a single matrix, achieving $O(1)$ inference complexity.
5. Empirical Validation: Extensive experiments demonstrating GeoStack's superiority in Multi-Domain Adaptation (MDA) and Class-Incremental Learning (CIL) compared to Task Arithmetic and naive BiCLIP stacking.

4. Experimental Results

The authors evaluated GeoStack on Multi-Domain Adaptation (MDA) and Class-Incremental Learning (CIL) using datasets including ImageNet, Caltech-101, Food-101, EuroSAT, DTD, and CIFAR-100.

4.1 Multi-Domain Adaptation (MDA)

In a "Quad-Stack" (4-domain) configuration, GeoStack significantly outperformed baselines:

• Foundational Preservation: In a "Hard Stack" (general to specialized domains), Task Arithmetic (TA) with BiCLIP adapters caused ImageNet accuracy to drop from 66.6% (Zero-Shot) to 52.6%. GeoStack maintained ImageNet accuracy at 62.8%, demonstrating superior preservation of foundational knowledge.
• Average Performance: GeoStack achieved the highest average accuracy across all domains (e.g., 84.0% in the Easy Stack vs. 73.2% for Zero-Shot and 70.5% for TA-Geo).
• Order Invariance: Experiments with different stacking permutations showed negligible variance in accuracy (e.g., standard deviation of 0.04% on ImageNet), confirming the Quasi-Abelian property.

4.2 Class-Incremental Learning (CIL)

In CIL experiments on CIFAR-100 (learning 4 tasks of 25 classes each):

• Catastrophic Forgetting: The BiCLIP baseline suffered severe forgetting, dropping from 86.20% (Task 0) to 60.08% after 4 tasks (an 8% drop below Zero-Shot). GeoStack maintained 69.47% accuracy, outperforming the baseline by 9.39%.
• Retention: GeoStack limited the loss of original knowledge (Task-0 retention) to 2.12%, compared to 14.16% for BiCLIP and 3.92% for Task Arithmetic.
• Deep Stacking: In a 10-task stress test, GeoStack retained 56.00% accuracy on the initial task, whereas BiCLIP collapsed to 21.50%.

4.3 Sensitivity Analysis

The parameter $\lambda$ in the COA loss was found to be critical. Increasing $\lambda$ (towards 0.99) exponentially reduced the Orthogonality Error (from 0.0332 to 0.0078) with only a marginal impact (2.6%) on domain-specific accuracy, confirming that high $\lambda$ is essential for stackability.

5. Significance and Claims

The paper claims that GeoStack provides a mathematically grounded solution to the trade-off between knowledge accumulation and catastrophic forgetting in VLMs. Its significance lies in:

• Scalability: The ability to integrate an arbitrary number of domain experts without increasing inference complexity ($O(1)$).
• Modularity: The capability to train experts independently on isolated data, removing the need for cross-domain data access or complex hyperparameter tuning for joint training.
• Stability: The use of geometric constraints (upper-triangular matrices and orthogonality) to ensure that the composition of new knowledge does not degrade the model's original capabilities.

The authors conclude that GeoStack effectively mitigates catastrophic forgetting while maintaining the performance of domain-specific models, offering a practical mechanism for long-term knowledge composition in Vision-Language Models. The framework is presented as a step toward more robust, adaptable, and efficient VLMs capable of lifelong learning.