💻 computer science

Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

This paper proposes a manifold-constrained optimization framework (DLRT) to compress large vision transformer-based geospatial foundation models during transfer learning, achieving significant parameter reduction with minimal accuracy loss while outperforming standard low-rank methods like LoRA for efficient edge deployment.

Original authors: Thomas Snyder, H. Lexie Yang, Stefan Schnake, Steffen Schotthöfer

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thomas Snyder, H. Lexie Yang, Stefan Schnake, Steffen Schotthöfer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Big to Fit in Your Pocket

Imagine you have a massive, incredibly smart library (a Geospatial Foundation Model) that knows everything about the Earth from space. It can identify forests, cities, and disaster zones just by looking at satellite photos.

However, this library is so huge it fills a skyscraper. You can't carry it in your pocket, and you can't run it on a small drone or a handheld device in the field because those devices have tiny batteries and weak processors.

The usual way to fix this is to try to shrink the library. But if you just cut out pages randomly to make it smaller, you lose important information, and the library stops being smart. The paper asks: How do we shrink this giant library so it fits in a pocket, without losing its ability to read the map?

The Solution: The "Manifold-Constrained" Folding Technique

The authors propose a new way to shrink these models called Manifold-Constrained Optimization (specifically using a method called DLRT).

Think of the model's knowledge as a giant, complex 3D sculpture.

Old Methods (like LoRA): Imagine trying to flatten this sculpture by just pressing it flat from the top. It gets smaller, but the details get squished and distorted.
The New Method (DLRT): Imagine the sculpture is made of a flexible, magical fabric. Instead of just squishing it, this method finds the specific "folds" in the fabric that hold the most important information. It carefully folds the fabric along these lines, keeping the shape of the most critical parts intact while removing the empty air in between.

In technical terms, the model is "compressed" by forcing its internal math to stay on a specific, low-dimensional path (a manifold). This ensures that when the model learns new tasks (like identifying a specific type of tree), it only updates the parts of the model that matter, keeping the rest compact.

The Experiment: Testing the Folded Library

The researchers tested this on three different "puzzles" (datasets) involving satellite images:

UCM: A dataset of US cities.
AID: A dataset of aerial images with many different scenes.
NWPU: A massive global dataset with 45 different categories.

They took two types of giant libraries:

ImageNet-trained models: Models trained on general photos (like cats and dogs).
OReole models: Models specifically trained on satellite imagery.

They tried to shrink these libraries using their new "folding" method (DLRT) and compared it to the current popular method (LoRA).

The Results: Smaller, but Just as Smart

The findings were very clear:

Massive Size Reduction: The new method successfully removed between 62% and 82% of the model's size. It's like turning a skyscraper into a small house.
Accuracy Stays High: Even after shrinking the model so much, it still got the right answer almost as often as the giant, unshrunk version.
- On the city dataset (UCM), the new method was almost identical to the giant version (only a tiny 0.5–1% drop in accuracy).
- On the harder, more complex datasets, the new method still performed better than the standard shrinking method (LoRA).
The "Warm-Up" Trick: They discovered that for the satellite-specific models (OReole), it helped to let the model run normally for just one round before starting the shrinking process. This "warm-up" helped the model get ready to be folded without breaking.

Why This Matters

The paper concludes that this method allows us to take these massive, powerful Earth-observing AI models and actually run them on edge devices (like drones, satellites, or handheld sensors) that have limited memory and power.

Instead of needing a supercomputer in the cloud to analyze a disaster zone, a local device could now run a highly accurate, compressed version of the model instantly. The paper proves that you don't have to choose between "small size" and "high intelligence"; with this specific folding technique, you can have both.

Technical Summary: Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

Problem Statement

The deployment of geospatial foundation models (FMs) on resource-constrained edge devices faces a critical bottleneck: while large-scale Vision Transformer (ViT) architectures achieve strong generalization across tasks like land-cover mapping and disaster response, their massive parameter counts (hundreds of millions to billions) and computational demands make them impractical for local or edge deployment. Standard model compression methods often introduce an ambiguous trade-off, where significant parameter reduction leads to unacceptable losses in downstream task accuracy and calibration. Existing low-rank adaptation methods, such as LoRA, offer a pathway for compression but may not fully preserve performance when applied to complex geospatial foundation models, particularly those utilizing self-supervised pretraining.

Methodology

The paper proposes the application of Dynamical Low-Rank Training (DLRT), a manifold-constrained optimization framework, to compress ViT-based geospatial foundation models during supervised transfer learning.

Core Framework

The method treats the weight matrices of the neural network layers as elements evolving on a low-rank manifold $\mathcal{M}_r$ . Instead of updating full-rank weights, the optimization restricts updates to a low-dimensional subspace defined by a factorization $W = USV^\top$ , where $U, V \in \mathbb{R}^{n \times r}$ and $S \in \mathbb{R}^{r \times r}$ . The DLRT process consists of three iterative steps:

Basis Augmentation: At each optimization step, the current orthonormal bases $U_t$ and $V_t$ are augmented with the gradient dynamics ( $\nabla_U \mathcal{L}, \nabla_V \mathcal{L}$ ). This creates a temporary subspace of rank $2r$ that contains the current basis and a first-order approximation of the next step, ensuring the loss does not increase during the augmentation phase.
Latent Space Training: The training dynamics are projected onto this augmented latent space. The latent coefficients $\tilde{S}$ are updated via stochastic gradient descent (or similar optimizers) for $s^*$ local iterations. Crucially, the full weight matrix is never reconstructed; only the low-rank factors are updated.
Truncation (Retraction): The updated solution is retracted back onto the manifold $\mathcal{M}_r$ (or a new rank $r_1$ ) using a truncated Singular Value Decomposition (SVD). This step discards the smallest singular values, effectively compressing the representation while maintaining the manifold constraint.

Experimental Setup

The study evaluates this approach on two families of geospatial foundation models:

ImageNet-21k Pretrained ViTs: Standard ViT-B16 and ViT-H14 models.
OReole-MR FMs: Geospatial-specific models pretrained via Masked Autoencoder (MAE) on the unlabeled Million-AID dataset.

The models were fine-tuned on three challenging remote sensing benchmarks with small train-to-validation ratios: UCM (21 classes), AID (30 classes), and NWPU (45 classes). The DLRT method was compared against:

Full-rank baseline transfer learning.
Standard LoRA (Low-Rank Adaptation).

A specific procedural finding was that OReole-MR models required a single "warm-up" epoch of standard uncompressed transfer learning before applying low-rank adaptation to achieve optimal results, a step not necessary for ImageNet-pretrained models.

Key Results

Experiments across 10 runs per configuration yielded the following findings:

Accuracy Retention: DLRT consistently preserved accuracy within 0.5–1% of the full-rank baseline on the UCM dataset while removing 64–82% of parameters.
Comparison with LoRA: On more challenging datasets (AID and NWPU) with smaller data ratios, low-rank methods generally suffered larger accuracy drops than the baseline. However, DLRT outperformed LoRA by 1–6% in validation accuracy across these datasets.
Compression Ratios: DLRT achieved compression ratios (c.r.) ranging from approximately 62% to 82%, significantly reducing parameter counts while maintaining high performance.
Model Sensitivity: OReole-MR based foundation models achieved the highest absolute validation accuracies but were found to be harder to compress into low-rank formats compared to ImageNet-based models. This suggests a misalignment between self-supervised pretraining schemes and the low-rank subspace, which the warm-up strategy helped mitigate.
Hyperparameter Robustness: The study found that learning rate was the most critical hyperparameter, while DLRT-specific parameters (augmentation threshold $\tau$ and local iterations $s^*$ ) showed weak correlation with performance, indicating robustness to their selection.

Significance and Claims

The paper claims that manifold-constrained optimization via DLRT offers a principled approach to compressing geospatial foundation models that outperforms "off-the-shelf" low-rank methods like LoRA.

Theoretical Guarantees: The method leverages the geometric structure of the low-rank manifold to provide provable approximation and descent guarantees. The orthogonality of the bases and the manifold constraint enable error control relative to full-rank training, ensuring the projected gradient flow remains close to the original gradient flow.
Practical Deployment: By achieving substantial parameter reduction with minimal accuracy loss, the approach enables the deployment of high-performing geospatial models on edge devices with limited memory and compute resources.
Adversarial Robustness: The framework is noted to be compatible with regularization schemes acting on the low-rank latent space, a property identified as important for improving adversarial robustness in geospatial applications.

The authors conclude that while the method is effective, future work is needed to better align self-supervised pretraining schemes with low-rank subspaces to further improve compression efficiency for models like OReole-MR.