Transfer learning RGB models to hyperspectral images with trainable tensor decompositions

Technical Summary: Transfer Learning RGB Models to Hyperspectral Images with Trainable Tensor Decompositions

Problem Statement
Hyperspectral imaging (HSI) captures data across hundreds of spectral wavelengths, offering rich information for tasks in remote sensing, medical imaging, and horticulture. However, training deep learning models on HSI is hindered by the scarcity of large, diverse datasets compared to RGB imagery. Consequently, transfer learning from pre-trained RGB backbone models (e.g., trained on ImageNet) is a desirable strategy. Yet, a fundamental incompatibility exists: standard backbone models expect 3-channel RGB inputs, while hyperspectral images have significantly higher channel dimensions ($\hat{C}_{in} > 3$).

Existing solutions to this dimensionality mismatch suffer from significant trade-offs:

1. Image-side adaptation: Reducing hyperspectral images to 3 channels via dimensionality reduction (e.g., PCA) or channel selection preserves the model architecture but discards crucial spectral information.
2. Model-side adaptation: Replacing the first convolutional layer with one having $\hat{C}_{in}$ input channels preserves the image data but requires training new weights from scratch. This sacrifices the pre-learned spatial patterns of the original filters, increases the number of trainable parameters, and heightens the risk of overfitting on small HSI datasets.

Methodology
The authors propose a novel approach that bridges this gap by utilizing partially trainable tensor decompositions. The core insight is that convolutional filters in backbone models can be decomposed into spatial and spectral components. By separating these, the method can retain the spatial patterns learned from large RGB datasets while adapting the spectral components to the specific hyperspectral domain.

The process involves the following steps:

1. Decomposition: The weights of the first convolutional layer ($W^1$) of a pre-trained backbone are decomposed using either CP (Canonic Polyadic) or Tucker tensor decompositions.
   • CP Decomposition: Approximates the filter as a sum of rank-one tensors, separating the filter into one spectral vector and two spatial vectors (horizontal and vertical).
   • Tucker Decomposition: Projects the tensor into a lower-dimensional core tensor via matrix multiplications. The authors apply a partial Tucker decomposition, compressing only the spectral dimension while preserving the spatial structure.
2. Component Replacement: The original 3-channel spectral components are discarded. They are replaced with new, trainable spectral components of dimension $\hat{C}_{in}$ (matching the hyperspectral input).
3. Freezing and Training: The spatial components (derived from the original pre-trained filters) are frozen. Only the new spectral components and the final classification layer are trained.
4. Implementation: The decomposed layers are implemented using grouped separable convolutions. For CP, this involves a pointwise convolution followed by two depthwise convolutions; for Tucker, a pointwise convolution followed by a standard 2D convolution.

Key Contributions

• Novel Transfer Mechanism: The paper introduces a method to adapt 3-channel RGB backbones to hyperspectral inputs without sacrificing the pre-trained spatial information or discarding spectral data.
• Parameter Efficiency: By freezing the spatial components, the method drastically reduces the number of trainable parameters compared to retraining the first layer from scratch, thereby mitigating overfitting risks in data-scarce scenarios.
• Preservation of Spatial Priors: The approach maintains the "colored, elliptical shapes" and edge-detection behaviors characteristic of pre-trained backbone filters, which are crucial for robust vision tasks.

Experimental Results
The authors evaluated their method on five datasets: three remote sensing datasets (Botswana, Indian Pines, Kennedy Space Center) and two horticultural datasets (Avocado, Grape leaves). They compared their CP and Tucker approaches against two baselines:

• Reduce: Dimensionality reduction to 3 channels.
• Scratch: Replacing the first layer with a new layer trained from scratch.

Findings:

• Accuracy: The proposed CP and Tucker methods consistently outperformed the "Reduce" baseline. They were generally more accurate than the "Scratch" baseline, particularly on complex datasets like 'Grape leaves' where "Scratch" performed poorly.
• Robustness: In overfitting experiments (varying learning rates and training for extended epochs), the "Scratch" method overfit significantly faster than the decomposed approaches. The decomposed methods demonstrated greater stability.
• Backbone Independence: The results held across different backbone architectures (AlexNet, DenseNet, ResNet18).
• Rank Sensitivity: While a decomposition rank of $R=2$ generally provided the best balance, increasing the rank to $R=3$ did not consistently yield further improvements, suggesting the need for hyperparameter tuning.

Significance and Claims
The paper claims that this approach successfully resolves the dimensionality mismatch between hyperspectral images and RGB backbones while preserving both the image's spectral information and the model's spatial priors. The authors assert that their method is more accurate and more robust against overfitting than existing hyperspectral transfer learning approaches that either compress the image or retrain the model from scratch.

The work positions itself as a step toward creating universal backbones for hyperspectral imaging. However, the authors remain modest regarding limitations, noting that tensor-decomposed layers can be sensitive to initialization and optimization strategies. They suggest that future work could focus on optimization strategies tailored specifically for these layers and exploring methods that might eventually allow for transfer learning without backpropagating through the entire network.