Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticism

Technical Summary: Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticism

Problem Statement
Current deep learning approaches for Multispectral and Hyperspectral Image (MS/HS) fusion face three fundamental limitations that hinder their real-world applicability and transferability:

1. Architectural Rigidity: Most models are designed with hard-coded channel dimensions, requiring separate training for sensors with different numbers of spectral bands (e.g., 31 vs. 191 bands) or costly fine-tuning.
2. Data Scarcity and Model Scale: HSI datasets are typically small and isolated. Training models on single, small datasets limits capacity and leads to overfitting, while the inability to pool data across heterogeneous sensors prevents the learning of generalizable features.
3. Fixed Spatial Scaling: Existing models are trained for specific integer upscaling factors (e.g., 4×, 8×), learning discrete grid-to-grid mappings that fail to handle arbitrary or non-integer scaling factors required in many applications.

Methodology: The SSA Framework
To address these challenges, the authors propose SSA, a universal framework designed with spectral-band and fusion-scale agnosticism. The architecture is an end-to-end, continuous representation-based fusion pipeline comprising two primary innovations:

1. Matryoshka Kernels (MK) for Spectral Agnosticism:
   Inspired by Matryoshka Representation Learning (MRL), the authors introduce the Matryoshka Kernel Layer (MKL). Instead of hard-coding input/output dimensions, the MKL utilizes a "nested" weight structure ($W_{nested}$) that supports a maximum number of channels ($C_{max}$). During the forward pass, the model dynamically slices a valid kernel ($W_{valid}$) from the nested weights based on the specific number of input channels ($C_{in}$) or desired output channels ($C_{out}$). This allows a single model to process heterogeneous spectral inputs and reconstruct outputs with varying band counts without architectural modification.

2. Implicit Neural Representation (INR) for Scale Agnosticism:
   The framework replaces discrete upsampling layers with an Implicit Neural Representation backbone. The fusion process models the High-Resolution HSI (HR-HSI) as a continuous function parameterized by a Multi-Layer Perceptron (MLP).

   • Encoding: Low-Resolution HSI (LR-HSI) and High-Resolution MSI (HR-MSI) are encoded into latent spectral and spatial codes.
   • Continuous Querying: For any target coordinate $p$ in the high-resolution space, the model queries the latent codes. It samples local spatial features and identifies the nearest spectral neighbor in the low-resolution latent space.
   • Decoding: A shared MLP decoder predicts the spectral residual vector for the query coordinate based on the latent codes and relative coordinates.
   • Reconstruction: The final HR-HSI is obtained by adding the predicted residual to the bicubic-interpolated base image. This coordinate-based approach enables reconstruction at arbitrary spatial resolutions, including non-integer scales.

Training Strategy
The authors employ a Cross-Sensor Joint Training Strategy. Rather than training separate models for each dataset, they construct mini-batches by sampling from a pool of heterogeneous datasets (e.g., CAVE, Pavia, Washington DC). Within a single mini-batch, samples share identical spatial dimensions and band counts (ensuring structural homogeneity for batch processing), but the model parameters are optimized across diverse datasets. This forces the network to learn a universal fusion mapping rather than memorizing dataset-specific features.

Key Contributions

• Matryoshka Kernels (MK): A novel architectural principle enabling native processing of arbitrary spectral band counts, facilitating joint training on heterogeneous datasets.
• Continuous Scale Modeling: The integration of INR to replace discrete scaling with a continuous function, allowing reconstruction at arbitrary spatial scales.
• Unified Framework: The SSA model achieves state-of-the-art (SOTA) performance while generalizing to unseen sensors and scales, significantly reducing the need for per-sensor retraining.

Experimental Results
Extensive experiments were conducted on seven public hyperspectral datasets (CAVE, Harvard, PaviaC, PaviaU, Chikusei, Botswana, Washington DC) covering diverse sensors and spectral ranges (31 to 191 bands).

• Performance: The single SSA model achieved SOTA performance across all in-distribution metrics (PSNR, SAM, ERGAS, SSIM) on the CAVE and PaviaC datasets.
• Generalization: In out-of-distribution evaluations (scales of 8×, 16×, 32×), SSA consistently secured the best or second-best results, outperforming specialized models trained individually for each dataset.
• Unseen Scenarios: The model demonstrated robust generalization to unseen datasets (Houston, Loukia) with different band configurations and unseen fractional scales (e.g., 3.2×, 5.7×) without retraining, outperforming baselines that rely on bicubic resizing or fixed-scale training.
• Ablation: Analysis confirmed that the Matryoshka Kernel design effectively mitigates interference from multi-dataset training, and that increasing model capacity improves reconstruction quality without sacrificing practical inference efficiency on modern GPUs.

Significance and Claims
The paper posits that SSA addresses the "trilemma" of architectural rigidity, data scarcity, and fixed spatial scaling in MS/HS fusion. By demonstrating that a single, jointly trained model can match or exceed the performance of specialized, dataset-specific models, the authors argue that their approach paves the way for future HSI foundation models. The work suggests that moving toward universal, agnostic frameworks is a viable path to overcoming the limitations of current sensor-specific deep learning methods. The authors acknowledge the computational cost of INR as a potential limitation but emphasize the robustness of their model as a foundation for general-purpose hyperspectral analysis.