SuperF: Neural Implicit Fields for Multi-Image Super-Resolution

Technical Summary: SuperF: Neural Implicit Fields for Multi-Image Super-Resolution

1. Problem Statement

High-resolution imagery is frequently constrained by sensor limitations, atmospheric interference, and acquisition costs, affecting domains ranging from satellite remote sensing to smartphone photography. While Single-Image Super-Resolution (SISR) methods attempt to solve this inverse problem using strong priors learned from high-resolution (HR) datasets, they often produce "hallucinated" structures that diverge from reality. This is particularly problematic for scientific and medical applications where fidelity is critical.

Multi-Image Super-Resolution (MISR) offers an alternative by leveraging multiple low-resolution (LR) frames captured with sub-pixel shifts. These shifts introduce varying aliasing artifacts that, when combined, contain complementary information to reconstruct the underlying HR signal. However, existing MISR approaches often rely on supervised learning requiring large paired LR-HR datasets (which are expensive to acquire) or test-time optimization (TTO) methods that may struggle with accurate sub-pixel alignment or require complex preprocessing steps like explicit registration.

2. Methodology

The authors propose SuperF, a test-time optimization (TTO) approach that utilizes Implicit Neural Representations (INRs) to solve the MISR problem without requiring any HR training data.

Core Architecture:

• Shared INR: The method employs a coordinate-based Multi-Layer Perceptron (MLP) that acts as a continuous function mapping 2D coordinates to RGB intensities. This MLP is shared across all input LR frames, effectively representing the underlying continuous HR signal.
• Joint Optimization: Unlike prior methods that treat alignment and reconstruction separately, SuperF jointly optimizes:
  1. The parameters of the shared MLP (the HR signal).
  2. The sub-pixel alignment of each LR frame relative to a base frame.
• Affine Transformation Parameterization: Instead of using a separate network to estimate transformation matrices (as done in similar burst fusion works), SuperF directly parameterizes the affine transformation for each frame using learnable translation ($\Delta x, \Delta y$) and rotation ($\alpha$) parameters. The base frame is fixed as the reference coordinate system ($A^{(1)} = I$).
• Spectral Projection: To handle brightness and contrast variations between frames, the model includes a frame-specific spectral projection layer (scale and shift per spectral band), which is optimized alongside the alignment.

Key Technical Components:

• Supersampling Strategy: To overcome the spectral bias of coordinate-based MLPs (which tend to learn low frequencies first), the model is optimized on a high-resolution coordinate grid corresponding to the target output resolution. The output is then downsampled (via average pooling/boxcar filter) to match the LR input frames for loss calculation. This forces the network to learn high-frequency details required to satisfy the LR constraints.
• Positional Encoding: The MLP input coordinates are transformed using Fourier features (random sine/cosine basis functions) to enable the network to represent high-frequency signals effectively. The scale of these features is domain-specific (e.g., $\sigma=10$ for satellite, $\sigma=3$ for ground-level).
• Uncertainty Estimation: To handle noise (e.g., clouds in satellite imagery or sensor noise), the MLP outputs a per-pixel uncertainty estimate (log-variance) alongside the RGB values. The optimization minimizes the Gaussian Negative Log-Likelihood (GNLL) loss, allowing the model to increase variance estimates for noisy pixels rather than forcing a perfect reconstruction, thereby improving robustness.

3. Key Contributions

1. SuperF Algorithm: A novel TTO method for MISR that shares an INR across multiple shifted LR frames while jointly optimizing sub-pixel alignment. The authors demonstrate that directly parameterizing affine transformations and optimizing on a supersampled grid is crucial for adapting INRs to MISR.
2. Improved Alignment and Representation: By directly parameterizing transformations and using supersampling, the method achieves superior sub-pixel alignment and a more continuous representation of the HR signal compared to existing INR baselines.
3. SatSynthBurst Dataset: The introduction of a synthetic satellite burst dataset derived from open HR images, featuring controlled sub-pixel shifts and noise, to facilitate MISR research.
4. Domain Generalization: The approach is validated on both satellite imagery and ground-level handheld camera bursts, demonstrating robustness across different domains without requiring domain-specific pretraining.

4. Experimental Results

The method was evaluated on SatSynthBurst (synthetic satellite) and SyntheticBurst (ground-level handheld) datasets with upsampling factors of $\times2$, $\times4$, and $\times8$.

• Performance: SuperF outperforms state-of-the-art TTO baselines, including steerable kernel regression (Lafenetre et al., 2023) and a modified INR burst fusion approach (NIR, Nam et al., 2022).
  • On SatSynthBurst, SuperF (GNLL loss) achieved a PSNR of 37.26 ($\times2$), 34.03 ($\times4$), and 29.28 ($\times8$), significantly surpassing the bilinear baseline and all other methods.
  • On SyntheticBurst, SuperF also consistently outperformed baselines, achieving 29.48 ($\times2$), 27.47 ($\times4$), and 26.58 ($\times8$) PSNR.
• Ablation Studies:
  • Removing the joint alignment optimization resulted in performance drops comparable to or worse than bilinear upsampling, confirming that sub-pixel alignment is critical.
  • Direct parameterization of affine transformations proved more effective than using a separate MLP to estimate them.
  • The supersampling strategy was shown to be essential for reducing alignment error and improving reconstruction quality.
• Real-World Application: The method was successfully applied to real Sentinel-2 satellite images, handling varying cloud cover and noise levels. The GNLL loss demonstrated specific robustness to cloudy pixels, effectively ignoring them during optimization.

5. Significance and Claims

The paper positions SuperF as a significant advancement in MISR by addressing the reliance on large HR training datasets and the risk of hallucination inherent in supervised SISR methods.

• Training-Free: As a TTO method, SuperF requires no offline training on HR data, making it immediately applicable to new domains (e.g., any location on Earth) and reducing the risk of generating non-existent structures.
• Robustness: The integration of uncertainty estimation allows the model to handle real-world noise (clouds, lighting changes) more effectively than standard MSE-based approaches.
• Limitations: The authors acknowledge that the iterative optimization process takes several seconds per image, which may limit real-time mobile applications but is acceptable for remote sensing and scientific analysis. They also note that the method assumes the observed frames depict the same scene; drastic scene changes or heavy occlusions remain challenging, though the uncertainty module offers some mitigation.

In conclusion, SuperF leverages the continuous nature of neural fields to unify frame alignment and super-resolution reconstruction, offering a robust, data-efficient solution for enhancing resolution in multi-image scenarios.