MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration

1. Problem Statement

Real-world image acquisition often results in pairs of images that suffer from two simultaneous challenges:

1. Severe Degradation: Images may be blurry, noisy, or low-resolution (Low-Quality, LQ).
2. Viewpoint Shifts: Images are captured from different angles or poses, requiring geometric alignment.

The Core Dilemma:

• Restoration (e.g., denoising, super-resolution) typically requires a clean reference or assumes a single image input. It struggles when the reference image is misaligned.
• Matching (estimating geometric correspondences) relies on clear features. Severe degradation (noise, blur) hinders the ability of matching networks to find accurate correspondences.
• Current Limitations: Existing methods usually treat these tasks independently. Joint approaches exist but are mostly limited to video or burst photography with very small baselines (small viewpoint changes). There is a lack of solutions for wide-baseline image pairs with uncorrelated degradations, largely due to the difficulty of acquiring ground-truth training data (clean images, exact transformation fields, and degradation kernels) for such complex scenarios.

2. Methodology: MATRES Framework

The authors propose MATRES, a Zero-Shot Test-Time Adaptation framework. It does not require offline training or additional datasets. Instead, it adapts pre-trained models on-the-fly using only a single input pair of a Low-Quality (LQ) and a High-Quality (HQ) image.

Core Architecture

The system consists of three components:

1. Pretrained Matching Network ($M_\theta$): Extracts generative priors (features) and estimates the geometric transformation ($T_{LQ \to HQ}$) between the LQ and HQ images. Models like RoMa, DIFT, or Mast3R can be used.
2. Pretrained Restoration Network ($R_\phi$): A standard restoration model (e.g., EDSR, SwinIR, Restormer) with an encoder-decoder structure.
3. Lightweight Adapter ($H_\psi$): A learnable, zero-initialized projection network (based on LoRA) that bridges the feature spaces of $M_\theta$ and $R_\phi$.

The Adaptation Process (Iterative Optimization)

The process occurs entirely at inference time:

1. Feature Extraction & Warping:
   • $M_\theta$ extracts generative features ($z_{LQ}, z_{HQ}$) from both images.
   • It estimates the transformation $T_{LQ \to HQ}$.
   • The LQ image and its features are warped to align with the HQ image.
2. Feature Adaptation:
   • The warped features are passed through the learnable adapter $H_\psi$.
   • $H_\psi$ modulates the features to align the "generative priors" from the matcher with the internal feature space of the restorer.
3. Restoration:
   • The restoration network $R_\phi$ generates an enhanced image ($I_{EQ}$) using the warped LQ image and the HQ reference, conditioned on the adapted features.
4. Loss Calculation & Update:
   • The system computes a Cost Volume between the restored image features and the HQ features.
   • Two loss terms guide the optimization of $H_\psi$ (while keeping $M_\theta$ and $R_\phi$ frozen):
     • Off-Diagonal Loss ($L_D$): Encourages the cost volume to be diagonal-dominant, ensuring sharp, unique correspondences (penalizing ambiguous matches).
     • Pixel-Matching Loss ($L_P$): Minimizes the pixel difference between the restored image ($I_{EQ}$) and the HQ reference ($I_{HQ}$) in overlapping regions.
   • The adapter parameters $\psi$ are updated via gradient descent.

3. Key Contributions

• Joint Optimization without Offline Training: MATRES is the first framework to effectively address simultaneous matching and restoration for wide-baseline image pairs using a zero-shot test-time approach, eliminating the need for ground-truth training data.
• Mutual Guidance Mechanism: It establishes a feedback loop where:
  • The Matching Network provides spatial alignment and generative priors to guide the restoration.
  • The Restoration Network provides cleaner, higher-fidelity features that improve the accuracy of the matching network.
• Plug-and-Play Compatibility: The framework is model-agnostic. It works with various state-of-the-art matching models (Diffusion-based, Auto-Regressive, Vision Transformers) and restoration models (Super-Resolution, Denoising, Deblurring).
• Efficiency: Only a lightweight adapter ($H_\psi$) is trained during inference, preserving the integrity and zero-shot capability of the large pre-trained backbones.

4. Experimental Results

The authors evaluated MATRES across three restoration tasks (Super-Resolution, Denoising, Deblurring) and various matching models on diverse benchmarks (RealSR, Set5, GoPro, HIDE, etc.).

• Restoration Performance:
  • MATRES significantly outperforms standalone restoration networks. For example, in Super-Resolution, PSNR improvements of +3.5 to +5.9 dB were observed when using the matching guidance compared to using the restorer alone.
  • It achieves competitive or superior results compared to task-specific State-of-the-Art (SOTA) methods (e.g., AutoLUT, CATANet, MRPNet) that do not utilize geometric guidance.
• Matching Performance:
  • The mutual guidance leads to substantial gains in geometric alignment accuracy (mAUC).
  • Matching scores improved by +3 to +9 points (mAUC) across different tasks compared to using the matching network alone.
  • Qualitative results show that the estimated transformation fields align much closer to the ground truth when the restoration network is active.
• Robustness: The method consistently improves performance across different degradation types (blur, noise, low light) and different combinations of matching/restoration architectures.

5. Significance

• Practical Real-World Application: The framework addresses a common user scenario: capturing multiple photos of a scene with varying quality and angles. It provides a solution to recover high-quality, aligned images without requiring a pre-trained dataset specific to the user's environment.
• Paradigm Shift: It challenges the traditional separation of "restoration" and "matching" pipelines. By showing that these tasks are mutually beneficial rather than interfering, it opens new avenues for unified multi-view enhancement.
• Zero-Shot Generalization: By relying on test-time adaptation rather than offline training, MATRES offers a robust solution for domains where collecting paired ground-truth data (clean image + transformation) is impossible or prohibitively expensive.

In conclusion, MATRES demonstrates that leveraging the mutual dependency between geometric alignment and image quality can yield superior results in both tasks, offering a practical, training-free solution for complex real-world image processing.