Guided Lensless Polarization Imaging

1. Problem Statement

Polarization imaging captures the orientation and phase of light waves, revealing critical information about surface geometry, material composition, and stress that is invisible to standard RGB cameras. However, conventional polarization cameras are often expensive, bulky, and complex (e.g., requiring rotating filters or division-of-focal-plane sensors with resolution loss).

Lensless imaging offers a compact, low-cost alternative by replacing lenses with simple optical elements (like diffusers) and using computational reconstruction. While lensless systems have been extended to polarization, they suffer from:

• Severely ill-posed inverse problems: The measurements are highly compressed and multiplexed, encoding both spatial and polarization information.
• Limited reconstruction quality: Classical physics-based solvers (e.g., FISTA, ADMM) often fail to recover high-frequency details and are sensitive to noise and model mismatches.
• Lack of data-driven refinement: Existing lensless polarization methods rely almost exclusively on physics-based optimization, lacking the generative power of deep learning to recover fine structures.

2. Methodology

The authors propose a two-stage, RGB-guided reconstruction pipeline that combines a compact lensless polarization sensor with an auxiliary, widely available conventional RGB camera.

A. Hardware Setup

• Lensless Polarization Camera: Uses a random diffuser and a custom striped polarization mask (linear polarizers at 0°, 45°, 90°, and 135°) placed directly on the sensor. This creates a single-shot, spatio-polarimetrically multiplexed measurement.
• RGB Guidance Camera: A standard lensed RGB camera captures the same scene simultaneously. This provides high-fidelity structural and edge information to guide the reconstruction.

B. The Two-Stage Reconstruction Pipeline

The process is divided into a physics-based initialization and a data-driven refinement.

Stage 1: Physics-Based Inversion

• Goal: Recover a coarse, physically consistent initial polarization estimate ($\hat{x}$) from the raw lensless measurement ($y$).
• Model: The forward model is defined as $y = Ax + e$, where $A$ combines the diffuser PSF and the polarization mask modulation.
• Solver: The authors employ classical optimization algorithms, specifically FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) or ADMM (Alternating Direction Method of Multipliers).
• Regularization: They use a 3D Total Variation (3DTV) prior to enforce smoothness across spatial and polarization dimensions, along with non-negativity constraints.
• Output: A low-quality, blurry polarization image that lacks high-frequency details due to the diffuser's low-pass nature.

Stage 2: RGB-Guided Deep Refinement

• Goal: Refine the coarse output from Stage 1 by recovering high-frequency details using the RGB image as a structural guide.
• Architecture: A modified Swin Transformer network (adapted from SwinFuSR).
  • Dual Branches: One branch processes the initial polarization reconstruction; the other processes the registered RGB guidance image.
  • Feature Extraction: Shallow convolutional layers followed by Swin Transformer Layers (STL).
  • Fusion Mechanism: Attention-guided Cross-Domain Fusion (ACF) blocks. These blocks alternate between self-attention (within a modality) and cross-attention (between polarization and RGB modalities) to integrate structural cues from the RGB image into the polarization data.
  • Skip Connection: The initial reconstruction is added to the final output to ensure consistency with the physical forward model.
• Training Strategy:
  • Trained on synthetic data generated from existing datasets (PIP, UPLight, ZJU-RGB-P) using the measured PSF.
  • Translation Augmentation: Random shifts (up to ±4 pixels) are applied to the RGB and ground-truth pairs during training to make the model robust to imperfect alignment in real-world setups.
  • Loss Function: Combines $\ell_1$ loss and LPIPS (Learned Perceptual Image Patch Similarity) to enhance perceptual quality.

3. Key Contributions

1. First RGB-Guided Lensless Polarization System: Introduces a novel framework that integrates a low-cost lensless polarization sensor with a standard RGB camera to overcome the limitations of purely lensless systems.
2. Hybrid Two-Stage Architecture: Successfully combines physics-based solvers (ensuring physical consistency) with deep learning (ensuring high-fidelity detail recovery) via a cross-modal Transformer.
3. State-of-the-Art Performance: Demonstrates significant improvements over existing lensless-only baselines (FISTA, ADMM) and learning-based baselines (FlatNet, PolarAnything) in terms of PSNR, SSIM, and LPIPS.
4. Robustness and Generalization: The method generalizes well across different datasets (including underwater scenes) and unseen Point Spread Functions (PSFs). It achieves high-quality results on a physical prototype without fine-tuning, and further improves with minimal target-domain fine-tuning.

4. Experimental Results

• Quantitative Metrics: On the PIP, UPLight, and ZJU-RGB-P datasets, the proposed method ("Ours") significantly outperforms baselines.
  • Example (PIP, Grayscale): Achieved 35.13 PSNR and 0.97 SSIM, compared to 13.87 PSNR for FISTA alone and 28.85 PSNR for FISTA + Transformer (without RGB guidance).
  • The method also outperformed generative models like PolarAnything and FlatNet, which struggled with the specific constraints of lensless polarization reconstruction.
• Qualitative Results: Visualizations show that the RGB guidance successfully recovers sharp edges and fine textures that are lost in the diffuser-blurred initial reconstruction.
• Real-World Validation: Tested on a physical prototype with a custom polarization mask. Despite hardware imperfections (mask fabrication artifacts, sensor cover glass distance) and domain shifts, the method produced sharp, high-fidelity polarization images, whereas the unguided FISTA+Transformer baseline failed to generalize.
• Ablation Studies:
  • Cross-Attention: Replacing the complex cross-attention fusion with simple channel concatenation resulted in visible intensity artifacts, proving the necessity of the attention mechanism.
  • PSF Mismatch: The model maintained high performance even when tested with PSFs from different diffusers, demonstrating strong generalization.
  • Initialization: The method remained robust even when the initial physics-based reconstruction used fewer iterations (weak initialization), highlighting the power of the RGB guidance.

5. Significance

This work bridges the gap between compact, low-cost hardware and high-quality imaging. By leveraging the ubiquity of RGB cameras as structural guides, the authors overcome the fundamental limitations of lensless polarization imaging (ill-posedness and loss of detail).

• Practical Impact: The system is highly suitable for size- and cost-constrained applications such as compact microscopy, endoscopy, autonomous driving, and remote sensing, where traditional polarization cameras are too bulky or expensive.
• Methodological Advance: It establishes a new paradigm for inverse problems in imaging, showing that a hybrid approach (Physics + Deep Learning + Cross-Modal Guidance) yields superior results compared to purely data-driven or purely physics-based methods.
• Open Science: The authors provide code, pretrained models, and data, facilitating further research in lensless and polarization imaging.