Technical Summary: UniISP

Problem Statement

Current imaging pipelines face a dichotomy between human visual perception and machine vision requirements. Traditional Image Signal Processing (ISP) pipelines are manually designed to produce visually pleasing RGB images for human consumption. However, these operations (e.g., denoising, sharpening, tone mapping) often introduce compression artifacts and information loss, degrading the performance of downstream computer vision tasks, particularly in challenging conditions like low-light environments.

Conversely, methods designed for machine vision often bypass traditional ISPs to utilize raw sensor data directly or employ jointly trained ISP networks. While these approaches improve task-specific metrics (e.g., object detection accuracy), the resulting RGB images frequently fail to meet human aesthetic standards, appearing unvisualizable or lacking in color fidelity. There is a lack of dedicated optimization frameworks that simultaneously satisfy the dual requirements of generating high-quality images for human viewing while preserving information integrity for machine interpretation.

Methodology

The paper proposes UniISP, a lightweight, end-to-end neural ISP framework based on a U-Net architecture. UniISP is designed to process raw sensor data into sRGB images that are simultaneously optimized for human visual quality and downstream machine vision tasks. The framework consists of three primary components:

1. Hybrid Attention Module (HAM)

To ensure high-quality image reconstruction suitable for human perception, UniISP employs a Hybrid Attention Module as its backbone.

• Channel-wise Self-Attention: Inspired by Restormer, the module applies self-attention along the channel dimension rather than the spatial dimension. This reduces computational complexity from $O((H \cdot W)^2)$ to $O(C^2)$, making global interactions feasible for high-resolution images.
• Dynamic Feature Enhancement: Unlike static attention mechanisms, the channel-wise self-attention dynamically amplifies informative channels and suppresses noisy ones based on context, which is crucial for RAW-to-RGB conversion.
• Relative Positional Encoding (RPE): RPE is integrated to preserve structural information by providing cues about the relative positions of channels, enhancing the modeling of both local and long-range dependencies.
• Supervised Learning: The HAM is trained with supervised learning to ensure the generated images align with human visual preferences.

2. Feature Adapter

To bridge the gap between the ISP stage and downstream computer vision networks (which are typically pre-trained on RGB data), a Feature Adapter is introduced.

• Frequency-Domain Fusion: The adapter utilizes a FreqFusion module to aggregate multi-scale encoder outputs (D1–D4). It separates features into low-frequency (global context) and high-frequency (fine-grained details) components.
• Multi-Stage Integration: The fused features are progressively merged into the backbone of the downstream network via concatenation and residual connections. This mechanism allows the network to flexibly incorporate RAW-domain information, reducing domain shift and facilitating the transfer of pre-trained weights.

3. Adaptive Training Framework

UniISP employs a joint optimization strategy to balance human vision and machine vision objectives.

• Loss Function: A total loss $L_{total}$ is defined as a weighted sum of human vision loss ($L_{human}$) and machine vision loss ($L_{machine}$).
• Dynamic Weighting: The weighting factor $\lambda$ is adaptively computed at each training step using Exponential Moving Averages (EMA) of the losses. If the human vision loss is high, $\lambda$ decreases to emphasize machine performance, and vice versa.
• Alignment Handling: For unaligned datasets (where RAW and reference RGB images come from different cameras), the framework incorporates a Global Color Mapping (GCM) module and a robust optical flow consistency mask. This mask filters out inaccurate alignment areas (e.g., occlusions, repeating patterns) to ensure reliable supervision.

Key Contributions

1. Unified Framework: The proposal of UniISP, a learnable, lightweight ISP framework capable of simultaneously meeting the needs of human visual perception and machine vision tasks.
2. Hybrid Attention Module (HAM): The introduction of a novel attention mechanism combining self-attention with classic convolutional attention. This design leverages different attention types to enhance feature representation and generalization while maintaining computational efficiency.
3. Feature Adapter: The development of a module that integrates and adapts multi-scale features from the ISP stage to downstream networks, enriching the model's understanding and improving downstream task performance without sacrificing visual quality.
4. Comprehensive Evaluation: Extensive experiments across multiple tasks (RAW-to-RGB mapping, object detection, semantic segmentation) and datasets demonstrate the framework's effectiveness and generalization capabilities.

Experimental Results

The paper evaluates UniISP on three main tasks using various datasets (ZRR, PASCAL RAW, LOD, ADE20K RAW):

• RAW-to-RGB Mapping: On the ZRR dataset, UniISP outperforms state-of-the-art methods (including AWNet, MW-ISPNet, LiteISP, and FourierISP) across PSNR, SSIM, and LPIPS metrics. It achieves these results with significantly fewer parameters and lower computational complexity (GFLOPs) due to its lightweight HAM design. Qualitative results show sharper images with richer details and higher color fidelity.
• Object Detection: On PASCAL RAW and LOD datasets, UniISP significantly improves detection performance in low-light scenarios compared to methods like RAW-Adapter. The inclusion of the Feature Adapter is shown to be critical for leveraging ISP-stage features. UniISP achieves a balance, providing superior human visual quality while avoiding the harsh contrast often seen in machine-only optimization, leading to robust detection across normal, over-exposed, and dark conditions.
• Semantic Segmentation: On the ADE20K RAW dataset, UniISP outperforms competing methods across different backbone sizes (MIT-B0, B3, B5) and illumination levels. Notably, a UniISP model with a smaller backbone (MIT-B3) surpasses other methods using larger backbones (MIT-B5) in low-light conditions, attributed to the effective feature fusion of the adapter.

Significance and Claims

The paper claims to be the first exploration of simultaneously optimizing ISP for both human visual perception and machine vision tasks. It posits that existing research has largely treated these objectives separately, leading to suboptimal solutions for either human viewing or machine analysis.

The significance of UniISP lies in its ability to:

• Preserve Information Integrity: By avoiding the information loss inherent in traditional two-stage pipelines (RAW $\to$ Human-ISP $\to$ Machine Task), UniISP retains physically meaningful information from raw sensors.
• Bridge the Domain Gap: The Feature Adapter effectively propagates RAW features to downstream networks, allowing pre-trained RGB models to function effectively on raw data.
• Achieve Dual Optimization: The adaptive training framework successfully balances the often conflicting goals of aesthetic quality and semantic performance, proving that a unified approach is feasible and superior to separate optimization strategies.

The authors conclude that this work suggests potential directions for future ISP systems, particularly in hybrid scenarios such as autonomous driving, where real-time visualization for users and environmental recognition for machines are both required.