POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

1. Problem Statement

Image Signal Processors (ISPs) convert raw sensor data into images suitable for human perception or machine vision. Traditional ISPs use fixed, hand-crafted pipelines designed for general photography, which often fail to align with specific downstream tasks (e.g., object detection, segmentation, or image enhancement).

While data-driven approaches have emerged to optimize ISPs for specific tasks, existing methods face significant challenges:

• Neural Architecture Search (NAS): Methods like ReconfigISP use differentiable proxies to mix module outputs during training but select modules discretely during inference. This creates a training-inference mismatch, leading to suboptimal performance.
• Step-wise Reinforcement Learning (RL): Methods like DRL-ISP and AdaptiveISP treat pipeline construction as a sequential decision-making process. They rely on Actor-Critic frameworks where a critic network estimates future rewards. This leads to:
  • Unstable training due to the difficulty of bootstrapped value estimation.
  • High computational overhead because the controller must be executed repeatedly at every stage to make decisions.
  • Inefficiency in capturing global dependencies between modules.

The core problem is how to jointly optimize the sequence of ISP modules and their parameters in a stable, efficient, and task-aware manner without the instability of step-wise RL or the mismatch of NAS.

2. Methodology: POS-ISP

The authors propose POS-ISP, a framework that formulates modular ISP optimization as a global sequence prediction problem solved via sequence-level Reinforcement Learning.

Core Concept

Instead of making step-by-step decisions, POS-ISP predicts the entire module sequence and its corresponding parameters in a single forward pass. The optimization is guided by a terminal task reward (the performance of the final output on the downstream task), eliminating the need for intermediate supervision or unstable future reward estimation.

Architecture Components

1. Sequence Predictor (Recurrent Policy Network):

   • Goal: Predicts the optimal sequence of modules ($A = a_1, ..., a_k$) for a given downstream task.
   • Design: Uses a Gated Recurrent Unit (GRU) based architecture.
   • Mechanism: At each recurrent step, it takes the previously selected module and the hidden state (containing context from preceding modules) to predict a probability distribution over candidate modules.
   • Benefit: This design captures inter-module dependencies, allowing the policy to understand how the order of operations affects the global pipeline context.
   • Training: Uses REINFORCE policy gradient to maximize the expected terminal reward. A temperature-controlled sampling strategy balances exploration and exploitation.

2. Parameter Predictor (Image-Adaptive Network):

   • Goal: Predicts the specific parameters ($\Theta$) for each module in the selected sequence, conditioned on the input image.
   • Design: A lightweight Encoder-Decoder CNN.
   • Mechanism: It processes a downsampled input image to extract features and outputs a set of parameters for all candidate modules. During inference, only the parameters corresponding to the selected sequence are applied.
   • Key Insight: The authors found that conditioning only on the image (rather than the sequence) yields better performance, acting as a regularizer while the sequence policy converges.

Optimization Process

• Reward Function: Defined as the improvement in downstream task performance (e.g., detection loss reduction) compared to the baseline, minus a penalty for degenerate outputs (e.g., extreme pixel values).
• Training: The Sequence Predictor and Parameter Predictor are trained jointly. The sequence predictor is updated via policy gradient, while the parameter predictor is updated via backpropagation to minimize the task loss directly.
• Inference: The sequence is fixed (greedy decoding of the most probable sequence) for a specific task, while the parameter predictor dynamically adapts to each input image.

3. Key Contributions

1. Sequence-Level Optimization Framework: Introduced POS-ISP, which predicts the full ISP pipeline in a single forward pass, avoiding the training-inference mismatch of NAS and the instability of step-wise RL.
2. Context-Aware Recurrent Predictor: Designed a GRU-based sequence predictor that explicitly models dependencies between modules, enabling the discovery of optimal global pipeline structures.
3. Efficiency and Stability: By eliminating the need for a critic network and repeated controller executions, the method achieves significantly lower computational costs and memory footprints while demonstrating more stable training dynamics.
4. Comprehensive Evaluation: Validated across diverse tasks including object detection, instance segmentation, image enhancement, and depth estimation.

4. Experimental Results

The authors evaluated POS-ISP on multiple datasets (LOD, LIS, Adobe FiveK, KITTI) and compared it against state-of-the-art methods (DRL-ISP, ReconfigISP, AdaptiveISP) and in-camera ISPs.

• Task Performance:

  • Object Detection (LOD Dataset): POS-ISP achieved the highest mAP (47.8 on LOD-Dark), outperforming AdaptiveISP (47.2) and significantly beating the in-camera ISP (37.6).
  • Instance Segmentation (LIS Dataset): POS-ISP achieved the best mAP (32.1 on LIS-Dark), showing robustness in dense prediction tasks where other RL methods struggled due to high-variance rewards.
  • Image Enhancement (Adobe FiveK): POS-ISP produced results closest to expert retouching styles (Expert C), outperforming others in PSNR, SSIM, and LPIPS metrics.
  • Depth Estimation: Demonstrated superior performance in monocular depth estimation, generalizing beyond recognition tasks.

• Computational Efficiency:

  • Runtime: POS-ISP is significantly faster. On a GPU, it reduced prediction runtime to 1.55 ms compared to 12.72 ms (AdaptiveISP) and 15.71 ms (DRL-ISP).
  • Memory: It requires only 14.4 MB of peak GPU memory, compared to 39.6 MB (AdaptiveISP) and 1013.9 MB (DRL-ISP).
  • On-Device: On a mobile CPU (Galaxy S10), POS-ISP runs ~4x faster than AdaptiveISP.

• Stability:

  • Multi-seed experiments showed POS-ISP has much lower variance (standard deviation) than baselines, confirming stable optimization.
  • Training curves showed steady improvement without the oscillations seen in Actor-Critic methods.

5. Significance

POS-ISP represents a paradigm shift in task-aware ISP design. By moving from step-wise decision making to global sequence prediction, it solves the fundamental instability issues inherent in applying standard RL to pipeline optimization.

• Practical Deployment: The drastic reduction in computational cost and memory footprint makes it feasible to deploy on mobile and edge devices, where traditional RL-based ISPs are too heavy.
• Hardware Alignment: The approach of finding a fixed sequence per task (with image-adaptive parameters) aligns well with hardware constraints where pipeline structures are often fixed in firmware, while parameters can be tuned per frame.
• Generalizability: The framework is not limited to recognition tasks; it successfully optimizes for perceptual quality and geometric tasks, proving its versatility as a general-purpose ISP optimizer.

In conclusion, POS-ISP offers a stable, efficient, and high-performing solution for tailoring image processing pipelines to specific downstream objectives, bridging the gap between theoretical optimization and practical deployment.