Modular Neural Image Signal Processing

This paper introduces a modular, fully learning-based neural image signal processing (ISP) framework that offers unprecedented control over intermediate rendering stages to enhance scalability, generalization, and flexibility, enabling a user-interactive photo-editing tool capable of unlimited post-editable re-rendering with competitive performance across multiple test sets.

The Gist

Imagine you just took a beautiful photo with your smartphone. Behind the scenes, your phone's computer (the ISP, or Image Signal Processor) has to do a massive amount of work to turn that raw, grainy, black-and-white sensor data into the vibrant, sharp, colorful image you see on your screen.

Traditionally, this process is like a factory assembly line run by a single, rigid robot. You feed in the raw data, and the robot spits out a finished photo. If you want to change the "look" (like making it warmer or cooler), you have to stop the whole line, reprogram the robot, and start over. If you want to fix a mistake, you can't just tweak one part; you have to rebuild the whole machine.

This paper introduces a new way of doing things: A "Modular Neural ISP."

Here is the simple breakdown of what they built, using some everyday analogies:

1. The "Lego" Approach Instead of the "Black Box"

Previous AI photo tools were like a sealed black box. You put raw data in, and a photo comes out. You have no idea what happened inside, and you can't change anything without breaking the whole thing.

The authors built a Lego set. Instead of one giant robot, they built a pipeline of small, specialized Lego blocks.

• Block 1: Cleans the noise (like dusting off a dirty window).
• Block 2: Fixes the colors (like a painter mixing the right shades of blue and red).
• Block 3: Adjusts the brightness and contrast (like a dimmer switch).
• Block 4: Sharpens the details (like a focus ring on a lens).

Why is this cool? Because each block does one specific job. If you want to change the "vibe" of the photo from "Cinematic" to "Vintage," you don't need to rebuild the whole factory. You just swap out the "Color" block or tweak the "Brightness" block. The rest of the pipeline stays exactly the same.

2. The "Universal Translator" (Generalization)

One of the biggest headaches in photo editing is that a tool trained on an iPhone often looks terrible on a Samsung, and vice versa. It's like teaching a chef to cook only Italian food; if you ask them to make sushi, they might struggle.

This new system is like a Universal Translator.

• They trained the "Cleaning" block to be a Generalist. It learned how to clean up noise from any camera, not just one specific brand.
• The "Style" blocks are the Specialists. They handle the specific look (like "Samsung Warm" or "iPhone Cool").
• The Result: You can take a raw photo from a camera the AI has never seen before (like an iPhone 15), and the system can still process it beautifully because the "Generalist" cleaning block knows what to do, and the "Specialist" blocks can be swapped in to give it the right look.

3. The "Time Machine" (Re-rendering)

Usually, once you save a photo as a JPEG, the raw data is gone. If you want to change the exposure or color later, you are stuck with what you have.

This paper introduces a Time Machine feature.

• When you save a photo, the tool secretly tucks the original, raw sensor data inside the JPEG file (like hiding a secret note inside a greeting card).
• Later, you can open that same JPEG, and the tool can "rewind" time, pull out the raw data, and re-process it with new settings.
• Analogy: Imagine ordering a burger. Usually, once you eat it, it's gone. With this tool, the restaurant gives you the burger and the raw ingredients in a separate container. If you decide you wanted it less salty or with more cheese, they can re-assemble the burger from the original ingredients without you ever having to go back to the farm.

4. The "DIY Photo Studio" (User Control)

Because the system is made of modular blocks, the authors built a Photo Editing Tool that feels like a pro studio but is easy to use.

• You can turn specific blocks on or off.
• You can mix and match styles (e.g., "Give me the contrast of Style A but the colors of Style B").
• You can manually adjust things like "Shadows" or "Highlights" at the exact stage where they should be adjusted, rather than guessing at the end result.

The Big Picture

In short, this paper says: "Stop treating photo processing like a magic trick where you can't see the wires. Let's build a transparent, flexible machine where you can swap parts, fix mistakes, and change styles instantly, all while using less computer power."

It's the difference between having a pre-packaged meal (old way) and having a kitchen with fresh ingredients and a recipe book (new way). You get a better meal, you can customize it to your taste, and you can even cook it for a friend who has a different set of ingredients.

Technical Summary

1. Problem Statement

Traditional Image Signal Processing (ISP) pipelines are complex, sequential chains of operations (denoising, white balancing, tone mapping, etc.) that transform raw sensor data into display-ready images.

• Limitations of Monolithic Neural ISPs: Recent learning-based approaches treat the entire ISP pipeline as a single "black-box" neural network trained end-to-end. While effective for specific cameras, these models suffer from:
  • Poor Generalization: They fail to adapt to unseen cameras without retraining.
  • Lack of Interpretability: It is difficult to debug specific stages or understand why artifacts occur.
  • Inflexibility: Supporting multiple picture styles or user edits requires training separate models or retraining the entire system, leading to high memory and computational costs.
  • Scalability Issues: Adding new features or styles often necessitates duplicating the entire pipeline.

2. Methodology

The authors propose a Modular Neural ISP Framework that decomposes the rendering process into distinct, functionally constrained stages. This design mimics the conventional raw-to-sRGB pipeline but replaces traditional operators with learnable neural modules that predict specific parameters.

Pipeline Architecture

The framework processes a demosaiced raw image ($I_{raw}$) through the following stages:

1. Raw Enhancement (Denoising):

   • A fully convolutional network ($\mathcal{D}_{raw}$) performs single-image denoising.
   • Training Strategy: Uses pseudo ground-truth generated by third-party AI denoisers (e.g., Adobe Lightroom) to avoid the need for physically captured clean references, balancing accuracy and generalization.

2. Color Correction:

   • Maps the denoised raw image to a camera-agnostic linear sRGB space ($I_{LsRGB}$).
   • Uses a diagonal White Balance (WB) matrix and an interpolated Color Correction Matrix (CCM).

3. Photofinishing Module (The Core):

   • Operates on a downsampled version of the linear sRGB image for efficiency.
   • Consists of five sequential, parameter-predicting neural networks:
     • Digital Gain ($\mathcal{D}_{gain}$): Predicts a global gain factor to adjust brightness.
     • Global Tone Mapping (GTM, $\mathcal{D}_{GTM}$): Predicts parameters for a parametric tone curve to adjust global contrast and highlights.
     • Local Tone Mapping (LTM, $\mathcal{D}_{LTM}$): Predicts spatially varying coefficients to enhance local contrast and details. It uses a multi-scale guidance network and a grid-prediction subnetwork to generate coefficient maps.
     • Chroma Mapping ($\mathcal{D}_{chroma}$): Predicts a 2D Lookup Table (LuT) in the CbCr space to adjust color saturation and artistic stylization. An optional 3D LuT can be added for complex artistic styles.
     • Gamma Correction ($\mathcal{D}_{gamma}$): Predicts a gamma factor to convert linear sRGB to a display-referred space.
   • Training: The entire module is trained end-to-end using a composite loss function (fidelity, perceptual, and regularization losses) to ensure each stage performs its intended semantic function without requiring ground truth for intermediate stages.

4. Guided Upsampling:

   • Uses a modified Bilateral Grid Upsampling (BGU) technique.
   • Innovation: Introduces per-channel gated regularization to prevent color crosstalk and eliminate the need for grid blurring, preserving sharp details better than standard BGU.

5. Detail Enhancement:

   • A lightweight network ($\mathcal{D}_{enh}$) refines the upsampled image to recover details lost during denoising and upsampling.

Key Design Principles

• Functional Decomposition: Each module is explicitly designed for a specific role (e.g., GTM handles global tone, LTM handles local contrast), enforced by architectural constraints and specific loss terms.
• Parameter Efficiency: The networks predict parameters for traditional operators (e.g., tone curves, LuTs) rather than learning a direct pixel-to-pixel mapping, keeping the model size small.
• Modularity: Stages can be replaced, updated, or swapped independently without retraining the entire system.

3. Key Contributions

1. Modular Neural ISP Framework: A learning-based pipeline that offers full control over every stage, enabling interpretability, scalability, and flexibility.
2. Cross-Camera Generalization: By decoupling camera-specific modules (like denoising) from generic ones, the framework can process raw images from unseen cameras (e.g., iPhones) without retraining, using a "generic" denoiser trained on diverse noise profiles.
3. User-Interactive Photo-Editing Tool: A practical application built on the framework that allows:
   • Re-rendering: Embedding compressed raw data into the final JPEG, allowing unlimited post-capture re-rendering with different styles and edits.
   • Style Interpolation/Mixing: Users can blend parameters between different picture styles or disable specific operators (e.g., turn off LTM) within a style.
   • Fine-Grained Control: Manual adjustments for exposure, white balance, contrast, highlights, shadows, saturation, and sharpening.
4. Efficiency: The entire pipeline (including multiple styles) requires only ~0.5M to ~3.9M parameters, significantly fewer than competing monolithic models (e.g., ISPDiffuser requires ~21M for a single style).

4. Experimental Results

• Quantitative Performance:
  • Achieved State-of-the-Art (SOTA) results on the S24 dataset across multiple picture styles (Default + 5 Artistic styles).
  • Outperformed strong baselines (LiteISP, ISPDiffuser, PyNet) in PSNR, SSIM, and $\Delta E$ metrics while using a fraction of the parameters.
  • On the MIT-Adobe FiveK dataset, the method achieved competitive results, ranking second in PSNR and first in SSIM among modular approaches.
• Generalization:
  • Successfully processed images from unseen cameras (iPhone 13/15, Samsung S9) without training on their data, achieving visual quality comparable to native ISPs and commercial software like Adobe Lightroom.
• User Study:
  • In a study with 20 participants comparing against Samsung S24 native ISP and Adobe Lightroom, the proposed method was preferred in 51.4% of overall cases, with statistically significant margins in color quality and overall preference.
• Re-rendering Capability:
  • Demonstrated that embedding raw data in JPEGs allows for high-quality re-rendering with negligible file size increase (~5-6 MB total for a 12MP image) compared to saving separate DNG files.

5. Significance

This work represents a paradigm shift in neural ISP design, moving away from monolithic black-box networks toward interpretable, modular systems.

• Practical Deployment: The low parameter count and ability to handle unseen cameras make it highly suitable for on-device deployment and software-based rendering tools.
• User Empowerment: The ability to edit raw-like data from standard JPEGs and mix styles dynamically bridges the gap between professional raw editing and consumer photography.
• Scientific Contribution: The paper provides a robust framework for analyzing and debugging ISP artifacts (e.g., halo effects in LTM) by isolating specific stages, a capability largely absent in previous end-to-end neural ISP research.

In summary, the paper presents a highly efficient, flexible, and generalizable neural ISP that not only matches or exceeds the quality of existing methods but also unlocks new possibilities for interactive image editing and cross-camera compatibility.