Uni-ISP: Toward Unifying the Learning of ISPs from Multiple Mobile Cameras

This paper introduces Uni-ISP, a novel pipeline that unifies the learning of image signal processors for diverse mobile cameras using device-aware embeddings and a specialized training scheme, supported by a new real-world 4K dataset called FiveCam, to achieve superior accuracy, adaptability, and versatility in both forward and inverse ISP tasks.

The Gist

Imagine you have a group of five friends, each with their own unique way of taking photos. One friend (let's call him "Apple") loves sharp, crisp images with cool tones. Another ("Samsung") prefers deep, rich shadows. A third ("Xiaomi") makes everything look warm and vibrant.

In the past, if you wanted a computer to learn how to take photos like Apple, you had to build a specific "Apple Brain." If you wanted a "Samsung Brain," you had to build a completely different one. This was inefficient, like hiring a separate translator for every single language you might ever speak.

Uni-ISP is a revolutionary new idea that says: "Why build five different brains when we can build one super-brain that understands all of them?"

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "One-Size-Fits-None" Dilemma

Currently, AI image processors are like custom-tailored suits. If you want a suit for a specific camera, you need a tailor to measure that exact camera. If you get a new camera, you need a whole new suit. This is slow, expensive, and doesn't let the cameras "talk" to each other to learn from one another.

2. The Solution: The "Universal Translator" (Uni-ISP)

The researchers built Uni-ISP, which is like a universal translator for cameras. Instead of learning five separate languages, it learns the grammar common to all of them, while keeping a special "name tag" for each specific camera.

• The Shared Brain: Imagine a master chef who knows the fundamental rules of cooking (how to chop, how to heat, how to season). This is the "shared backbone" of the AI.
• The Device "Name Tags": To make the dish taste like a specific restaurant's style, the chef just needs to know which restaurant they are cooking for. In Uni-ISP, these are called "Device-Aware Embeddings." They are like little digital ID cards. When the AI sees the "Apple ID card," it tweaks the shared brain to make the photo look like an Apple photo. When it sees the "Samsung ID," it tweaks it for Samsung.

3. The Secret Sauce: The "Five-Camera" Dance Floor

To teach this universal translator, the researchers couldn't just use old photos. They needed a synchronized dance floor.

They built a rig with five different smartphones (iPhone, Pixel, Samsung, Huawei, Xiaomi) all taking a picture of the exact same scene at the exact same millisecond. This created a massive new dataset called FiveCam.

• Why is this cool? It's like having five people describe the same sunset simultaneously. By comparing them, the AI learns not just what a "sunset" looks like, but exactly how each person sees it differently.

4. The Magic Tricks Uni-ISP Can Do

Because this AI understands the "DNA" of multiple cameras, it can do things previous AIs couldn't:

• The "Style Shifter" (Appearance Transfer):
  Imagine taking a photo with your Samsung phone but wanting it to look like it was taken by an iPhone. Uni-ISP can instantly "translate" the Samsung photo into the iPhone style, keeping all the details but changing the mood and colors perfectly. It's like putting on a different pair of glasses that changes the world's color palette.

• The "Mix-and-Match" (Interpolation):
  You can ask the AI to create a photo that is 50% Samsung and 50% Xiaomi. It doesn't just blur them together; it invents a brand-new, perfect camera style that sits exactly in the middle of the two. You can even "extrapolate" (go beyond the limits) to create a style that is darker than any real camera, or brighter than the sun.

• The "Lie Detector" (Zero-Shot Forensics):
  This is the spy gadget. If someone edits a photo (like pasting a fake object into a picture), the Uni-ISP can spot it.

  • How? Every camera leaves a tiny, invisible fingerprint (a specific way it processes light). If a photo claims to be from an iPhone, but the "fingerprint" doesn't match the iPhone's style in the AI's brain, the AI knows it's a fake. It can even highlight exactly where the photo was tampered with.

5. The "Blur Fix" (Frequency Bias Correction)

There was one tricky problem. When the researchers tried to align the photos from the five different cameras (which are slightly different angles), the images got a little blurry, like a photo taken through a foggy window.

• The Fix: They invented a special "de-blur" tool (called Frequency Bias Correction) that acts like a sharpening filter. It tells the AI, "Don't learn to make things blurry just because the training data is blurry. Keep the sharp edges!"

The Bottom Line

Uni-ISP is a move from "specialized tools" to a "universal toolkit."

• Before: You needed a different AI for every camera.
• Now: You have one smart AI that knows every camera, can switch styles instantly, can invent new styles, and can even catch photo fakes.

It's like upgrading from having five different dictionaries to having one Omnilingual Dictionary that not only translates between languages but also helps you write poetry in a language you've never spoken before.

Technical Summary

1. Problem Statement

Modern Image Signal Processors (ISPs) are critical for transforming raw sensor data into viewable images (e.g., sRGB). While deep learning has enabled "learned ISPs" that can approximate forward (RAW $\to$ sRGB) and inverse (sRGB $\to$ RAW/XYZ) mappings, current approaches suffer from significant limitations:

• Lack of Versatility: Existing methods train separate models for individual camera devices. As the number of camera models grows, maintaining a unique ISP for each device becomes unsustainable.
• Missed Synergies: Training on single devices ignores the underlying commonalities and differences across diverse camera systems, limiting performance potential.
• Data Scarcity for Unified Learning: There is a lack of datasets containing synchronized sRGB-RAW pairs captured simultaneously by multiple different cameras, which is essential for training a unified model that can transfer styles or perform cross-camera tasks.
• Alignment Challenges: Simply mixing data from different cameras introduces misalignment and frequency bias (blurring) due to optical flow warping, degrading model performance.

2. Methodology: Uni-ISP

The authors propose Uni-ISP, a unified neural ISP framework designed to learn the forward and inverse ISP behaviors of multiple mobile cameras simultaneously.

A. Model Architecture

Uni-ISP consists of two main modules sharing an encoder-decoder structure:

1. Inverse ISP Module ($g$): Converts sRGB images back to linear XYZ space (device-independent raw representation).
2. Forward ISP Module ($h$): Converts linear XYZ space to sRGB.

Key architectural components include:

• Local Feature Extraction Blocks (LFEB): Handle detailed local processing (e.g., tone mapping, highlight recovery) using convolutions, attention mechanisms, and instance normalization.
• Global Feature Manipulation Blocks (GFMB): Incorporate EXIF metadata (exposure, ISO, f-number) to adjust global image characteristics, mimicking real ISP behavior.
• Device-aware Embedding Interaction Module (DEIM): The core innovation. It uses learnable device-aware embeddings ($E_a, E_b, \dots$) specific to each camera. Through a cross-attention mechanism, DEIM interacts with the bottleneck features of the network, allowing the shared backbone to adapt to the specific characteristics of different devices while learning common ISP behaviors.

B. Training Scheme

The authors introduce a specialized training strategy comprising two objectives:

1. Self-Camera Training: The model learns to map sRGB $\leftrightarrow$ XYZ for the same camera (standard ISP tasks).
2. Cross-Camera Training: The model learns to transfer the appearance of one camera to another (e.g., Input from Camera A $\to$ Output looking like Camera B).
   • Frequency Bias Correction (FBC) Loss: Since cross-camera training requires warping images to align them (using optical flow), this introduces high-frequency loss (blur). The authors propose an FBC loss that combines a low-frequency L1 term (for color/tones) with a frequency-domain term (Focal Frequency Loss) to preserve sharpness and prevent the model from learning to blur images.
   • Neutral Rendering Regularization: A loss term that guides the model to learn a "neutral" camera behavior when the device embedding is a zero vector, providing a baseline for interpolation/extrapolation.

C. Dataset: FiveCam

To support unified learning, the authors constructed FiveCam, a novel real-world dataset containing:

• 2,464 synchronized sRGB-RAW pairs captured simultaneously by five different smartphones (Apple iPhone 14 Pro Max, Google Pixel 6 Pro, Huawei P40, Samsung Galaxy S20, Xiaomi Mi 12).
• 4K resolution covering diverse scenes (landscape, urban, indoor, outdoor, day/night).
• Images are processed into linear XYZ space to serve as the raw modality.

3. Key Contributions

1. Unified ISP Framework: First work to unify the learning of forward and inverse ISPs across multiple diverse mobile cameras using device-aware embeddings, rather than training separate models.
2. Novel Applications: Enables capabilities previously inaccessible to learned ISPs, including:
   • Photographic Appearance Transfer: Transferring the "look" of one camera to another.
   • Inter/Extrapolation: Smoothly blending or extending photographic styles between cameras (e.g., creating a "half-Apple, half-Samsung" look).
   • Zero-Shot Image Forensics: Source camera identification and image splice detection without specific training on these tasks, leveraging the self-consistency of ISP behaviors.
3. FiveCam Dataset: A high-quality, synchronized multi-camera dataset essential for advancing cross-camera ISP research.
4. Frequency Bias Correction: A novel loss function to mitigate the blurring artifacts caused by optical flow alignment in cross-camera training.

4. Experimental Results

Extensive experiments validate Uni-ISP against state-of-the-art methods (CycleISP, InvISP, ParamISP, etc.):

• ISP Performance:
  • Inverse ISP: Achieved 32.70 dB PSNR (vs. 31.21 dB for the next best, ParamISP), an improvement of +1.5 dB.
  • Forward ISP: Achieved 29.15 dB PSNR (vs. 26.74 dB for ParamISP), an improvement of +2.4 dB.
  • Uni-ISP outperforms single-camera trained baselines even when those baselines are trained on specific devices, despite Uni-ISP using only one unified model.
• Photographic Appearance Transfer:
  • Outperformed style transfer baselines (HDRnet, NILUT) with a PSNR of 24.24 dB and the lowest DISTS score (0.1392), indicating superior perceptual quality and structural fidelity.
• Inter/Extrapolation: Successfully demonstrated smooth transitions and extrapolations between camera styles (e.g., adjusting shadow depth or night brightness) by manipulating the interpolation weight $\alpha$ in the feature space.
• Zero-Shot Forensics:
  • Source Identification: Achieved 81.67% accuracy in identifying the source camera without specific training, significantly outperforming specialized forensic networks (CMIResNet: 38.33%, MSFFN: 43.33%).
  • Splice Detection: Successfully generated confidence maps to detect tampered regions by analyzing ISP self-consistency.
• Downstream Tasks:
  • HDR Rendering: Produced the highest quality HDR results (23.26 dB PSNR) due to accurate inverse ISP.
  • Raw-Domain Deblurring: When used to convert sRGB to raw for deblurring networks, Uni-ISP yielded the best results (40.85 dB PSNR).
• Few-Shot Adaptation: Demonstrated the ability to adapt to unseen cameras (e.g., iPhone 15 Pro) by training only the new device embedding, outperforming full re-training baselines like ParamISP.

5. Significance

Uni-ISP represents a paradigm shift in computational photography by moving from device-specific models to a unified, adaptable framework.

• Scalability: It solves the scalability issue of maintaining thousands of individual ISP models for every new phone release.
• Creative Control: It democratizes professional photographic styles, allowing users to apply the aesthetic "look" of one camera brand to images taken by another.
• Forensic Utility: It provides a powerful, zero-shot tool for image authentication, leveraging the unique "fingerprint" of ISP processing chains.
• Efficiency: A single model can serve multiple devices, reducing storage and computational overhead compared to maintaining a library of specialized models.

In conclusion, Uni-ISP not only improves the accuracy of standard ISP tasks but also unlocks a new class of applications by treating ISP learning as a unified, multi-device problem rather than a collection of isolated tasks.