Raw-JPEG Adapter: Efficient Raw Image Compression with JPEG

This paper introduces RawJPEG Adapter, a lightweight and invertible preprocessing pipeline that transforms raw images for efficient standard JPEG compression while enabling accurate reconstruction by storing compact transform parameters in the JPEG comment field.

The Gist

Imagine you have a digital camera that takes photos. Inside the camera, there are two ways to save a picture:

1. The "Raw" File (DNG): This is like taking a photo with a professional chef. It captures every single detail of the ingredients (light, color, shadows) exactly as the sensor saw them. It's incredibly high quality, perfect for editing later, but the file is massive (like a giant, heavy suitcase). It's hard to carry around or share quickly.
2. The "JPEG" File: This is like a fast-food burger. It's small, light, and fits in your pocket. Everyone can open it. But to make it small, the chef throws away a lot of the "secret ingredients." If you try to edit it later, it looks mushy, grainy, or has weird color bands.

The Problem:
For years, if you wanted the high quality of the "Raw" file, you had to carry the heavy suitcase. If you wanted the small size of the "JPEG," you had to accept the mushy burger. You couldn't have both.

The Solution: The "Raw-JPEG Adapter"
This paper introduces a clever new tool called the Raw-JPEG Adapter. Think of it as a magic translator or a specialized packing cube.

Here is how it works, using a simple analogy:

1. The Packing Problem

Imagine you are trying to fit a delicate, fragile glass sculpture (your Raw Photo) into a small cardboard box (the JPEG format).

• The Old Way: You just shove the glass sculpture into the box. It breaks. When you open the box later, the glass is shattered (bad quality, banding, color errors).
• The New Way (The Adapter): Before you put the sculpture in the box, you wrap it in a special, custom-molded foam. This foam changes the shape of the sculpture slightly so it fits perfectly into the box's corners without breaking.

2. The Secret Note (The "Comment Field")

The tricky part is: How do you know how to unwrap the sculpture later?
If you just wrap it, you might forget how to unwrap it.

• The Adapter's Trick: The system writes a tiny, secret note on the outside of the box (inside the JPEG file's "comment" section). This note is only about the size of a few text messages (less than 64 KB).
• The note says: "To unwrap this, first stretch the left side, then rotate the top, then apply this specific curve."

3. The Unpacking (Decoding)

When you want to see your photo again:

1. You open the small box (the JPEG).
2. You read the tiny secret note.
3. You apply the reverse of the instructions in the note.
4. Poof! The glass sculpture is restored to its original, perfect, high-definition state.

Why is this a big deal?

• It's Tiny: The "note" is so small it barely adds to the file size. You get a file that is 100 times smaller than the original Raw file (e.g., 34 MB becomes 0.4 MB), but it still looks almost exactly like the original.
• It's Reversible: Unlike normal JPEGs, where you lose information forever, this method is fully reversible. You can get the "Raw" data back perfectly.
• It's Fast: The "packing" and "unpacking" happen in a split second. You don't need a supercomputer to do it; even a phone can handle it.
• It Works Everywhere: It uses the standard JPEG format, so your phone, computer, and printer can all open the file. You don't need special software.

The "Chef" Analogy

Think of the camera sensor as a super-precise scale that weighs ingredients in grams (Raw data).

• JPEG is like a scale that only measures in "cups" (8-bit data). If you try to weigh a tiny pinch of salt in cups, you lose the precision.
• The Adapter is a special conversion chart. It tells the scale: "If you see 1 cup, it actually means 1.234 grams. If you see 2 cups, it means 2.567 grams."
• By using this chart (the parameters stored in the file), the computer can look at the "cup" measurement and perfectly calculate the original "gram" weight.

Summary

The Raw-JPEG Adapter is a smart, lightweight software tool that lets you store your high-quality, professional camera photos in tiny, standard JPEG files. It does this by "pre-treating" the photo so it fits the JPEG format without breaking, and then "un-treating" it perfectly when you want to view or edit it later.

It's like getting the quality of a museum painting in the size of a postcard.

Technical Summary

1. Problem Statement

Digital cameras capture light as linear raw sensor data (typically 12–14 bits per channel), which preserves full sensor information essential for post-capture editing and computer vision tasks. However, storing raw data in standard formats like Digital Negative (DNG) or lossless PNG/TIFF results in massive file sizes (tens of megabytes), making them impractical for mobile storage, cloud synchronization, and large-scale datasets.

While JPEG is the ubiquitous standard for image compression, it is fundamentally designed for 8-bit, gamma-corrected, display-referred (sRGB) images. Applying JPEG directly to linear raw data causes severe degradation, including:

• Banding: Visible steps in smooth gradients due to quantization.
• Clipping: Loss of highlight and shadow details.
• Color Distortion: Mismatch between JPEG's color space transforms and raw sensor characteristics.
• Irreversibility: Direct JPEG compression destroys the linear raw signal, preventing accurate reconstruction.

Existing alternatives like JPEG 2000 or HEIC lack broad ecosystem support, and neural ISP methods often require storing large auxiliary data or suffer from low reconstruction fidelity.

2. Methodology: Raw-JPEG Adapter

The authors propose Raw-JPEG Adapter, a lightweight, learnable, and fully invertible pre-processing pipeline that adapts raw images for standard JPEG compression. The core idea is to transform the raw data before JPEG encoding so that it aligns with JPEG's quantization behavior, and then invert these transformations after decoding to recover the original raw data.

A. The Pipeline

The method consists of three main stages:

1. Pre-encoding (Adaptation):
   A compact neural network (approx. 37K parameters) analyzes a thumbnail of the raw image and predicts coefficients for three invertible operators:

   • Channel-wise 1D Look-Up Tables (LuTs): Modulates the global tone and intensity distribution of each color channel (R, G, B) to improve robustness against bit-depth reduction.
   • Pixel-wise Gamma Mapping: A $100 \times 100$gamma map ($\Gamma$) is bilinearly upsampled and applied pixel-wise. This applies local nonlinear corrections to alleviate banding in dark regions and reduce color shifts.
   • Blockwise DCT Component Scaling (Optional): An $8 \times 8$scaling matrix ($S$) is applied to the Discrete Cosine Transform (DCT) coefficients of image blocks. This adapts frequency-domain statistics to the specific camera's sensor noise and color distribution, aligning them with JPEG's fixed quantization tables.

2. Compression:
   The adapted image is compressed using a standard JPEG encoder. The learned parameters ($\Gamma$, $S$, and LuTs) are serialized, compressed (using zlib/Base64), and embedded into the JPEG Comment (COM) segment, keeping the overhead under 64 KB.

3. Post-decoding (Reconstruction):
   Upon decoding the JPEG, the process is reversed without running any deep learning models:

   • Retrieve parameters from the COM segment.
   • Apply inverse gamma correction.
   • Inverse DCT scaling (if used).
   • Apply inverse LuTs.
   • The result is a high-fidelity reconstruction of the original linear raw image.

B. Training Strategy

• Self-Supervised Learning: The network is trained end-to-end using a differentiable JPEG simulator. The loss function minimizes the difference between the original raw input and the reconstructed raw output ($\mathcal{L} = \lambda_{L1}\mathcal{L}_{L1} + \lambda_{SSIM}\mathcal{L}_{SSIM} + \lambda_{FFT}\mathcal{L}_{FFT}$).
• Augmentation: To ensure generalization across different lighting and sensors, the training includes intensity scaling and random color matrix augmentations.
• Invertibility Constraints: The network outputs are constrained (e.g., via tanh and softplus) to ensure the operators are strictly monotonic and invertible.

3. Key Contributions

1. First Invertible Raw-to-JPEG Pipeline: Introduces a method to store raw data directly in standard JPEG files with high-fidelity reconstruction, bypassing the need for proprietary formats or large auxiliary files.
2. Extreme Efficiency: The entire system is lightweight (37K parameters) with negligible runtime overhead (0.1s per image). The side information fits within the standard 64 KB JPEG comment field.
3. Universal Compatibility: Works with standard JPEG decoders (no special software needed to view the image, though the adapter is needed for raw reconstruction) and is compatible with other codecs (tested with JPEG 2000 and deep learning codecs like LIC-TCM).
4. Superior Performance: Outperforms direct JPEG storage, fixed gamma baselines, and state-of-the-art raw reconstruction methods (like bidirectional neural ISPs) in terms of reconstruction accuracy and storage efficiency.

4. Experimental Results

The method was evaluated on the S24 dataset (Samsung S24 Ultra) and cross-validated on S7, MIT-Adobe FiveK, and NUS datasets.

• Reconstruction Quality:
  • At JPEG Quality 100, Raw-JPEG Adapter achieves 49.04 dB PSNR and 0.9935 SSIM (on S24), significantly outperforming direct JPEG (46.00 dB) and fixed gamma baselines.
  • At lower qualities (e.g., Q=25), it maintains high fidelity (39.82 dB PSNR) where direct JPEG fails (36.58 dB).
• Storage Efficiency:
  • Reduces raw file sizes from ~34 MB (DNG) to < 1 MB (JPEG with adapter) while retaining the ability to re-render the image.
  • In controlled bitrate experiments (matching BPP), the adapter improves PSNR by ~2.6 dB over direct JPEG.
• Generalization:
  • The spatial-domain components (LuTs and Gamma) generalize well to unseen cameras (S7, NUS, FiveK).
  • The optional DCT component improves performance on the training camera (S24) but offers limited gains on other cameras, suggesting it captures device-specific noise profiles.
• Comparison with SOTA:
  • Outperforms bidirectional neural ISPs (e.g., Invertible ISP) which require ~7.9s runtime and often lower accuracy.
  • Outperforms metadata-based methods (e.g., R2LCM, INF) which require storing large auxiliary data (1–2 MB) or complex fitting at test time.

5. Significance and Applications

• Practical Raw Storage: Enables photographers and researchers to store raw data in the universally supported JPEG format, drastically reducing storage costs and bandwidth requirements without sacrificing the flexibility of raw editing.
• Post-Capture Flexibility: Allows users to re-render images with different white balances, tone mappings, or color profiles from a tiny JPEG file, mimicking the experience of working with a large DNG file.
• Ecosystem Integration: Since the side information is stored in the JPEG comment field, the method does not break compatibility with existing JPEG viewers or editors; the raw reconstruction simply requires a lightweight decoder step.
• Future-Proofing: The framework is codec-agnostic and can be adapted to emerging compression standards or deep learning-based codecs, offering a robust solution for the "raw storage bottleneck" in the era of high-resolution mobile photography.

In summary, Raw-JPEG Adapter successfully bridges the gap between the efficiency of JPEG compression and the fidelity requirements of raw image data, offering a practical, lightweight, and high-performance solution for modern imaging workflows.